From 81f72f00038e16b2d41ee989c9b780d928875ce8 Mon Sep 17 00:00:00 2001 From: KubaPro010 Date: Mon, 1 Dec 2025 13:18:53 +0100 Subject: [PATCH] Add original files from 25.02.2009 19:20 --- License.txt | 342 +++++++ hash58.py | 278 ++++++ iTunesDB.py | 1035 ++++++++++++++++++++++ mp3info.py | 607 +++++++++++++ qtparse.py | 581 ++++++++++++ repear.py | 2015 ++++++++++++++++++++++++++++++++++++++++++ repear_playlists.ini | 32 + repear_scrobble.ini | 12 + scrobble.py | 243 +++++ usage.html | 349 ++++++++ 10 files changed, 5494 insertions(+) create mode 100644 License.txt create mode 100644 hash58.py create mode 100644 iTunesDB.py create mode 100644 mp3info.py create mode 100644 qtparse.py create mode 100644 repear.py create mode 100644 repear_playlists.ini create mode 100644 repear_scrobble.ini create mode 100644 scrobble.py create mode 100644 usage.html diff --git a/License.txt b/License.txt new file mode 100644 index 0000000..ecef765 --- /dev/null +++ b/License.txt @@ -0,0 +1,342 @@ + GNU GENERAL PUBLIC LICENSE + Version 2, June 1991 + + Copyright (C) 1989, 1991 Free Software Foundation, Inc. + 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The licenses for most software are designed to take away your +freedom to share and change it. By contrast, the GNU General Public +License is intended to guarantee your freedom to share and change free +software--to make sure the software is free for all its users. This +General Public License applies to most of the Free Software +Foundation's software and to any other program whose authors commit to +using it. (Some other Free Software Foundation software is covered by +the GNU Library General Public License instead.) You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +this service if you wish), that you receive source code or can get it +if you want it, that you can change the software or use pieces of it +in new free programs; and that you know you can do these things. + + To protect your rights, we need to make restrictions that forbid +anyone to deny you these rights or to ask you to surrender the rights. +These restrictions translate to certain responsibilities for you if you +distribute copies of the software, or if you modify it. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must give the recipients all the rights that +you have. You must make sure that they, too, receive or can get the +source code. And you must show them these terms so they know their +rights. + + We protect your rights with two steps: (1) copyright the software, and +(2) offer you this license which gives you legal permission to copy, +distribute and/or modify the software. + + Also, for each author's protection and ours, we want to make certain +that everyone understands that there is no warranty for this free +software. If the software is modified by someone else and passed on, we +want its recipients to know that what they have is not the original, so +that any problems introduced by others will not reflect on the original +authors' reputations. + + Finally, any free program is threatened constantly by software +patents. We wish to avoid the danger that redistributors of a free +program will individually obtain patent licenses, in effect making the +program proprietary. To prevent this, we have made it clear that any +patent must be licensed for everyone's free use or not licensed at all. + + The precise terms and conditions for copying, distribution and +modification follow. + + GNU GENERAL PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. This License applies to any program or other work which contains +a notice placed by the copyright holder saying it may be distributed +under the terms of this General Public License. The "Program", below, +refers to any such program or work, and a "work based on the Program" +means either the Program or any derivative work under copyright law: +that is to say, a work containing the Program or a portion of it, +either verbatim or with modifications and/or translated into another +language. (Hereinafter, translation is included without limitation in +the term "modification".) Each licensee is addressed as "you". + +Activities other than copying, distribution and modification are not +covered by this License; they are outside its scope. The act of +running the Program is not restricted, and the output from the Program +is covered only if its contents constitute a work based on the +Program (independent of having been made by running the Program). +Whether that is true depends on what the Program does. + + 1. You may copy and distribute verbatim copies of the Program's +source code as you receive it, in any medium, provided that you +conspicuously and appropriately publish on each copy an appropriate +copyright notice and disclaimer of warranty; keep intact all the +notices that refer to this License and to the absence of any warranty; +and give any other recipients of the Program a copy of this License +along with the Program. + +You may charge a fee for the physical act of transferring a copy, and +you may at your option offer warranty protection in exchange for a fee. + + 2. You may modify your copy or copies of the Program or any portion +of it, thus forming a work based on the Program, and copy and +distribute such modifications or work under the terms of Section 1 +above, provided that you also meet all of these conditions: + + a) You must cause the modified files to carry prominent notices + stating that you changed the files and the date of any change. + + b) You must cause any work that you distribute or publish, that in + whole or in part contains or is derived from the Program or any + part thereof, to be licensed as a whole at no charge to all third + parties under the terms of this License. + + c) If the modified program normally reads commands interactively + when run, you must cause it, when started running for such + interactive use in the most ordinary way, to print or display an + announcement including an appropriate copyright notice and a + notice that there is no warranty (or else, saying that you provide + a warranty) and that users may redistribute the program under + these conditions, and telling the user how to view a copy of this + License. (Exception: if the Program itself is interactive but + does not normally print such an announcement, your work based on + the Program is not required to print an announcement.) + +These requirements apply to the modified work as a whole. If +identifiable sections of that work are not derived from the Program, +and can be reasonably considered independent and separate works in +themselves, then this License, and its terms, do not apply to those +sections when you distribute them as separate works. But when you +distribute the same sections as part of a whole which is a work based +on the Program, the distribution of the whole must be on the terms of +this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote it. + +Thus, it is not the intent of this section to claim rights or contest +your rights to work written entirely by you; rather, the intent is to +exercise the right to control the distribution of derivative or +collective works based on the Program. + +In addition, mere aggregation of another work not based on the Program +with the Program (or with a work based on the Program) on a volume of +a storage or distribution medium does not bring the other work under +the scope of this License. + + 3. You may copy and distribute the Program (or a work based on it, +under Section 2) in object code or executable form under the terms of +Sections 1 and 2 above provided that you also do one of the following: + + a) Accompany it with the complete corresponding machine-readable + source code, which must be distributed under the terms of Sections + 1 and 2 above on a medium customarily used for software interchange; or, + + b) Accompany it with a written offer, valid for at least three + years, to give any third party, for a charge no more than your + cost of physically performing source distribution, a complete + machine-readable copy of the corresponding source code, to be + distributed under the terms of Sections 1 and 2 above on a medium + customarily used for software interchange; or, + + c) Accompany it with the information you received as to the offer + to distribute corresponding source code. (This alternative is + allowed only for noncommercial distribution and only if you + received the program in object code or executable form with such + an offer, in accord with Subsection b above.) + +The source code for a work means the preferred form of the work for +making modifications to it. For an executable work, complete source +code means all the source code for all modules it contains, plus any +associated interface definition files, plus the scripts used to +control compilation and installation of the executable. However, as a +special exception, the source code distributed need not include +anything that is normally distributed (in either source or binary +form) with the major components (compiler, kernel, and so on) of the +operating system on which the executable runs, unless that component +itself accompanies the executable. + +If distribution of executable or object code is made by offering +access to copy from a designated place, then offering equivalent +access to copy the source code from the same place counts as +distribution of the source code, even though third parties are not +compelled to copy the source along with the object code. + + 4. You may not copy, modify, sublicense, or distribute the Program +except as expressly provided under this License. Any attempt +otherwise to copy, modify, sublicense or distribute the Program is +void, and will automatically terminate your rights under this License. +However, parties who have received copies, or rights, from you under +this License will not have their licenses terminated so long as such +parties remain in full compliance. + + 5. You are not required to accept this License, since you have not +signed it. However, nothing else grants you permission to modify or +distribute the Program or its derivative works. These actions are +prohibited by law if you do not accept this License. Therefore, by +modifying or distributing the Program (or any work based on the +Program), you indicate your acceptance of this License to do so, and +all its terms and conditions for copying, distributing or modifying +the Program or works based on it. + + 6. Each time you redistribute the Program (or any work based on the +Program), the recipient automatically receives a license from the +original licensor to copy, distribute or modify the Program subject to +these terms and conditions. You may not impose any further +restrictions on the recipients' exercise of the rights granted herein. +You are not responsible for enforcing compliance by third parties to +this License. + + 7. If, as a consequence of a court judgment or allegation of patent +infringement or for any other reason (not limited to patent issues), +conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot +distribute so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you +may not distribute the Program at all. For example, if a patent +license would not permit royalty-free redistribution of the Program by +all those who receive copies directly or indirectly through you, then +the only way you could satisfy both it and this License would be to +refrain entirely from distribution of the Program. + +If any portion of this section is held invalid or unenforceable under +any particular circumstance, the balance of the section is intended to +apply and the section as a whole is intended to apply in other +circumstances. + +It is not the purpose of this section to induce you to infringe any +patents or other property right claims or to contest validity of any +such claims; this section has the sole purpose of protecting the +integrity of the free software distribution system, which is +implemented by public license practices. Many people have made +generous contributions to the wide range of software distributed +through that system in reliance on consistent application of that +system; it is up to the author/donor to decide if he or she is willing +to distribute software through any other system and a licensee cannot +impose that choice. + +This section is intended to make thoroughly clear what is believed to +be a consequence of the rest of this License. + + 8. If the distribution and/or use of the Program is restricted in +certain countries either by patents or by copyrighted interfaces, the +original copyright holder who places the Program under this License +may add an explicit geographical distribution limitation excluding +those countries, so that distribution is permitted only in or among +countries not thus excluded. In such case, this License incorporates +the limitation as if written in the body of this License. + + 9. The Free Software Foundation may publish revised and/or new versions +of the General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + +Each version is given a distinguishing version number. If the Program +specifies a version number of this License which applies to it and "any +later version", you have the option of following the terms and conditions +either of that version or of any later version published by the Free +Software Foundation. If the Program does not specify a version number of +this License, you may choose any version ever published by the Free Software +Foundation. + + 10. If you wish to incorporate parts of the Program into other free +programs whose distribution conditions are different, write to the author +to ask for permission. For software which is copyrighted by the Free +Software Foundation, write to the Free Software Foundation; we sometimes +make exceptions for this. Our decision will be guided by the two goals +of preserving the free status of all derivatives of our free software and +of promoting the sharing and reuse of software generally. + + NO WARRANTY + + 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY +FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN +OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES +PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED +OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS +TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE +PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, +REPAIR OR CORRECTION. + + 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR +REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, +INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING +OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED +TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY +YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER +PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE +POSSIBILITY OF SUCH DAMAGES. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +convey the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + + Copyright (C) + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + + +Also add information on how to contact you by electronic and paper mail. + +If the program is interactive, make it output a short notice like this +when it starts in an interactive mode: + + Gnomovision version 69, Copyright (C) year name of author + Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, the commands you use may +be called something other than `show w' and `show c'; they could even be +mouse-clicks or menu items--whatever suits your program. + +You should also get your employer (if you work as a programmer) or your +school, if any, to sign a "copyright disclaimer" for the program, if +necessary. Here is a sample; alter the names: + + Yoyodyne, Inc., hereby disclaims all copyright interest in the program + `Gnomovision' (which makes passes at compilers) written by James Hacker. + + , 1 April 1989 + Ty Coon, President of Vice + +This General Public License does not permit incorporating your program into +proprietary programs. If your program is a subroutine library, you may +consider it more useful to permit linking proprietary applications with the +library. If this is what you want to do, use the GNU Library General +Public License instead of this License. + + diff --git a/hash58.py b/hash58.py new file mode 100644 index 0000000..d103264 --- /dev/null +++ b/hash58.py @@ -0,0 +1,278 @@ +#!/usr/bin/env python +# +# hash generation library for rePear, the iPod database management tool +# Copyright (C) 2008 Martin J. Fiedler +# based on original code by William Whistler (wtbw) +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + +import os, types, re, sha + +try: + import _winreg + HaveWin32 = True +except ImportError: + HaveWin32 = False + +def GetFWIDs_Win32(): + # phase 1: enumerate all mass storage devices + if not HaveWin32: return [] + try: + key = _winreg.OpenKey(_winreg.HKEY_LOCAL_MACHINE, "SYSTEM\\CurrentControlSet\\Services\\Disk\\Enum", 0, _winreg.KEY_QUERY_VALUE) + except: + raise + return [] + devs = [] + devid = 0 + try: + while True: + dev = _winreg.QueryValueEx(key, str(devid))[0] + if type(dev) == types.UnicodeType: + dev = dev.encode('ascii', 'replace') + devs.append(dev.upper()) + devid += 1 + except: + pass + key.Close() + + # phase 2: find iPods and their FWIDs there (I'm being very careful here) + fwids = [] + for dev in devs: + dev = dev.upper().replace("\\", "&").split("&") + info = dict([x.split('_', 1) for x in dev if '_' in x]) + vendor = info.get('VEN', None) or info.get('VID', None) + product = info.get('DEV', None) or info.get('PROD', None) or info.get('PID', None) + if not(vendor in ('APPLE', '05AC')): continue + if (product != 'IPOD') and not(product[:2] in ('12', '13')): continue + fwid = dev[-2].upper() + for item in dev: + if item.startswith("000A27"): fwid = item + if len(fwid) == 16: fwids.append(fwid) + return fwids + +def GetFWIDs_Linux26(): + LINUX_DEVDIR_BASE = "/sys/bus/usb/devices" + try: + devs = os.listdir(LINUX_DEVDIR_BASE) + except OSError: + return [] + def sysfile(filename): + try: + f = open(os.path.join(devdir, filename), "rb") + data = f.read() + f.close() + return data.split("\0", 1)[0].strip().upper() + except IOError: + return "" + fwids = [] + re_devdir = re.compile(r'^\d+-\d+$') + for dev in devs: + if not re_devdir.match(dev): continue + devdir = os.path.join(LINUX_DEVDIR_BASE, dev) + if not os.path.isdir(devdir): continue + if not((sysfile("idVendor") == "05AC") \ + or (sysfile("manufacturer") == "APPLE")): continue + if not((sysfile("idProduct")[:2] in ("12", "13")) \ + or (sysfile("product") == "IPOD")): continue + fwid = sysfile("serial") + if fwid.startswith("000A27"): + fwids.append(fwid) + return fwids + +re_ioreg = re.compile(r'"(.*?)"\s+=\s+"?(.*?)"?$') +def GetFWIDs_Darwin(): + devs = [] + try: + f = os.popen("/usr/sbin/ioreg -l", 'r') + valid = False + for line in f: + line = line.strip("\r\n\t |+-") + if not line: + continue + if line.startswith('o'): + valid = (line[1:].strip().split()[0].split('@', 1)[0] == "iPod") + elif valid and line.startswith('"'): + m = re_ioreg.match(line) + if m: + key, value = m.groups() + if (key.lower() == "usb serial number") and (len(value) == 16): + devs.append(value.upper()) + f.close() + except (OSError, IOError, EOFError): + pass + return devs + +def GetFWIDs(): + if os.name == 'nt': + return GetFWIDs_Win32() + elif os.name == 'posix': + try: + uname = os.uname()[0].lower() + except (AttributeError, OSError): + return [] + if uname == 'linux': + return GetFWIDs_Linux26() + elif uname == 'darwin': + return GetFWIDs_Darwin() + return [] + + +################################################################################ + + +inv = [ + 0x74, 0x85, 0x96, 0xA7, 0xB8, 0xC9, 0xDA, 0xEB, 0xFC, 0x0D, 0x1E, 0x2F, 0x40, 0x51, 0x62, 0x73, + 0x84, 0x95, 0xA6, 0xB7, 0xC8, 0xD9, 0xEA, 0xFB, 0x0C, 0x1D, 0x2E, 0x3F, 0x50, 0x61, 0x72, 0x83, + 0x94, 0xA5, 0xB6, 0xC7, 0xD8, 0xE9, 0xFA, 0x0B, 0x1C, 0x2D, 0x3E, 0x4F, 0x60, 0x71, 0x82, 0x93, + 0xA4, 0xB5, 0xC6, 0xD7, 0xE8, 0xF9, 0x0A, 0x1B, 0x2C, 0x3D, 0x4E, 0x5F, 0x70, 0x81, 0x92, 0xA3, + 0xB4, 0xC5, 0xD6, 0xE7, 0xF8, 0x09, 0x1A, 0x2B, 0x3C, 0x4D, 0x5E, 0x6F, 0x80, 0x91, 0xA2, 0xB3, + 0xC4, 0xD5, 0xE6, 0xF7, 0x08, 0x19, 0x2A, 0x3B, 0x4C, 0x5D, 0x6E, 0x7F, 0x90, 0xA1, 0xB2, 0xC3, + 0xD4, 0xE5, 0xF6, 0x07, 0x18, 0x29, 0x3A, 0x4B, 0x5C, 0x6D, 0x7E, 0x8F, 0xA0, 0xB1, 0xC2, 0xD3, + 0xE4, 0xF5, 0x06, 0x17, 0x28, 0x39, 0x4A, 0x5B, 0x6C, 0x7D, 0x8E, 0x9F, 0xB0, 0xC1, 0xD2, 0xE3, + 0xF4, 0x05, 0x16, 0x27, 0x38, 0x49, 0x5A, 0x6B, 0x7C, 0x8D, 0x9E, 0xAF, 0xC0, 0xD1, 0xE2, 0xF3, + 0x04, 0x15, 0x26, 0x37, 0x48, 0x59, 0x6A, 0x7B, 0x8C, 0x9D, 0xAE, 0xBF, 0xD0, 0xE1, 0xF2, 0x03, + 0x14, 0x25, 0x36, 0x47, 0x58, 0x69, 0x7A, 0x8B, 0x9C, 0xAD, 0xBE, 0xCF, 0xE0, 0xF1, 0x02, 0x13, + 0x24, 0x35, 0x46, 0x57, 0x68, 0x79, 0x8A, 0x9B, 0xAC, 0xBD, 0xCE, 0xDF, 0xF0, 0x01, 0x12, 0x23, + 0x34, 0x45, 0x56, 0x67, 0x78, 0x89, 0x9A, 0xAB, 0xBC, 0xCD, 0xDE, 0xEF, 0x00, 0x11, 0x22, 0x33, + 0x44, 0x55, 0x66, 0x77, 0x88, 0x99, 0xAA, 0xBB, 0xCC, 0xDD, 0xEE, 0xFF, 0x10, 0x21, 0x32, 0x43, + 0x54, 0x65, 0x76, 0x87, 0x98, 0xA9, 0xBA, 0xCB, 0xDC, 0xED, 0xFE, 0x0F, 0x20, 0x31, 0x42, 0x53, + 0x64, 0x75, 0x86, 0x97, 0xA8, 0xB9, 0xCA, 0xDB, 0xEC, 0xFD, 0x0E, 0x1F, 0x30, 0x41, 0x52, 0x63 +] + +table1 = [ + 0x3A, 0x3F, 0x3E, 0x72, 0xBD, 0xA2, 0xD6, 0xB4, 0x63, 0xC0, 0x6E, 0x62, 0x59, 0x1E, 0xE2, 0x71, + 0xB5, 0x0D, 0xE8, 0x0C, 0x25, 0x38, 0xCE, 0x23, 0x7C, 0xB7, 0xAD, 0x16, 0xDF, 0x47, 0x3D, 0xB3, + 0x7E, 0x8C, 0xAA, 0x61, 0x31, 0x66, 0xBE, 0x4F, 0x97, 0x14, 0x54, 0xF0, 0x70, 0xEB, 0x30, 0xC4, + 0x27, 0x4E, 0xFA, 0x1A, 0x2B, 0x11, 0xF4, 0x45, 0x8E, 0x5D, 0x73, 0xED, 0x22, 0x2E, 0x7D, 0xA4, + 0x28, 0xDA, 0x2F, 0xC5, 0x92, 0x09, 0x05, 0x13, 0x9D, 0x32, 0x51, 0x4A, 0xC8, 0xBA, 0x96, 0xA7, + 0x6A, 0x50, 0xF3, 0xBC, 0x93, 0xBF, 0xB0, 0xD2, 0xD5, 0x82, 0x19, 0x98, 0x35, 0xCF, 0x6B, 0xB6, + 0x83, 0x56, 0x15, 0xF2, 0x9A, 0x9C, 0xCA, 0x74, 0x34, 0x58, 0x8D, 0xA6, 0x03, 0xFF, 0x46, 0x7B, + 0xD0, 0x7A, 0x33, 0x76, 0xDD, 0xAC, 0xCB, 0x24, 0x7F, 0xB1, 0x85, 0x60, 0xC3, 0x26, 0x8A, 0x1D, + 0x1C, 0x8F, 0x2A, 0xEF, 0x06, 0xDE, 0x67, 0x5E, 0xE7, 0xAE, 0xD9, 0xCC, 0x07, 0x6C, 0xF8, 0x0A, + 0xD3, 0x40, 0x36, 0x1F, 0x2D, 0x95, 0x43, 0xDB, 0x01, 0x89, 0x4B, 0xF7, 0xB9, 0x39, 0xC2, 0x52, + 0x53, 0xFD, 0x65, 0xF5, 0x68, 0xC1, 0xC7, 0x9F, 0x4D, 0xEA, 0xAF, 0x6D, 0x10, 0x44, 0x87, 0xD8, + 0xEE, 0x1B, 0xFE, 0x3C, 0xDC, 0x84, 0x69, 0x48, 0x6F, 0xD1, 0x57, 0x55, 0xD4, 0xA5, 0x49, 0x5B, + 0xE5, 0x0B, 0x94, 0xC9, 0x5F, 0xE1, 0x17, 0x81, 0xBB, 0xEC, 0xD7, 0xC6, 0x02, 0x4C, 0x42, 0x75, + 0xA3, 0x99, 0xE4, 0xA1, 0x9B, 0x5A, 0xF1, 0x29, 0xA0, 0x64, 0x9E, 0x18, 0x41, 0x80, 0x2C, 0x79, + 0x20, 0x8B, 0xAB, 0x90, 0x08, 0xB8, 0xA9, 0x77, 0x12, 0xF9, 0x0E, 0x88, 0xE9, 0x04, 0xFB, 0x86, + 0x0F, 0xE0, 0xA8, 0x5C, 0xE6, 0x21, 0xCD, 0x3B, 0x00, 0x78, 0xFC, 0xF6, 0xE3, 0x37, 0xB2, 0x91 +] + +table2 = [ + 0xF3, 0xE4, 0x1B, 0x38, 0xE5, 0x6F, 0xE8, 0x9D, 0x3E, 0x55, 0xBA, 0xC7, 0xAC, 0xEA, 0x66, 0xA2, + 0xB9, 0x7A, 0x34, 0x43, 0x02, 0x4E, 0xFE, 0x36, 0x41, 0x57, 0x1A, 0xB1, 0x31, 0x87, 0x04, 0x52, + 0x21, 0x22, 0xE1, 0x13, 0x7F, 0x03, 0x3A, 0x90, 0xF7, 0x69, 0x78, 0x12, 0x83, 0x0B, 0x9A, 0x97, + 0x4D, 0xB7, 0x8C, 0xBF, 0x2D, 0x94, 0xD1, 0x93, 0x2F, 0x42, 0x23, 0xA4, 0xE0, 0x92, 0xDC, 0x68, + 0xD3, 0xDD, 0xAF, 0x91, 0x9F, 0xED, 0x3D, 0x8F, 0xA1, 0x51, 0xD9, 0xE9, 0x70, 0x28, 0xEF, 0xB3, + 0x49, 0xA5, 0x0D, 0xC5, 0xD0, 0x60, 0xB4, 0x2B, 0x07, 0xF8, 0xDF, 0xE6, 0x16, 0xC0, 0x30, 0x71, + 0x85, 0xFD, 0x72, 0x95, 0x29, 0x79, 0x0A, 0x7B, 0x46, 0x11, 0x7D, 0x88, 0x1D, 0x2A, 0x48, 0x1F, + 0x45, 0x89, 0x47, 0xEE, 0xBB, 0xBE, 0x6E, 0xC3, 0x6C, 0xCE, 0x10, 0x5A, 0x2C, 0xCA, 0xFB, 0xB2, + 0xCB, 0x1C, 0x9C, 0xEC, 0x2E, 0x56, 0x59, 0x9B, 0xA6, 0x53, 0xAE, 0x17, 0x25, 0xC1, 0x3F, 0x6A, + 0x0F, 0x09, 0x01, 0xA3, 0xD6, 0xA0, 0xD8, 0x08, 0xE3, 0x74, 0x06, 0x6D, 0x19, 0x98, 0x1E, 0x77, + 0x76, 0xBC, 0xEB, 0x3C, 0xB0, 0xC4, 0xC8, 0x64, 0x0E, 0x86, 0x63, 0xD7, 0xDB, 0xBD, 0xA7, 0x82, + 0x39, 0x4F, 0x27, 0xD2, 0x5F, 0x73, 0xF4, 0x75, 0x6B, 0xC2, 0xD5, 0x67, 0x5D, 0x80, 0xAB, 0x81, + 0xDE, 0xF0, 0xAD, 0xAA, 0xCD, 0xB6, 0xF6, 0x7C, 0xFC, 0x33, 0x05, 0x14, 0x96, 0x15, 0xC9, 0x9E, + 0x35, 0x5C, 0x7E, 0x44, 0x54, 0x58, 0x3B, 0x40, 0x20, 0xA8, 0x8B, 0x5E, 0x4A, 0x24, 0x99, 0x8E, + 0xF5, 0xB5, 0x62, 0x00, 0x37, 0x5B, 0x18, 0x65, 0x8D, 0x32, 0xE2, 0xF9, 0xDA, 0x8A, 0xD4, 0xCC, + 0x26, 0xF2, 0xF1, 0xE7, 0x4B, 0xC6, 0xCF, 0xFF, 0x4C, 0x84, 0x61, 0xFA, 0xB8, 0x0C, 0xA9, 0x50 +] + +fixed = [ + 0x67, 0x23, 0xFE, 0x30, 0x45, 0x33, 0xF8, 0x90, 0x99, 0x21, 0x07, 0xC1, 0xD0, 0x12, 0xB2, 0xA1, 0x07, 0x81 +] + + +def gcd(a, b): + while b > 0: + a, b = b, a % b + return a + +def lcm(a, b): + if not(a) or not(b): + return 1 + return a * b / gcd(a, b) + + +def UpdateHash(db, fwid): + # extract dbid, zero out all hash stuff and add hash indicator + dbid = db[24:32] + hash2 = db[50:70] + z = 20 * "\0" + db = db[:24] + (8 * "\0") + db[32:48] + "\x01\x00" + z + db[70:88] + z + db[108:] + + # convert fwid to byte array + fwid = [int(fwid[i:i+2], 16) for i in xrange(0, 16, 2)] + + # key generation, step 1: take LCM of each two bytes in the FWID in turn + key = 16 * [0] + for i in (0, 2, 4, 6): + l = lcm(*fwid[i:i+2]) + hi = (l & 0xFF00) >> 8 + lo = l & 0x00FF + j = i << 1 + key[j] = ((table1[hi] * 0xB5) - 0x03) & 0xFF + key[j|1] = ((table2[hi] * 0xB7) + 0x49) & 0xFF + key[j|2] = ((table1[lo] * 0xB5) - 0x03) & 0xFF + key[j|3] = ((table2[lo] * 0xB7) + 0x49) & 0xFF + # step 2: invert key + key = [inv[x] for x in key] + # step 3: create hash key + key = fixed + key + key = map(ord, sha.new("".join(map(chr, key))).digest()) + + # first XOR + key = [(x ^ 0x36) for x in key] + 44 * [0x36] + + # first SHA + h = sha.new("".join(map(chr, key)) + db).digest() + + # second XOR + key = [(x ^ (0x36 ^ 0x5C)) for x in key] + + # second SHA + h = sha.new("".join(map(chr, key)) + h).digest() + + # reassemble database + return db[:24] + dbid + db[32:50] + hash2 + db[70:88] + h + db[108:] + + +################################################################################ + + +if __name__ == "__main__": + import sys + fwids = GetFWIDs() + print "detected FWIDs:", fwids + if fwids: + fwid = fwids[0] + else: + fwid = "000A27001B3EAD37" + print "no FWID detected, using default FWID for BIST" + + try: + old = file(sys.argv[1], "rb").read() + except IndexError: + sys.exit(0) + new = UpdateHash(old, "000A27001B3EAD37") + print "old =>", " ".join(["%02X" % ord(c) for c in old[88:108]]) + print "new =>", " ".join(["%02X" % ord(c) for c in new[88:108]]) + if old == new: + print "MATCH!" + else: + print "no match :(" + try: + file(sys.argv[2], "wb").write(new) + except IndexError: + pass diff --git a/iTunesDB.py b/iTunesDB.py new file mode 100644 index 0000000..2b2bbef --- /dev/null +++ b/iTunesDB.py @@ -0,0 +1,1035 @@ +#!/usr/bin/env python +# +# iTunesDB generator library for rePear, the iPod database management tool +# Copyright (C) 2006-2008 Martin J. Fiedler +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + +import struct, random, types, array, sys, os, stat, time +try: + import Image, JpegImagePlugin, PngImagePlugin + PILAvailable = True +except ImportError: + PILAvailable = False + + +def DefaultLoggingFunction(text, force_flush=True): + sys.stdout.write(text) + if force_flush: + sys.stdout.flush() +log = DefaultLoggingFunction + + +################################################################################ +## some helper classes to represent ITDB records, and some helper functions ## +################################################################################ + +class Field: + def __str__(self): raise Exception, "abstract function call" + def __len__(self): raise Exception, "abstract function call" + +class F_Tag(Field): + def __init__(self, tag): self.tag = tag + def __str__(self): return self.tag + def __len__(self): return len(self.tag) + +class F_Formatable(Field): + def __init__(self, format, value): + self.format = format + self.value = value + def __str__(self): return struct.pack("<"+self.format, self.value) + def __len__(self): return struct.calcsize(self.format) + +class F_Int64(F_Formatable): + def __init__(self, value): F_Formatable.__init__(self, "Q", value) +class F_Int32(F_Formatable): + def __init__(self, value): F_Formatable.__init__(self, "L", value) +class F_Int16(F_Formatable): + def __init__(self, value): F_Formatable.__init__(self, "H", value) +class F_Int8(F_Formatable): + def __init__(self, value): F_Formatable.__init__(self, "B", value) + +class F_HeaderLength(F_Int32): + def __init__(self): F_Int32.__init__(self, 0) +class F_TotalLength(F_Int32): + def __init__(self): F_Int32.__init__(self, 0) +class F_ChildCount(F_Int32): + def __init__(self): F_Int32.__init__(self, 0) + +class F_Padding(Field): + def __init__(self, length): self.length = length + def __str__(self): return self.length * "\0" + def __len__(self): return self.length + + +class Record: + def __init__(self, header): + self.header_length_at = None + self.total_length_at = None + self.child_count_at = None + data = "" + for field in header: + if field.__class__ == F_HeaderLength: self.header_length_at = len(data) + if field.__class__ == F_TotalLength: self.total_length_at = len(data) + if field.__class__ == F_ChildCount: self.child_count_at = len(data) + data += str(field) + if self.header_length_at: + data = data[:self.header_length_at] + struct.pack(" 86402: return False + return ((diff % 3600) in (0, 1, 2, 3598, 3599)) + + +################################################################################ +## some higher-level ITDB record classes ## +################################################################################ + +class StringDataObject(Record): + def __init__(self, mhod_type, content): + if type(content) != types.UnicodeType: + content = unicode(content, sys.getfilesystemencoding(), 'replace') + content = content.encode('utf_16_le', 'replace') + Record.__init__(self, ( + F_Tag("mhod"), + F_Int32(0x18), + F_TotalLength(), + F_Int32(mhod_type), + F_Padding(8), + F_Int32(1), + F_Int32(len(content)), + F_Int32(1), + F_Padding(4) + )) + self.add(content) + + +class OrderDataObject(Record): + def __init__(self, order): + Record.__init__(self, ( + F_Tag("mhod"), + F_Int32(0x18), + F_Int32(0x2C), + F_Int32(100), + F_Padding(8), + F_Int32(order), + F_Padding(16) + )) + + +class TrackItemRecord(Record): + def __init__(self, info): + if not 'id' in info: + raise KeyError, "no track ID set" + format = info.get('format', "mp3-cbr") + if info.get('artwork', None): + default_has_artwork = True + default_artwork_size = 1 + else: + default_has_artwork = False + default_artwork_size = 0 + if 'video format' in info: + media_type = 2 + else: + media_type = 1 + Record.__init__(self, ( + F_Tag("mhit"), + F_HeaderLength(), + F_TotalLength(), + F_ChildCount(), + F_Int32(info.get('id', 0)), # !!! + F_Int32(info.get('visible', 1)), # visible + F_Tag({"mp3": " 3PM", "aac": " CAA", "mp4a": "A4PM"}.get(format[:3], "\0\0\0\0")), + F_Int16({"mp3-cbr": 0x100, "mp3-vbr": 0x101, "aac": 0, "mp4a": 0}.get(format, 0)), + F_Int8(info.get('compilation', 0)), + F_Int8(info.get('rating', 0)), + F_Int32(unixtime2mactime(info.get('mtime', 0))), + F_Int32(info.get('size', 0)), # !!! + F_Int32(int(info.get('length', 0) * 1000)), # !!! + F_Int32(info.get('track number', 0)), + F_Int32(info.get('total tracks', 0)), + F_Int32(info.get('year', 0)), + F_Int32(info.get('bitrate', 0)), # !!! + F_Int16(0), + F_Int16(info.get('sample rate', 0)), # !!! + F_Int32(info.get('volume', 0)), + F_Int32(info.get('start time', 0)), + F_Int32(info.get('stop time', 0)), + F_Int32(info.get('soundcheck', 0)), + F_Int32(info.get('play count', 0)), + F_Int32(0), + F_Int32(unixtime2mactime(info.get('last played time', 0))), + F_Int32(info.get('disc number', 0)), + F_Int32(info.get('total discs', 0)), + F_Int32(info.get('user id', 0)), + F_Int32(info.get('date added', 0)), + F_Int32(int(info.get('bookmark time', 0) * 1000)), + F_Int64(info.get('dbid', 0)), # !!! + F_Int8(info.get('checked', 0)), + F_Int8(info.get('application rating', 0)), + F_Int16(info.get('BPM', 0)), + F_Int16(info.get('artwork count', 1)), + F_Int16({"wave": 0, "audible": 1}.get(format, 0xFFFF)), + F_Int32(info.get('artwork size', default_artwork_size)), + F_Int32(0), + F_Formatable("f", info.get('sample rate', 0)), + F_Int32(info.get('release date', 0)), + F_Int16({"aac": 0x0033, "mp4a": 0x0033, "audible": 0x0029, "wave:": 0}.get(format, 0x0C)), + F_Int16(info.get('explicit flag', 0)), + F_Padding(8), + F_Int32(info.get('skip count', 0)), + F_Int32(unixtime2mactime(info.get('last skipped time', 0))), + F_Int8(2 - int(info.get('has artwork', default_has_artwork))), + F_Int8(not info.get('shuffle flag', 1)), + F_Int8(info.get('bookmark flag', 0)), + F_Int8(info.get('podcast flag', 0)), + F_Int64(info.get('dbid', 0)), + F_Int8(info.get('lyrics flag', 0)), + F_Int8(info.get('movie flag', 0)), + F_Int8(info.get('played mark', 1)), + F_Padding(9), + F_Int32(ifelse(format[:3]=="mp3", 0, info.get('sample count', 0))), + F_Padding(16), + F_Int32(media_type), + F_Int32(0), # season number + F_Int32(0), # episode number + F_Padding(28), + F_Int32(info.get('gapless data', 0)), + F_Int32(0), + F_Int16(info.get('gapless track flag', 0)), + F_Int16(info.get('gapless album flag', 0)), + F_Padding(20), # hash + F_Padding(18), # misc unknowns + F_Int16(info.get('album id', 0)), + F_Padding(52), # padding before mhii link + F_Int32(info.get('mhii link', 0)) + )) + for mhod_type, key in ((1,'title'), (4,'artist'), (3,'album'), (5,'genre'), (6,'filetype'), (2,'path')): + if key in info: + value = info[key] + if key=="path": + value = ":" + value.replace("/", ":").replace("\\", ":") + self.add(StringDataObject(mhod_type, value)) + + +class PlaylistItemRecord(Record): + def __init__(self, order, trackid, timestamp=0): + Record.__init__(self, ( + F_Tag("mhip"), + F_HeaderLength(), + F_TotalLength(), + F_ChildCount(), + F_Int32(0), + F_Int32((trackid + 0x1337) & 0xFFFF), + F_Int32(trackid), + F_Int32(timestamp), + F_Int32(0), + F_Padding(40) + )) + self.add(OrderDataObject(order)) + + +class PlaylistRecord(Record): + def __init__(self, name, track_count, order=0, master=0, timestamp=0, plid=None, sort_order=1): + if not plid: plid = random.randrange(0L, 18446744073709551615L) + Record.__init__(self, ( + F_Tag("mhyp"), + F_HeaderLength(), + F_TotalLength(), + F_ChildCount(), + F_Int32(track_count), + F_Int32(master), + F_Int32(timestamp), + F_Int64(plid), + F_Int32(0), + F_Int16(1), + F_Int16(0), + F_Int32(sort_order), + F_Padding(60) + )) + self.add(StringDataObject(1, name)) + self.add(OrderDataObject(order)) + + def add_index(self, tracklist, index_type, fields): + order = range(len(tracklist)) + order.sort(lambda a,b: compare_dict(tracklist[a], tracklist[b], fields)) + mhod = Record(( + F_Tag("mhod"), + F_Int32(24), + F_TotalLength(), + F_Int32(52), + F_Padding(8), + F_Int32(index_type), + F_Int32(len(order)), + F_Padding(40) + )) + arr = array.array('L', order) + # the array module doesn't directly support endianness, so we detect + # the machine's endianness and swap if it is big-endian + if ord(array.array('L', [1]).tostring()[3]): + arr.byteswap() + data = arr.tostring() + mhod.add(data) + self.add(mhod) + + def set_playlist(self, track_ids): + for i in xrange(len(track_ids)): + self.add(PlaylistItemRecord(i+1, track_ids[i]), 0) + + + +################################################################################ +## the toplevel ITDB class ## +################################################################################ + +class iTunesDB: + def __init__(self, tracklist, name="Unnamed", dbid=None, dbversion=0x19): + if not dbid: dbid = random.randrange(0L, 18446744073709551615L) + + self.mhbd = Record(( + F_Tag("mhbd"), + F_HeaderLength(), + F_TotalLength(), + F_Int32(0), + F_Int32(dbversion), + F_ChildCount(), + F_Int64(dbid), + F_Int16(2), + F_Padding(14), + F_Int16(0), # hash indicator (set later by hash58) + F_Padding(20), # first hash + F_Tag("en"), # language = 'en' + F_Tag("\0rePear!"), # library persistent ID + F_Padding(20), # hash58 + F_Padding(80) + )) + + self.mhsd = Record(( + F_Tag("mhsd"), + F_HeaderLength(), + F_TotalLength(), + F_Int32(1), + F_Padding(80) + )) + self.mhlt = Record(( + F_Tag("mhlt"), + F_HeaderLength(), + F_ChildCount(), + F_Padding(80) + )) + + for track in tracklist: + self.mhlt.add(TrackItemRecord(track)) + + self.mhsd.add(self.mhlt) + del self.mhlt + self.mhbd.add(self.mhsd) + + self.mhsd = Record(( + F_Tag("mhsd"), + F_HeaderLength(), + F_TotalLength(), + F_Int32(2), + F_Padding(80) + )) + self.mhlp = Record(( + F_Tag("mhlp"), + F_HeaderLength(), + F_ChildCount(), + F_Padding(80) + )) + + mhyp = PlaylistRecord(name, len(tracklist), master=1, sort_order=10) + mhyp.add_index(tracklist, 0x03, ('title',)) + mhyp.add_index(tracklist, 0x04, ('album','disc number','track number','title')) + mhyp.add_index(tracklist, 0x05, ('artist','album','disc number','track number','title')) + mhyp.add_index(tracklist, 0x07, ('genre','artist','album','disc number','track number','title')) + mhyp.add_index(tracklist, 0x12, ('composer','title')) + mhyp.set_playlist([track['id'] for track in tracklist]) + self.mhlp.add(mhyp) + + def add_playlist(self, tracks, name="Unnamed"): + mhyp = PlaylistRecord(name, len(tracks), sort_order=1) + mhyp.set_playlist([track['id'] for track in tracks]) + self.mhlp.add(mhyp) + + def finish(self): + self.mhsd.add(self.mhlp) + del self.mhlp + self.mhbd.add(self.mhsd) + del self.mhsd + result = str(self.mhbd) + del self.mhbd + return result + + + +################################################################################ +## ArtworkDB / PhotoDB record classes ## +################################################################################ + +class RGB565_LE: + bpp = 16 + def convert(data): + res = array.array('B', [0 for x in xrange(len(data)/3*2)]) + io = 0 + for ii in xrange(0, len(data), 3): + g = ord(data[ii+1]) >> 2 + res[io] = ((g & 7) << 5) | (ord(data[ii+2]) >> 3) + res[io|1] = (ord(data[ii]) & 0xF8) | (g >> 3) + io += 2 + return res.tostring() + convert = staticmethod(convert) + +ImageFormats = { + 'nano': ((1027, 100, 100, RGB565_LE), + (1031, 42, 42, RGB565_LE)), + 'photo': ((1016, 140, 140, RGB565_LE), + (1017, 56, 56, RGB565_LE)), + 'video': ((1028, 100, 100, RGB565_LE), + (1029, 200, 200, RGB565_LE)), + 'nano3g': ((1055, 128, 128, RGB565_LE), + (1060, 320, 320, RGB565_LE), + (1061, 55, 56, RGB565_LE)), + 'nano4g': ((1055, 128, 128, RGB565_LE), + (1078, 80, 80, RGB565_LE), + (1071, 240, 240, RGB565_LE), + (1074, 50, 50, RGB565_LE)), + '4g': 'photo', + '5g': 'video', + '6g': 'nano3g', + 'classic': 'nano3g', + 'nano1g': 'nano', + 'nano2g': 'nano', +} + +class ImageInfo: + pass + +class ArtworkFormat: + def __init__(self, descriptor, cache_info=(0,0)): + self.fid, self.height, self.width, self.format = descriptor + self.filename = "F%04d_1.ithmb" % self.fid + self.size = self.width * self.height * self.format.bpp/8 + self.fullname = "iPod_Control/Artwork/" + self.filename + + # check if the cache file can be used + try: + s = os.stat(self.fullname) + use_cache = stat.S_ISREG(s[stat.ST_MODE]) \ + and compare_mtime(cache_info[0], s[stat.ST_MTIME]) \ + and (s[stat.ST_SIZE] == cache_info[1]) + except OSError: + use_cache = False + + # load the cache + if use_cache: + try: + f = open(self.fullname, "rb") + self.cache = f.read() + f.close() + except IOError: + use_cache = False + if not use_cache: + self.cache = None + + # open the destination file + try: + self.f = open(self.fullname, "wb") + except IOError, e: + log("WARNING: Error opening the artwork data file `%s'\n", self.filename) + self.f = None + + def close(self): + if self.f: + self.f.close() + try: + s = os.stat(self.fullname) + cache_info = (s[stat.ST_MTIME], s[stat.ST_SIZE]) + except OSError: + cache_info = (0, 0) + return (self.fid, cache_info) + + def GenerateImage(self, image, index, cache_entry=None): + if cache_entry and self.cache: + offset = self.size * cache_entry['index'] + data = self.cache[offset : offset+self.size] + sx = cache_entry['dim'][self.fid]['sx'] + sy = cache_entry['dim'][self.fid]['sy'] + mx = cache_entry['dim'][self.fid]['mx'] + my = cache_entry['dim'][self.fid]['my'] + else: + log(" [%dx%d]" % (self.width, self.height), True) + + # sx/sy = resulting image size + sx = self.width + sy = image.size[1] * sx / image.size[0] + if sy > self.height: + sy = self.height + sx = image.size[0] * sy / image.size[1] + # mx/my = margin size + mx = self.width - sx + my = self.height - sy + + # process the image + temp = image.resize((sx, sy), Image.ANTIALIAS) + thumb = Image.new('RGB', (self.width, self.height), (255, 255, 255)) + thumb.paste(temp, (mx/2, my/2)) + del temp + data = self.format.convert(thumb.tostring()) + del thumb + + # save the image + try: + self.f.seek(self.size * index) + self.f.write(data) + except IOError: + log(" [WRITE ERROR]", True) + + # return image metadata + iinfo = ImageInfo() + iinfo.format = self + iinfo.index = index + iinfo.sx = sx + iinfo.sy = sy + iinfo.mx = mx + iinfo.my = my + return iinfo + + + +class ArtworkDBStringDataObject(Record): + def __init__(self, mhod_type, content): + if type(content) != types.UnicodeType: + content = unicode(content, sys.getfilesystemencoding(), 'replace') + content = content.encode('utf_16_le', 'replace') + padding = len(content) % 4 + if padding: padding = 4 - padding + Record.__init__(self, ( + F_Tag("mhod"), + F_Int32(0x18), + F_TotalLength(), + F_Int16(mhod_type), + F_Int16(padding), + F_Padding(8), + F_Int32(len(content)), + F_Int32(2), + F_Int32(0) + )) + self.add(content) + if padding: + self.add("\0" * padding) + + +class ImageDataObject(Record): + def __init__(self, iinfo): + Record.__init__(self, ( + F_Tag("mhod"), + F_Int32(0x18), + F_TotalLength(), + F_Int32(2), + F_Padding(8) + )) + + mhni = Record(( + F_Tag("mhni"), + F_Int32(0x4C), + F_TotalLength(), + F_ChildCount(), + F_Int32(iinfo.format.fid), + F_Int32(iinfo.format.size * iinfo.index), + F_Int32(iinfo.format.size), + F_Int16(iinfo.my), + F_Int16(iinfo.mx), + F_Int16(iinfo.sy), + F_Int16(iinfo.sx), + F_Padding(4), + F_Int32(iinfo.format.size), + F_Padding(32) + )) + + mhod = ArtworkDBStringDataObject(3, ":" + iinfo.format.filename) + mhni.add(mhod) + self.add(mhni) + + +class ImageItemRecord(Record): + def __init__(self, img_id, dbid, iinfo_list, orig_size=0): + Record.__init__(self, ( + F_Tag("mhii"), + F_Int32(0x98), + F_TotalLength(), + F_ChildCount(), + F_Int32(img_id), + F_Int64(dbid), + F_Padding(20), + F_Int32(orig_size), + F_Padding(100) + )) + + for iinfo in iinfo_list: + self.add(ImageDataObject(iinfo)) + + +def ArtworkDB(model, imagelist, base_id=0x40, cache_data=({}, {})): + while type(ImageFormats.get(model, None)) == types.StringType: + model = ImageFormats[model] + if not model in ImageFormats: + return None + + format_cache, image_cache = cache_data + formats = [] + for descriptor in ImageFormats[model]: + formats.append(ArtworkFormat(descriptor, + cache_info = format_cache.get(descriptor[0], (0,0)))) + # if there's at least one format whose image file isn't cache-clean, + # invalidate the cache + if not formats[-1].cache: + image_cache = {} + + # Image List + mhsd = Record(( + F_Tag("mhsd"), + F_HeaderLength(), + F_TotalLength(), + F_Int32(1), + F_Padding(80) + )) + mhli = Record(( + F_Tag("mhli"), + F_HeaderLength(), + F_ChildCount(), + F_Padding(80) + )) + + img_id = base_id + index = 0 + output_image_cache = {} + image_count = 0 + dbid2mhii = {} + for source, dbid_list in imagelist.iteritems(): + log(source, False) + + # stat this image + try: + s = os.stat(source) + except OSError, e: + log(" [Error: %s]\n" % e.strerror, True) + continue + + # check if the image is cacheworthy + cache_entry = image_cache.get(source, None) + if cache_entry: + if (cache_entry['size'] != s[stat.ST_SIZE]) \ + or not(compare_mtime(cache_entry['mtime'], s[stat.ST_MTIME])): + cache_entry = None + + # if it's not cached, open the image + if not cache_entry: + try: + image = Image.open(source) + image.tostring() + except IOError, e: + log(" [Error: %s]\n" % e, True) + continue + else: + log(" [cached]", True) + image = None + + # generate the image data and ArtworkDB records + iinfo_list = [format.GenerateImage(image, index, cache_entry) for format in formats] + for dbid in dbid_list: + mhli.add(ImageItemRecord(img_id, dbid, iinfo_list, s[stat.ST_SIZE])) + dbid2mhii[dbid] = img_id + img_id += 1 + del image + + # add the image into the new cache + dim = {} + for iinfo in iinfo_list: + dim[iinfo.format.fid] = { + 'sx': iinfo.sx, + 'sy': iinfo.sy, + 'mx': iinfo.mx, + 'my': iinfo.my + } + output_image_cache[source] = { + 'index': index, + 'size': s[stat.ST_SIZE], + 'mtime': s[stat.ST_MTIME], + 'dim': dim + } + + # done with this image + del iinfo_list + index += 1 + image_count += len(dbid_list) + log(" [OK]\n", True) + + # Date File Header + mhfd = Record(( + F_Tag("mhfd"), + F_HeaderLength(), + F_TotalLength(), + F_Int32(0), + F_Int32(2), + F_Int32(3), + F_Int32(0), + F_Int32(base_id + image_count), + F_Padding(16), + F_Int32(2), + F_Padding(80) + )) + + mhsd.add(mhli) + mhfd.add(mhsd) + + # Album List (dummy) + mhsd = Record(( + F_Tag("mhsd"), + F_HeaderLength(), + F_TotalLength(), + F_Int32(2), + F_Padding(80) + )) + mhsd.add(Record(( + F_Tag("mhla"), + F_HeaderLength(), + F_Int32(0), + F_Padding(80) + ))) + mhfd.add(mhsd) + + # File List + mhsd = Record(( + F_Tag("mhsd"), + F_HeaderLength(), + F_TotalLength(), + F_Int32(3), + F_Padding(80) + )) + + mhlf = Record(( + F_Tag("mhlf"), + F_HeaderLength(), + F_Int32(len(formats)), + F_Padding(80) + )) + + for format in formats: + mhlf.add(Record(( + F_Tag("mhif"), + F_HeaderLength(), + F_TotalLength(), + F_Int32(0), + F_Int32(format.fid), + F_Int32(format.size), + F_Padding(100) + ))) + + # finalize ArtworkDB + mhsd.add(mhlf) + mhfd.add(mhsd) + output_format_cache = dict([format.close() for format in formats]) + del formats + output_cache_data = (output_format_cache, output_image_cache) + return (str(mhfd), output_cache_data, dbid2mhii) + + +################################################################################ +## a rudimentary ITDB reader (only reads titles, no playlists, and isn't very ## +## fault-tolerant) for the "dissect" action ## +################################################################################ + +mhod_type_map = { + 1: 'title', + 2: 'path', + 3: 'album', + 4: 'artist', + 5: 'genre', + 6: 'filetype', + 8: 'comment', + 12: 'composer' +} + +class InvalidFormat(Exception): pass + +class DatabaseReader: + def __init__(self, f="iPod_Control/iTunes/iTunesDB"): + if type(f)==types.StringType: + f = open(f, "rb") + self.f = f + self._skip_header("mhbd") + while True: + h = self._skip_header("mhsd") + if len(h) < 16: + raise InvalidFormat + size, mhsd_type = struct.unpack(' yeah! + if size < len(h): + raise InvalidFormat + self.f.seek(size - len(h), 1) + self._skip_header("mhlt") + + def _skip_header(self, tag): # a little helper function + hh = self.f.read(8) + if (len(hh) != 8) or (hh[:4] != tag): + raise InvalidFormat + size = struct.unpack(' 40) and (data[:4] == "mhod"): + size, mhod_type = struct.unpack('> 16, (x >> 8) & 0xFF, x & 0xFF) + +SD_type_map = { "aac": 2, "mp4a": 2, "wave": 4} + +def MakeSDEntry(info): + path = info['path'] + if type(path) != types.UnicodeType: + path = unicode(path, sys.getfilesystemencoding(), 'replace') + path = u'/' + path + return "\0\x02\x2E\x5A\xA5\x01" + (20*"\0") + \ + "\x64\0\0%c\0\x02\0" % (SD_type_map.get(info.get('type', None), 1)) + \ + path.encode("utf_16_le", 'replace') + \ + ((261 - len(path)) * 2) * "\0" + \ + "%c%c\0" % (info.get('shuffle flag', 1), info.get('bookmark flag', 0)) + +def iTunesSD(tracklist): + header = "\0\x02\x2E\x5A\xA5\x01" + (20*"\0") + "\x64\0\0\0x01\0\0x02\0" + return be3(len(tracklist)) + "\x01\x06\0\0\0\x12" + (9*"\0") + \ + "".join(map(MakeSDEntry, tracklist)) + + +################################################################################ +## some useful helper functions for "fine tuning" of track lists ## +################################################################################ + +def GenerateIDs(tracklist): + trackid = random.randint(0, (0xFFFF-0x1337) - len(tracklist)) + dbid = random.randrange(0, 18446744073709551615L - len(tracklist)) + for track in tracklist: + track['id'] = trackid + track['dbid'] = dbid + trackid += 1 + dbid += 1 + + +def GuessTitleAndArtist(filename): + info = {} + filename = os.path.split(filename)[1] + filename = os.path.splitext(filename)[0] + filename = filename.replace('_', ' ') + n = "" + for i in xrange(len(filename)): + c = filename[i] + if c in "0123456789": + n += c + continue + if c in " -": + if n: info['track number'] = int(n) + filename = filename[i+1:] + break + parts = filename.split('-', 1) + if len(parts)==2: + info['artist'] = parts[0].strip() + info['title'] = parts[1].strip(" -\r\n\t\v") + else: + info['title'] = filename.strip() + return info + +def FillMissingTitleAndArtist(track_or_list): + if type(track_or_list)==types.ListType: + for track in track_or_list: + FillMissingTitleAndArtist(track) + else: + if track_or_list.get('title',None) and track_or_list.get('artist',None): + return # no need to do something, it's fine already + guess = GuessTitleAndArtist(track_or_list['path']) + for key in ('title', 'artist', 'track number'): + if not(track_or_list.get(key,None)) and guess.get(key,None): + track_or_list[key] = guess[key] + + +################################################################################ +## some additional general purpose helper functions ## +################################################################################ + +def ASCIIMap(c): + if ord(c) < 32: return "." + if ord(c) == 127: return "." + return c + +def HexDump(obj): + s = str(obj) + offset = 0 + while s: + line = "%08X | " % offset + for i in xrange(16): + if i < len(s): + line += "%02X " % ord(s[i]) + else: + line += " " + line += "| " + "".join(map(ASCIIMap, s[:16])) + print line + offset += 16 + s = s[16:] + + +def DisplayTitle(info): + s = kill_unicode(info.get('title', "")) + if 'album' in info: s = "%s -> %s" % (kill_unicode(info['album']), s) + if 'artist' in info: s = "%s: %s" % (kill_unicode(info['artist']), s) + q = [str(kill_unicode(info[key])) for key in ('genre','year') if key in info] + if q: s = "%s [%s]" % (s, ", ".join(q)) + return s + + +################################################################################ + +if __name__ == "__main__": + print "Do not start this file directly, start repear.py instead." diff --git a/mp3info.py b/mp3info.py new file mode 100644 index 0000000..13020c4 --- /dev/null +++ b/mp3info.py @@ -0,0 +1,607 @@ +#!/usr/bin/env python +# +# audio file information library for rePear, the iPod database management tool +# Copyright (C) 2006-2008 Martin J. Fiedler +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + +import sys, re, zlib, struct, os, stat +import qtparse + + +################################################################################ +## a sh*tload of constants ## +################################################################################ + +ID3v1Genres = { 0:"Blues", 1:"Classic Rock", 2:"Country", 3:"Dance", 4:"Disco", +5:"Funk", 6:"Grunge", 7:"Hip-Hop", 8:"Jazz", 9:"Metal", 10:"New Age", +11:"Oldies", 12:"Other", 13:"Pop", 14:"R&B", 15:"Rap", 16:"Reggae", 17:"Rock", +18:"Techno", 19:"Industrial", 20:"Alternative", 21:"Ska", 22:"Death Metal", +23:"Pranks", 24:"Soundtrack", 25:"Euro-Techno", 26:"Ambient", 27:"Trip-Hop", +28:"Vocal", 29:"Jazz&Funk", 30:"Fusion", 31:"Trance", 32:"Classical", +33:"Instrumental", 34:"Acid", 35:"House", 36:"Game", 37:"Sound Clip", +38:"Gospel", 39:"Noise", 40:"Alternative Rock", 41:"Bass", 42:"Soul", +43:"Punk", 44:"Space", 45:"Meditative", 46:"Instrumental Pop", +47:"Instrumental Rock", 48:"Ethnic", 49:"Gothic", 50:"Darkwave", +51:"Techno-Industrial", 52:"Electronic", 53:"Pop-Folk", 54:"Eurodance", +55:"Dream", 56:"Southern Rock", 57:"Comedy", 58:"Cult", 59:"Gangsta", +60:"Top 40", 61:"Christian Rap", 62:"Pop/Funk", 63:"Jungle", 64:"Native US", +65:"Cabaret", 66:"New Wave", 67:"Psychedelic", 68:"Rave", 69:"Showtunes", +70:"Trailer", 71:"Lo-Fi", 72:"Tribal", 73:"Acid Punk", 74:"Acid Jazz", +75:"Polka", 76:"Retro", 77:"Musical", 78:"Rock & Roll", 79:"Hard Rock", +80:"Folk", 81:"Folk-Rock", 82:"National Folk", 83:"Swing", 84:"Fast Fusion", +85:"Bebop", 86:"Latin", 87:"Revival", 88:"Celtic", 89:"Bluegrass", +90:"Avantgarde", 91:"Gothic Rock", 92:"Progressive Rock", +93:"Psychedelic Rock", 94:"Symphonic Rock", 95:"Slow Rock", 96:"Big Band", +97:"Chorus", 98:"Easy Listening", 99:"Acoustic", 100:"Humour", 101:"Speech", +102:"Chanson", 103:"Opera", 104:"Chamber Music", 105:"Sonata", 106:"Symphony", +107:"Booty Bass", 108:"Primus", 109:"Porn Groove", 110:"Satire", +111:"Slow Jam", 112:"Club", 113:"Tango", 114:"Samba", 115:"Folklore", +116:"Ballad", 117:"Power Ballad", 118:"Rhytmic Soul", 119:"Freestyle", +120:"Duet", 121:"Punk Rock", 122:"Drum Solo", 123:"Acapella", 124:"Euro-House", +125:"Dance Hall", 126:"Goa", 127:"Drum & Bass", 128:"Club-House", +129:"Hardcore", 130:"Terror", 131:"Indie", 132:"BritPop", 133:"Negerpunk", +134:"Polsk Punk", 135:"Beat", 136:"Christian Gangsta", 137:"Heavy Metal", +138:"Black Metal", 139:"Crossover", 140:"Contemporary Christian", +141:"Christian Rock", 142:"Merengue", 143:"Salsa", 144:"Thrash Metal", +145:"Anime", 146:"JPop", 147:"SynthPop" } + + +ID3v2FrameMap = { + "TIT1": "content group", + "TIT2": "title", + "TIT3": "subtitle", + "TALB": "album", + "TOAL": "original album", + "TRCK": "/track number/total tracks", + "TPOS": "/disc number/total discs", + "TPE1": "artist", + "TPE2": "band", + "TPE3": "conductor", + "TPE4": "interpreted by", + "TOPE": "original artist", + "TEXT": "lyrics", + "TOLY": "original lyrics", + "TCOM": "composer", + "TMCL": "musician credits", + "TIPL": "involved people", + "TENC": "encoded by", + "TBPM": "#BPM", + "TYER": "#year", +# "TLEN": "length", # unreliable, rather use Xing/FhG tags or scan the file + "TKEY": "initial key", + "TLAN": "language", + "TCON": "genre", + "TFLT": "file type", + "TMED": "media type", + "TMOO": "mood", + "TCOP": "copyright", + "TPRO": "produced", + "TPUB": "publisher", + "TOWN": "owner", + "TRSN": "station name", + "TRSO": "station owner", + "TOFN": "original file name", + "TDLY": "playlist delay", + "TDEN": "encoding time", + "TDOR": "original release time", + "TDRC": "recording time", + "TDRL": "release time", + "TDTG": "tagging time", + "TSSE": "encoding settings", + "TSOA": "album sort order", + "TSOP": "performer sort order", + "TSOT": "title sort order", + "WCOM": "commercial information URL", + "WCOP": "copyright URL", + "WOAF": "audio file URL", + "WOAR": "artist URL", + "WOAS": "audio source URL", + "WORS": "station URL", + "WPAY": "payment URL", + "WPUB": "publisher URL", + "COMM": "comment" +} + +RE_ID3v2_Frame_Type = re.compile(r'[A-Z0-9]{4}') +RE_ID3v2_Strip_Genre = re.compile(r'\([0-9]+\)(.*)') + + +################################################################################ +## ID3v1 decoder ## +################################################################################ + +def GetID3v1(f, info): + try: + f.seek(-128, 2) + data = f.read(128) + except IOError: + return 0 + if len(data)!=128 or data[:3]!="TAG": + return 0 + info['tag'] = "id3v1" + field = data[3:33].split("\0",1)[0].strip() + if field: info['title'] = unicode(field, sys.getfilesystemencoding(), 'replace') + field = data[33:63].split("\0",1)[0].strip() + if field: info['artist'] = unicode(field, sys.getfilesystemencoding(), 'replace') + field = data[63:93].split("\0",1)[0].strip() + if field: info['album'] = unicode(field, sys.getfilesystemencoding(), 'replace') + field = data[93:97].split("\0",1)[0].strip() + if field: + try: + info['year'] = int(field) + except ValueError: + pass + field = data[97:127].split("\0",1)[0].strip() + if field: info['comment'] = unicode(field, sys.getfilesystemencoding(), 'replace') + if data[125]=='\0' and data[126]!='\0': + info['track number'] = ord(data[126]) + try: + info['genre'] = ID3v1Genres[ord(data[127])] + except KeyError: + pass + return -128 + + +################################################################################ +## ID3v2 decoder ## +################################################################################ + +def DecodeInteger(s): + res = 0 + for c in s: + res = (res << 8) | ord(c) + return res + +def DecodeSyncsafeInteger(s): + res = 0 + for c in s: + res = (res << 7) | (ord(c) & 0x7F) + return res + + +def GetCharset(encoding): + if encoding=="\1": return "utf_16" + if encoding=="\2": return "utf_16_be" + if encoding=="\3": return "utf_8" + else: return "iso-8859-1" + + +def GetEndID3v2(f, offset=0): + try: + f.seek(offset-10, 2) + marker = f.read(10) + if len(marker)!=10 or marker[:3]!="3DI": + return None + size = DecodeSyncsafeInteger(marker[-4:]) + 10 + f.seek(offset-10-size, 2) + data = f.read(size) + if len(data)!=size or data[:3]!="ID3": + return None + return data + except IOError: + return None + + +def GetStartID3v2(f): + try: + f.seek(0) + marker = f.read(10) + if len(marker)!=10 or marker[:3]!="ID3": + return None + size = DecodeSyncsafeInteger(marker[-4:]) + payload = f.read(size) + if len(payload)!=size: + return None + return marker+payload + except IOError: + return None + + +def DecodeID3v2(data, info): + info['tag'] = "id3v2.%d.%d" % (ord(data[3]), ord(data[4])) + if ord(data[3]) >= 4: + decode_size = DecodeSyncsafeInteger + else: + decode_size = DecodeInteger + + # parse header flags, strip header(s) + flags = ord(data[5]) + data = data[10:] + if flags & 0x40: # extended header + size = decode_size(data[:4]) + data = data[size:] + + # parse frames + while len(data)>=10: + frame = data[:4] + if not RE_ID3v2_Frame_Type.match(frame): + break # invalid frame name or start of padding => bail out + size = decode_size(data[4:8]) + payload = data[10:size+10] + flags = ord(data[9]) + if flags & 0x02: + payload = payload.replace("\xff\0", "\xff") + if flags & 0x04: + try: + payload = zlib.decompress(payload) + except zlib.error: + continue # this frame is broken + HandleID3v2Frame(frame, payload, flags, info) + data = data[size+10:] + + +def HandleID3v2Frame(frame, payload, flags, info): + text = None + if not payload: return # empty payload + + if frame[0]=='T' and frame!="TXXX": + # text frame + charset = GetCharset(payload[0]) + text = unicode(payload[1:], charset, 'replace').split(u'\0', 1)[0] + + elif frame[0]=='W' and frame!="WXXX": + # URL + text = unicode(payload.split("\0", 1)[0], "iso-8859-1", 'replace') + + elif frame=="COMM": + # comment + charset = GetCharset(payload[0]) + lang = payload[1:4].split("\0", 1)[0] + parts = unicode(payload[4:], charset, 'replace').split(u'\0', 2) + if len(parts)<2: return # broken frame + text = parts[1] + + if text: ##### apply the current textual frame #### + key = ID3v2FrameMap.get(frame, frame) + text = text.strip() + + if frame=="TCON": # strip crappy numerical genre comment + m = RE_ID3v2_Strip_Genre.match(text.encode('iso-8859-1', 'replace')) + if m: text = m.group(1) + + if key[0]=="#": # numerical key + try: + text = int(text.strip()) + except ValueError: + return # broken frame + key = key[1:] + + if key[0]=="/": # multipart numerical key + keys = key[1:].split("/") + values = text.split("/") + for key, value in zip(keys, values): + try: + info[key] = int(value) + except: + pass + return # already done here + + info[key] = text + + +################################################################################ +## ultra-simple (and not very fault-tolerant) Ogg Vorbis metadata decoder ## +################################################################################ + +def DecodeVorbisHeader(f, info): + try: + f.seek(0) + data = f.read(4096) # almost one page, should be enough + except IOError: + return False + if data[:4]!="OggS": return False # no Ogg -- don't bother + data = data.split("vorbis", 3) + if len(data)!=4: return False # no Vorbis packets + info['format'] = "ogg" # at this point, we can assume the stream is valid + info['filetype'] = "Ogg Vorbis" + data = data[2] + if len(data)<8: return True # comment packet too short + + # encoder version + size = struct.unpack(">3) & 1 + samples = 576 * (version+1) + b2 = header[2] + bitrate = mp3_bitrates[version][b2>>4] + samplerate = mp3_samplerates[version][(b2>>2) & 3] + padding = (b2>>1) & 1 + framesize = 72000 * (version+1) * bitrate / samplerate + padding + + # skip frame data + try: + frame = f.read(framesize-4) + # accumulate the data of the first 10 frames + if total_frames < 10: + data += frame + except IOError: + break + + # fix statistics + total_samples += samples + total_frames += 1 + total_bytes += framesize + used_bitrates[bitrate] = None + + # after 10 frames, check for Xing/LAME/FhG headers + if total_frames == 10: + valid = False + # check for Xing/LAME VBR header + p2 = data.find("Xing\0\0\0") + if (p2 > 0) and (ord(data[p2 + 7]) & 1): + force_vbr = True + # check for LAME CBR header + p = data.find("Info\0\0\0") + if force_vbr or ((p > 0) and (ord(data[p + 7]) & 1)): + if force_vbr: p = p2 + total_frames, total_bytes = struct.unpack(">ii", data[p+8:p+16]) + if not(ord(data[p + 7]) & 2): + total_bytes = info['size'] # size not specified, estimate + total_samples = total_frames * samples + valid = True + # check for FhG header + else: + p = data.find("VBRI\0\1") + if p > 0: + force_vbr = True + total_bytes, total_frames = struct.unpack(">ii", data[p+10:p+18]) + total_samples = total_frames * samples + valid = True + # final sanity check + if valid: + if (total_frames < 10) or (total_bytes < 1000) or (total_bytes > info['size']): + valid = False + if valid: + # verify computed bitrate + check_bitrate = total_bytes*8*0.001/total_frames/samples*samplerate + if force_vbr: + # valid range for VBR files: all the way through + min_rate = 30.0 + max_rate = 330.0 + else: + # valid range for CBR files: current bitrate +/- 10% + min_rate = bitrate * 0.9 + max_rate = bitrate * 1.1 + valid = (check_bitrate > min_rate) and (check_bitrate < max_rate) + if valid: + break + else: + # this didn't work out, continue conventionally + total_samples = 10 * samples + total_frames = 10 + total_bytes = 0 + force_vbr = False + + # scan complete, finish things + if total_frames < 10: + return False # less than 10 frames? that's a little bit short ... + info['filetype'] = "MPEG-%d Audio Layer 3" % (2-version) + info['sample rate'] = samplerate + info['sample count'] = total_samples + info['length'] = total_samples * 1.0 / samplerate + if force_vbr or (len(used_bitrates) > 1): + info['format'] = "mp3-vbr" + info['bitrate'] = int(total_bytes*8*0.001 / info['length']) + else: + info['format'] = "mp3-cbr" + info['bitrate'] = bitrate + return True + + +################################################################################ +## MP4 decoder wrapper ## +################################################################################ + +def DecodeMP4(f, info): + try: + f.seek(0) + first_atom = f.read(8)[4:] + except IOError: + return False + if not(first_atom in ('moov', 'ftyp')): + return False # no MP4 file + try: + qt = qtparse.QTParser(f) + except IOError: + return False + info.update(qt.get_repear_info()) + del qt + return True + + +################################################################################ +## toplevel GetAudioFileInfo() function ## +################################################################################ + +def GetAudioFileInfo(filename, stat_only=False): + try: + s = os.stat(filename) + except OSError: + return None + + if not stat.S_ISREG(s[stat.ST_MODE]): + return None + info = {'path': filename, 'size':s[stat.ST_SIZE], 'mtime':s[stat.ST_MTIME]} + if stat_only: return info + + # try to extract a track number from the file name + track = 0 + for c in os.path.split(filename)[-1]: + if c in "0123456789": + track = (10 * track) + ord(c) - 48 + else: + break + if track: + info['track number'] = track + + # open the file + try: + f = file(filename, "rb") + except IOError: + return None + + # MP4 probing + if DecodeMP4(f, info): + return info + + # Ogg Vorbis probing + if DecodeVorbisHeader(f, info): + return info + + # some ID3 probing + end_offset = GetID3v1(f, info) + id3v2_data = GetEndID3v2(f, end_offset) + if id3v2_data: + DecodeID3v2(id3v2_data, info) + id3v2_data = GetStartID3v2(f) + if id3v2_data: + start_offset = len(id3v2_data) + DecodeID3v2(id3v2_data, info) + else: + start_offset = 0 + ScanMP3(f, info, start_offset) + + return info + + +################################################################################ +## a demo main function ## +################################################################################ + +if __name__=="__main__": + if len(sys.argv)<2: + print "Usage:", sys.argv[0], "..." + sys.exit(1) + for filename in sys.argv[1:]: + print + print "[%s]" % filename + info = GetAudioFileInfo(filename) + if not info: continue + keys = info.keys() + keys.sort() + fmt = "%%-%ds= %%s" % (max(map(len, keys)) + 1) + for key in keys: + value = info[key] + try: + value = value.encode('iso-8859-1', 'replace') + except: + pass + print fmt % (key, value) + print diff --git a/qtparse.py b/qtparse.py new file mode 100644 index 0000000..ecede9c --- /dev/null +++ b/qtparse.py @@ -0,0 +1,581 @@ +#!/usr/bin/env python +# +# QuickTime parser library for rePear, the iPod database management tool +# Copyright (C) 2006-2008 Martin J. Fiedler +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + +import sys, struct, types + +ID3v1Genres = { 0:"Blues", 1:"Classic Rock", 2:"Country", 3:"Dance", 4:"Disco", +5:"Funk", 6:"Grunge", 7:"Hip-Hop", 8:"Jazz", 9:"Metal", 10:"New Age", +11:"Oldies", 12:"Other", 13:"Pop", 14:"R&B", 15:"Rap", 16:"Reggae", 17:"Rock", +18:"Techno", 19:"Industrial", 20:"Alternative", 21:"Ska", 22:"Death Metal", +23:"Pranks", 24:"Soundtrack", 25:"Euro-Techno", 26:"Ambient", 27:"Trip-Hop", +28:"Vocal", 29:"Jazz&Funk", 30:"Fusion", 31:"Trance", 32:"Classical", +33:"Instrumental", 34:"Acid", 35:"House", 36:"Game", 37:"Sound Clip", +38:"Gospel", 39:"Noise", 40:"Alternative Rock", 41:"Bass", 42:"Soul", +43:"Punk", 44:"Space", 45:"Meditative", 46:"Instrumental Pop", +47:"Instrumental Rock", 48:"Ethnic", 49:"Gothic", 50:"Darkwave", +51:"Techno-Industrial", 52:"Electronic", 53:"Pop-Folk", 54:"Eurodance", +55:"Dream", 56:"Southern Rock", 57:"Comedy", 58:"Cult", 59:"Gangsta", +60:"Top 40", 61:"Christian Rap", 62:"Pop/Funk", 63:"Jungle", 64:"Native US", +65:"Cabaret", 66:"New Wave", 67:"Psychedelic", 68:"Rave", 69:"Showtunes", +70:"Trailer", 71:"Lo-Fi", 72:"Tribal", 73:"Acid Punk", 74:"Acid Jazz", +75:"Polka", 76:"Retro", 77:"Musical", 78:"Rock & Roll", 79:"Hard Rock", +80:"Folk", 81:"Folk-Rock", 82:"National Folk", 83:"Swing", 84:"Fast Fusion", +85:"Bebop", 86:"Latin", 87:"Revival", 88:"Celtic", 89:"Bluegrass", +90:"Avantgarde", 91:"Gothic Rock", 92:"Progressive Rock", +93:"Psychedelic Rock", 94:"Symphonic Rock", 95:"Slow Rock", 96:"Big Band", +97:"Chorus", 98:"Easy Listening", 99:"Acoustic", 100:"Humour", 101:"Speech", +102:"Chanson", 103:"Opera", 104:"Chamber Music", 105:"Sonata", 106:"Symphony", +107:"Booty Bass", 108:"Primus", 109:"Porn Groove", 110:"Satire", +111:"Slow Jam", 112:"Club", 113:"Tango", 114:"Samba", 115:"Folklore", +116:"Ballad", 117:"Power Ballad", 118:"Rhytmic Soul", 119:"Freestyle", +120:"Duet", 121:"Punk Rock", 122:"Drum Solo", 123:"Acapella", 124:"Euro-House", +125:"Dance Hall", 126:"Goa", 127:"Drum & Bass", 128:"Club-House", +129:"Hardcore", 130:"Terror", 131:"Indie", 132:"BritPop", 133:"Negerpunk", +134:"Polsk Punk", 135:"Beat", 136:"Christian Gangsta", 137:"Heavy Metal", +138:"Black Metal", 139:"Crossover", 140:"Contemporary Christian", +141:"Christian Rock", 142:"Merengue", 143:"Salsa", 144:"Thrash Metal", +145:"Anime", 146:"JPop", 147:"SynthPop" } + + + +QTAtomTypeMap = { + 'moov': 'container', + 'udta': 'container', + 'trak': 'container', + 'mdia': 'container', + 'minf': 'container', + 'stbl': 'container', + 'pinf': 'container', + 'schi': 'container', + 'ilst': 'container', +} + +QTTrackTypeMap = { + 'vide': 'video', + 'soun': 'audio', +} + +QTMetaDataMap = { + '$nam': ('text', 'title'), + '$alb': ('text', 'album'), + '$art': ('text', 'artist'), + '$ART': ('text', 'artist'), # FAAC writes this tag in captital letters + 'aART': ('text', 'album artist'), + '$cmt': ('text', 'comment'), + '$day': ('year', 'year'), + '$gen': ('text', 'genre'), + '$wrt': ('text', 'composer'), + '$too': ('text', 'encoder'), + 'cprt': ('text', 'copyright'), + 'trkn': ('track', None), + 'disk': ('disc', None), + 'covr': ('artwork', None), + 'cpil': ('flag', 'compilation'), + '$lyr': ('text', 'lyrics'), + 'desc': ('text', 'description'), + 'purl': ('text', 'podcast url'), + 'egid': ('text', 'episode id'), + 'catg': ('text', 'category'), + 'keyw': ('text', 'keyword'), + 'gnre': ('genre', None), +# gnre Genre 1 | 0 text | uint8 iTunes 4.0 +# tmpo BPM 21 uint8 iTunes 4.0 +# rtng Rating/Advisory 21 uint8 iTunes 4.0 +# stik ?? (stik) 21 uint8 ?? +# pcst Podcast 21 uint8 iTunes 4.9 +# tvnn TV Network Name 1 text iTunes 6.0 +# tvsh TV Show Name 1 text iTunes 6.0 +# tven TV Episode No 1 text iTunes 6.0 +# tvsn TV Season 21 uint8 iTunes 6.0 +# tves TV Episode 21 uint8 iTunes 6.0 +# pgap Gapless Play 21 uin8 iTunes 7.0 +} + +MP4DescriptorMap = { + 0x03: 'MP4ESDescr', + 0x04: 'MP4DecConfigDescr', + 0x05: 'MP4DecSpecificDescr', + 0x06: 'MP4SLConfigDescr', +} + +MP4ObjectTypeMap = { + 0x20: 'MPEG4Visual', + 0x40: 'MPEG4Audio', +} + +MP4ProfileMap = { + 1: "AAC Main", + 2: "LC-AAC", + 3: "AAC SSR", + 4: "AAC LTP", + 5: "HE-AAC", + 6: "Scalable", + 7: "TwinVQ", + 8: "CELP", + 9: "HVXC", + 12: "TTSI", + 13: "Main Synthetic Profile", + 14: "Wavetable synthesis", + 15: "General MIDI", + 16: "Algorithmic Synthesis and Audio FX", + 17: "LC-AAC with error recovery", + 19: "AAC LTP with error recovery", + 20: "AAC SSR with error recovery", + 21: "TwinVQ with error recovery", + 22: "BSAC with error recovery", + 23: "AAC LD with error recovery", + 24: "CELP with error recovery", + 25: "HXVC with error recovery", + 26: "HILN with error recovery", + 27: "Parametric with error recovery", +} + +H264ProfileMap = { + 66: "BP", + 77: "MP", + 88: "EP", + 100: "HP", + 110: "H10P", + 144: "H444P", +} + + +def chop(s): + if s: return (ord(s[0]), s[1:]) + return (0, "") + +def dictremove(d, rlist): + for r in rlist: + if r in d: + del d[r] + + +class QTParser: + def __init__(self, f, verbose=False): + self.f = f + self.verbose = verbose + self.info = {} + self.time_scale = 1 + self.tracks = {} + self.trackid = None + self.artwork = [] + self.errors = [] + + self.f.seek(0, 2) + self.info['size'] = self.f.tell() + self.parse_container(0, self.info['size']) + + def log_path(self, path, atom, size, start=None): + if not self.verbose: return + if start is None: + print "%s%s (%d bytes)" % (" " * len(path), atom, size) + else: + print "%s%s (%d bytes @ %d)" % (" " * len(path), atom, size, start) + + def err(self, path, message): + self.errors.append((self.repr_path(path), message)) + + def reject(self, path, size, minsize, need_track=True): + atom = path[-1] + if need_track: + if self.trackid is None: + return self.err(path, "%s outside of a track" % atom) + if not(self.trackid in self.tracks): + return True + if size < minsize: + return self.err(path, "atom too small") + return False + + def gettrack(self, prop, default=None): + return self.tracks[self.trackid].get(prop, default) + def settrack(self, prop, value): + self.tracks[self.trackid][prop] = value + + def repr_path(self, path): + if not path: return "" + return ".".join(path) + + def parse_container(self, start=0, size=0, path=[]): + end = start + size + while (start + 8) < end: + self.f.seek(start) + head = self.f.read(8) + start += 8 + size = struct.unpack(">L", head[:4])[0] - 8 + if size < 0: + return self.err(path, "invalid sub-atom size") + atom = head[4:].strip("\0 ").replace('\xa9', '$') + if not atom: + break + self.log_path(path, atom, size, start) + if atom in QTMetaDataMap: + alias = 'container' + else: + alias = QTAtomTypeMap.get(atom, atom) + try: + parser = getattr(self, "parse_" + alias) + except AttributeError: + parser = None + if parser: + parser(start, min(size, end - start), path + [atom]) + start += size + if start < end: + return self.err(path, "%d orphaned bytes" % (end - start)) + if start > end: + return self.err(path, "%d missing bytes" % (start - end)) + + def parse_trak(self, start, size, path): + self.track = None + self.parse_container(start, size, path) + self.track = None + + def parse_mvhd(self, start, size, path): + if self.reject(path, size, 20, False): return + data = self.f.read(20) + self.time_scale, length = struct.unpack(">LL", data[12:]) + self.info['length'] = float(length) / self.time_scale + + def parse_tkhd(self, start, size, path): + if self.reject(path, size, 24, False): return + data = self.f.read(min(size, 84)) + self.trackid, dummy, length = struct.unpack(">LLL", data[12:24]) + if not self.trackid in self.tracks: + self.tracks[self.trackid] = {} + self.settrack('length', float(length) / self.time_scale) + if len(data) >= 84: + w, h = struct.unpack(">LL", data[76:84]) + self.settrack('width', w >> 16) + self.settrack('height', h >> 16) + + def parse_mdhd(self, start, size, path): + if self.reject(path, size, 20): return + data = self.f.read(20) + time_scale, length = struct.unpack(">LL", data[12:]) + self.settrack('length', float(length) / time_scale) + + def parse_hdlr(self, start, size, path): + if 'udta' in path: + return + if self.reject(path, size, 12): return + data = self.f.read(12) + try: + self.tracks[self.trackid]['type'] = QTTrackTypeMap[data[8:]] + except KeyError: + del self.tracks[self.trackid] + + def parse_stsd(self, start, size, path): + if self.reject(path, size, 8): return + data = self.f.read(8) + count = struct.unpack(">L", data[4:8])[0] + end = start + size + start += 8 + media_type = self.gettrack('type') + for i in xrange(count): + if start > (end - 16): + return self.err(path, "description #%d too small" % (i+1)) + self.f.seek(start) + data = self.f.read(16) + start += 16 + size = struct.unpack(">L", data[:4])[0] - 16 + format = data[4:8].strip("\0 ") + refidx = struct.unpack(">H", data[14:])[0] + self.log_path(path, format, size, start) + try: + parser = getattr(self, "parse_stsd_" + media_type) + except KeyError: + if not i: self.err(path, "descriptions found, but no handler defined") + parser = None + except AttributeError: + parser = None + if parser: + self.settrack('format', format) + parser(start, min(size, end - start), path + [format]) + start += size + + def parse_stsd_audio(self, start, size, path): + if self.reject(path, size, 20): return + data = self.f.read(20) + version, rev, ven, chan, res, compid, packsize, rate_hi, rate_lo = \ + struct.unpack(">HHLHHHHHH", data) + if version == 0: + hsize = 20 + elif version == 1: + hsize = 24 + else: + return self.err(path, "unknown audio stream description version") + if size < hsize: + return self.err(path, "stream description too small") + start += hsize + size -= hsize + self.settrack('channels', chan) + self.settrack('bits per sample', res) + self.settrack('sample rate', rate_hi) + if self.gettrack('length'): + self.settrack('sample count', int(rate_hi * self.gettrack('length') + 0.5)) + self.parse_container(start, size, path) + + def parse_stsd_video(self, start, size, path): + if self.reject(path, size, 70): return + version, rev, ven, tq, sq, w, h, hres, vres, zero, frames = \ + struct.unpack(">HHLLLHHLLLH", self.f.read(34)) + if (w != self.gettrack('width')) or (h != self.gettrack('height')): + self.err(path, "video size doesn't match track header value") + data = self.f.read(32) + clen = ord(data[0]) + if clen > 31: + self.err(path, "invalid compressor name length") + elif clen: + self.settrack('compressor', data[1:clen+1]) + self.parse_container(start + 70, size - 70, path) + + def parse_avcC(self, start, size, path): + if self.reject(path, size, 4): return + data = self.f.read(4) + profile = H264ProfileMap.get(ord(data[1]), None) + level = ord(data[3]) + if level % 10: + level = "%d.%d" % (level / 10, level % 10) + else: + level = str(level / 10) + if (level == "1.1") and (ord(data[2]) & 0x10): + level = "1b" + format = "H.264" + if profile: format += " " + profile + self.settrack('video format', format + "@L" + level) + + def parse_esds(self, start, size, path): + try: + if not(path[-2] in ('mp4a', 'mp4v')): + return # unknown format, ignore it + except IndexError: + return self.err(path, "esds atom found at root level") + if self.reject(path, size, 4, False): return + self.f.seek(start + 4) + self.parse_mp4desc(path, self.f.read(size - 4)) + + def parse_mp4desc(self, path, data): + while data: + tag, data = chop(data) + size = 0 + while True: + if not data: + return self.err(path, "descriptor ends while decoding length") + byte, data = chop(data) + size = (size << 7) | (byte & 0x7F) + if not(byte & 0x80): break + if size > len(data): + self.err(path, "%d missing bytes in descriptor" % (size - len(data))) + size = len(data) + tag = MP4DescriptorMap.get(tag, "0x%02X" % tag) + self.log_path(path, tag, size) + try: + parser = getattr(self, "parse_" + tag) + except AttributeError: + parser = None + if parser: + parser(path + [tag], data[:size]) + data = data[size:] + + def parse_MP4ESDescr(self, path, data): + if self.reject(path, len(data), 3, False): return + esid, flags = struct.unpack(">BH", data[:3]) + data = data[3:] + if flags & 0x80: # stream_dependence + if self.reject(path, len(data), 2, False): return + data = data[2:] + if flags & 0x40: # URL + if self.reject(path, len(data), 1, False): return + size, data = chop(data) + if self.reject(path, len(data), size, False): return + data = data[size:] + if flags & 0x20: # ocr_stream + if self.reject(path, len(data), 2, False): return + data = data[2:] + self.parse_mp4desc(path, data) + + def parse_MP4DecConfigDescr(self, path, data): + if self.reject(path, len(data), 13, False): return + otid, flags, buf_hi, buf_lo, rate_max, rate_avg = \ + struct.unpack(">BBBHLL", data[:13]) + self.settrack('bitrate', int(rate_avg / 1000)) + objtype = MP4ObjectTypeMap.get(otid, None) + if not objtype: return # some unknown format + self.parse_mp4desc(path + [objtype], data[13:]) + + def parse_MP4DecSpecificDescr(self, path, data): + try: + parser = getattr(self, "parse_MP4DecSpecificDescr_" + path[-2]) + except AttributeError: + return # unknown format + except IndexError: + return self.err("internal error") + parser(path, data) + + def parse_MP4DecSpecificDescr_MPEG4Audio(self, path, data): + if self.reject(path, len(data), 2, False): return + a, data = chop(data) + b, data = chop(data) + profile = (a >> 3) & 0x1F + freq = ((a << 1) | (b >> 7)) & 0x0F + try: + self.settrack('filetype', "MPEG-4 " + MP4ProfileMap[profile]) + except KeyError: + pass + if freq == 15: + if self.reject(path, len(data), 3, False): return + freq = b & 0x7F + b, data = chop(data) + freq = (freq << 7) | b + b, data = chop(data) + freq = (freq << 7) | b + b, data = chop(data) + freq = (freq << 1) | (b >> 7) + else: + try: + freq = (96000, 88200, 64000, 48000, 44100, 32000, 24000, 22050, 16000, 12000, 11025, 8000, 7350)[freq] + except IndexError: + self.err(path, "invalid sampling rate code %d" % freq) + freq = 0 + if freq: + ref = self.gettrack('sample rate') + if not ref: + self.settrack('sample rate', freq) + elif freq != ref: + self.err(path, "sample rate in AAC descriptor (%d) doesn't match sample rate in stream description (%d)" % (freq, ref)) + if data: + return self.err(path, "descriptor is longer than expected") + + def parse_MP4DecSpecificDescr_MPEG4Visual(self, path, data): + self.settrack('video format', 'MPEG-4 ASP') + + def parse_meta(self, start, size, path): + self.parse_container(start + 4, size - 4, path) + + def parse_data(self, start, size, path): + if self.reject(path, size, 8, False): return + try: + parser, key = QTMetaDataMap[path[-2]] + except IndexError: + return self.err(path, "data atom found at root level") + except KeyError: + return self.err(path, "no parser defined for this atom") + format = struct.unpack(">L", self.f.read(8)[:4])[0] & 0x00FFFFFF + try: + parser = getattr(self, "format_" + parser) + except AttributeError: + return self.err(path, "format parser `%s' doesn't exist" % parser) + res = parser(path, self.f.read(size - 8)) + if key: + if res is None: + return self.err(path, "decoding failed, no value assigned") + self.info[key] = res + + def format_text(self, path, data): + data = data.strip("\0") + if data.startswith("\xfeff"): + return unicode(data, 'utf_16') + else: + return unicode(data, 'utf_8') + + def format_year(self, path, data): + data = data.strip("\0").split('-', 1)[0] + try: + return int(data) + except ValueError: + return self.err(path, "invalid date format") + + def format_byte(self, path, data): + if not data: + return self.err(path, "zero-length data block") + return ord(data[0]) + + def format_genre(self, path, data): + if not data: + return self.err(path, "zero-length data block") + genre = ID3v1Genres.get(ord(data[-1]) - 1, None) + if genre: self.info["genre"] = genre + + def format_track(self, path, data, item='track'): + if self.reject(path, len(data), 6, False): return + current, total = struct.unpack(">HH", data[2:6]) + if current: self.info["%s number" % item] = current + if total: self.info["total %ss" % item] = total + + def format_disc(self, path, data): + return self.format_track(path, data, 'disc') + + def format_artwork(self, path, data): + self.artwork.append(data) + + def format_flag(self, path, data): + return len(data.strip("\0")) != 0 + + def get_repear_info(self): + info = {} + have_video = False + have_audio = False + for track in self.tracks.itervalues(): + ttype = track.get('type', '?') + if not(have_audio) and (ttype == 'audio'): + ainfo = track.copy() + dictremove(ainfo, ('type', 'width', 'height')) + info.update(ainfo) + have_audio = True + if not(have_video) and (ttype == 'video'): + vinfo = track.copy() + dictremove(vinfo, ('type',)) + if have_audio: dictremove(vinfo, ('format', )) + info.update(vinfo) + have_video = True + info.update(self.info) + if ('album artist' in info) and not('artist' in info): + info['artist'] = info['album artist'] + if have_video: + info['filetype'] = "MPEG-4 Video file" + elif not('filetype' in info): + info['filetype'] = "MPEG-4 Audio File" + return info + +################################################################################ + +def dump_dict(d): + keys = d.keys() + keys.sort() + for key in keys: + print " %s = %s" % (key, repr(d[key])) + +if __name__ == "__main__": + qt = QTParser(file(sys.argv[1], "rb"), True) + print + + print "Raw file information:" + dump_dict(qt.info) + for track in qt.tracks: + print "Raw track information (id %s):" % track + dump_dict(qt.tracks[track]) + print + + print "rePear-compliant information:" + dump_dict(qt.get_repear_info()) + print + + if qt.errors: + print "Errors:" + for e in qt.errors: print " %s: %s" % e + print diff --git a/repear.py b/repear.py new file mode 100644 index 0000000..ee723b6 --- /dev/null +++ b/repear.py @@ -0,0 +1,2015 @@ +#!/usr/bin/env python +# +# rePear, the iPod database management tool +# Copyright (C) 2006-2008 Martin J. Fiedler +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + +__title__ = "rePear" +__version__ = "0.4.1" +__author__ = "Martin J. Fiedler" +__email__ = "martin.fiedler@gmx.net" +banner = "Welcome to %s, version %s" % (__title__, __version__) + +""" +TODO: preserve .m3u playlists on update + +0.4.1: + - added artwork formats for nano 4G + - added support for the 'mhii link' field, required for artwork on nano 4G + +0.4.0: + - added command-line options to override master playlist and scrobble config + file names (either relative to the root directory or relative to the + working directory from which rePear is being run) + - fixed crash bug for rare broken ID3v2 tags + - added 'help' action + +0.4.0-rc2: + - fixed crash after scrobbling + +0.4.0-rc1: + - added configuration actions + - root directory auto-detection now checks current working directory, too + - fixed time calculations (required for proper scrobbling) + - fixed artwork processing if an artwork file is broken + - fixed broken sort function + +0.4.0-beta1: + - added support for 2007 models (nano 3G, classic) + - added support for MPEG-4 audio files + - added experimental support for MPEG-4 video files + - added Play Counts import to update play and skip counts and ratings + - added last.fm scrobble support + - added 'update' action + - added playlist sort functionality + - added global playlist option "skip album playlists = no" to disable + pruning of album playlists + - added global playlist option "directory playlists = yes" to turn every + directory into a playlist + - fixed iTunesDB parser so it reads post-iTunes 7.1 files + - sped up MP3 parser by using Xing/LAME or FhG info tags, where available + - ^C menu asks whether to skip a single track or completely cancel freezing + - added --nowait option to bypass Win32 keypress waiting on quit + - made the freeze process much more error robust -- I/O errors in single + files won't stop the whole process any longer + - added crash handler + - fixed crash in the directory playlist construction code for Ogg->MP3 + transcoded files + - fixed ID3v2.3.0 parser (thanks to Ian Camaclang for the patch!) + - added msvcr71.dll to Win32 distribution + - model list in help screen is now sorted + - unfreezing empty databases now works + - improved mtime comparison + - not importing dot-files and directories any longer + +0.3.0: + - added playlist support -- two methods are availabe: + - the "master playlist file" in /repear_playlists.ini + - every *.m3u is collected and converted to a playlist, unless it exactly + covers an album (in which case it would be pointless) + - added Balanced Shuffle feature + - MP3 detection code is now more error-tolerant (doesn't clip the file at the + first broken frame any longer) + - added automatic inference of the compilation flag: if the album tag of all + files in a directory is the same, but the artists differ, the whole + directory will be marked as a compilation + - fixed crash when OggDec was not present + - added a filename allocator; should improve big (>10GB) iPod compatibility + - now guessing the track number from the file name even if there is an ID3v1.0 + tag available + - when freezing, the cache file is now saved as early as possible to minimize + data loss if rePear crashes at a later point (e.g. playlist processing) + +0.2.2: + - fixed endianness issue + - filename-based metadata guessing now includes track numbers + +0.2.1: + - fixed album sort order + +0.2.0: + - Artwork support + - limited automatic pathfinding on Windows systems: rePear needs to be + installed somewhere on the iPod volume, but not necessarily in the root + directory + +0.1.1: + - make dissect less destructive (keep filenames) + - accept all full-hour time differences + - auto-create an iTunesDB backup + - create iTunesSD et al. -> iPod shuffle support + - automatic transcoding of Ogg Vorbis tracks + - some bugfixes +""" + + + +DISSECT_BASE_DIR = "Dissected Tracks/" +DIRECTORY_COUNT = 10 +DEFAULT_LAME_OPTS = "--quiet -h -V 5" +MASTER_PLAYLIST_FILE = "repear_playlists.ini" +SCROBBLE_CONFIG_FILE = "repear_scrobble.ini" +SUPPORTED_FILE_FORMATS = (".mp3", ".ogg", ".m4a", ".m4b", ".mp4") +MUSIC_DIR = "iPod_Control/Music/" +CONTROL_DIR = "iPod_Control/iTunes/" +ARTWORK_DIR = "iPod_Control/Artwork/" +DB_FILE = CONTROL_DIR + "iTunesDB" +CACHE_FILE = CONTROL_DIR + "repear.cache" +MODEL_FILE = CONTROL_DIR + "repear.model" +FWID_FILE = CONTROL_DIR + "fwid" +SCROBBLE_QUEUE_FILE = CONTROL_DIR + "repear.scrobble_queue" +ARTWORK_CACHE_FILE = ARTWORK_DIR + "repear.artwork_cache" +ARTWORK_DB_FILE = ARTWORK_DIR + "ArtworkDB" +def OLDNAME(x): return x.replace("repear", "retune") + +import sys, optparse, os, fnmatch, stat, string, time, types, cPickle, random +import re, warnings, traceback, getpass, md5 +warnings.filterwarnings('ignore', category=RuntimeWarning) # for os.tempnam() +import iTunesDB, mp3info, hash58, scrobble +Options = {} + + +################################################################################ +## Some internal management functions ## +################################################################################ + +broken_log = False +homedir = "" + +def open_log(): + global logfile + Options['log'] = os.path.abspath(Options['log']) + try: + logfile = open(Options['log'], "w") + except IOError: + logfile = None + +def log(line, flush=True): + global logfile + sys.stdout.write(line) + if flush: sys.stdout.flush() + if logfile: + try: + logfile.write(line) + if flush: logfile.flush() + except IOError: + broken_log = True +iTunesDB.log = log + +def quit(code=1): + global logfile, broken_log + if logfile: + try: + logfile.close() + except IOError: + broken_log = True + logfile = None + log("\nLog written to `%s'\n" % Options['log']) + if broken_log: + log("WARNING: there were errors while writing the log file\n") + if not Options.get('nowait', True): # Windows: wait for keypress + log("Press ENTER to close this window. ", True) + try: + raw_input() + except (IOError, EOFError, KeyboardInterrupt): + pass # I don't care at this point, we're going to leave anyway + sys.exit(code) + +def fatal(line): + log("FATAL: %s\n" % line) + quit() + +def confirm(prompt): + sys.stdout.write("%sDo you really want to continue? (y/N) " % prompt) + sys.stdout.flush() + try: + answer = raw_input() + except (IOError, EOFError, KeyboardInterrupt): + answer = "" + if answer.strip().lower() in ("y", "yes"): + return + log("Action aborted by user.\n") + quit() + + + +def goto_root_dir(): + global homedir + homedir = os.path.abspath(os.path.split(sys.argv[0])[0]).replace("\\", "/") + if homedir[-1] != '/': homedir += '/' + if Options['root']: + rootdir = Options['root'].replace("\\", "/") + if rootdir[-1] != '/': rootdir += '/' + else: + # no root directory specified -- try the current directory + rootdir = os.getcwd().replace("\\", "/") + if rootdir[-1] != '/': rootdir += '/' + if not os.path.isfile(rootdir + "iPod_Control/iTunes/iTunesDB"): + # not found? then try the executable's directory + rootdir = homedir + # special case on Windows: if the current directory doesn't contain + # a valid iPod directory structure, reduce the pathname to the first + # three characters, as in 'X:/', which is usually the root directory + if (os.name == 'nt') and not(os.path.isfile(rootdir + DB_FILE)): + rootdir = rootdir[:3] + + if os.path.isfile(rootdir + DB_FILE): + log("iPod root directory is `%s'\n" % rootdir) + else: + fatal("root directory `%s' contains no iPod database" % rootdir) + + try: + os.chdir(rootdir) + except OSError, e: + fatal("can't change to the iPod root directory: %s" % e.strerror) + + + +def load_cache(return_on_error=None): + try: + f = open(CACHE_FILE, "rb") + except IOError: + try: + f = open(OLDNAME(CACHE_FILE), "rb") + except IOError: + return return_on_error + try: + content = cPickle.load(f) + f.close() + except (IOError, EOFError, cPickle.PickleError): + return return_on_error + return content + +def save_cache(content=None): + try: + f = open(CACHE_FILE, "wb") + cPickle.dump(content, f) + f.close() + delete(OLDNAME(CACHE_FILE), True) + except (IOError, EOFError, cPickle.PickleError): + log("ERROR: can't save the rePear cache\n") + + +def execute(program, args): + global homedir + if os.name == "nt": + spawn = os.spawnv + path = homedir + program + ".exe" + args = ["\"%s\"" % arg for arg in args] + else: + spawn = os.spawnvp + path = program + try: + return spawn(os.P_WAIT, path, [program] + args) + except OSError, e: + log("ERROR: can't execute %s: %s\n" % (program, e.strerror)) + except KeyboardInterrupt: + return -2 + + + + + +################################################################################ +## Some generic tool functions ## +################################################################################ + +def printable(x, kill_chars=""): + if type(x)==types.UnicodeType: + x = x.encode(sys.getfilesystemencoding(), 'replace') + x = str(x) + for c in kill_chars: + x = x.replace(c, "_") + return x + + +def move_file(src, dest): + # check if source file exists + if not os.path.isfile(src): + log("[FAILED]\nERROR: source file `%s' doesn't exist\n" % + printable(src), True) + return 'missing' + + # don't clobber files (wouldn't work on Windows anyway) + if os.path.isfile(dest): + log("[FAILED]\nERROR: destination file `%s' already exists\n" % + printable(dest), True) + return 'exists' + + # create parent directories if necessary + dest_dir = os.path.split(dest)[0] + if dest_dir and not(os.path.isdir(dest_dir)): + try: + os.makedirs(dest_dir) + except OSError, e: + log("[FAILED]\nERROR: can't create destination directory `%s': %s\n" % + (printable(dest_dir), e.strerror), True) + return 'mkdir' + + # finally rename it + try: + os.rename(src, dest) + except OSError, e: + log(" [FAILED]\nERROR: can't move `%s' to `%s': %s\n" % + (printable(src), printable(dest), e.strerror), True) + return 'move' + log("[OK]\n", True) + return None + + +def backup(filename): + dest = "%s.repear_backup" % filename + if os.path.exists(dest): return + try: + os.rename(filename, dest) + return True + except OSError, e: + log("WARNING: Cannot backup `%s': %s\n" % (filename, e.strerror)) + return False + + +def delete(filename, may_fail=False): + if not os.path.exists(filename): return + try: + os.remove(filename) + return True + except OSError, e: + if not may_fail: + log("ERROR: Cannot delete `%s': %s\n" % (filename, e.strerror)) + return False + + +class ExceptionLogHelper: + def write(self, s): + log(s) +Logger = ExceptionLogHelper() + + +# path and file name sorting routines +re_digit = re.compile(r'(\d+)') +def tryint(s): + try: return int(s) + except ValueError: return s.lower() +def fnrep(fn): + return tuple(map(tryint, re_digit.split(fn))) +def fncmp(a, b): + return cmp(fnrep(a), fnrep(b)) +def pathcmp(a, b): + a = a.split(u'/') + b = b.split(u'/') + # compare base directories + for i in xrange(min(len(a), len(b)) - 1): + r = fncmp(a[i], b[i]) + if r: return r + # subdirectories first + r = len(b) - len(a) + if r: return r + # finally, compare leaf file name + return fncmp(a[-1], b[-1]) +def trackcmp(a, b): + return pathcmp(a.get('original path', None) or a.get('path', '???'), \ + b.get('original path', None) or b.get('path', '???')) + + +################################################################################ +## Filename Allocator ## +################################################################################ + +class Allocator: + def __init__(self, root, files_per_dir=100, max_dirs=100): + self.root = root + self.files_per_dir = files_per_dir + self.max_dirs = max_dirs + self.names = {} + self.files = {} + digits = [] + digits = [] + try: + dirs = os.listdir(root) + except OSError: + os.mkdir(root) + dirs = [] + for elem in dirs: + try: + index = self.getindex(elem) + except ValueError: + continue + self.names[index] = elem + self.files[index] = self.scandir(os.path.join(root, elem)) + digits.append(len(elem) - 1) + if digits: + digits.sort() + self.fmt = "F%%0%dd" % (digits[len(digits) / 2]) + else: + self.fmt = "F%02d" + if not self.files: + self.mkdir(0) + self.current_dir = min(self.files.iterkeys()) + + def getindex(self, name): + if not name: raise ValueError + if name[0].upper() != 'F': raise ValueError + return int(name[1:], 10) + + def scandir(self, root): + try: + dir_contents = os.listdir(root) + except OSError: + return [] + dir_contents = [os.path.splitext(x)[0].upper() for x in dir_contents if x[0] != '.'] + return dict(zip(dir_contents, [None] * len(dir_contents))) + + def __len__(self): + return sum(map(len, self.files.itervalues())) + + def __repr__(self): + return "" % (len(self), len(self.files)) + + def allocate_ex(self, index): + while True: + name = "".join([random.choice(string.ascii_uppercase) for x in range(4)]) + if not(name in self.files[index]): + break + self.files[index][name] = None + return self.names[index] + '/' + name + + def mkdir(self, index): + if index in self.files: + return + name = self.fmt % index + try: + os.mkdir(os.path.join(self.root, name)) + except OSError: + pass + self.names[index] = name + self.files[index] = {} + + def allocate(self): + count, index = min([(len(d[1]), d[0]) for d in self.files.iteritems()]) + # need to allocate a new directory + if (count >= self.files_per_dir) and (len(self.files) < self.max_dirs): + available = [i for i in range(self.max_dirs) if not i in self.files] + index = available[0] + self.mkdir(index) + # generate a file name + while True: + name = "".join([random.choice(string.ascii_uppercase) for x in range(4)]) + if not(name in self.files[index]): + break + self.files[index][name] = None + return self.root + '/' + self.names[index] + '/' + name + + def add(self, fullname): + try: + dirname, filename = fullname.split('/')[-2:] + index = self.getindex(dirname) + except ValueError: + return + filename = os.path.splitext(filename)[0] + if not index in self.files: + self.names[index] = dirname + self.files[index] = {} + self.files[index][filename] = None + + +################################################################################ +## Balanced Shuffle ## +################################################################################ + +class BalancedShuffle: + def __init__(self): + self.root = { None: [] } + + def add(self, path, data): + if type(path) == types.UnicodeType: + path = path.encode('ascii', 'replace') + path = path.replace("\\", "/").lower().split("/") + if path and not(path[0]): + path.pop(0) + if not path: + return # broken path + root = self.root + while True: + if len(path) == 1: + # tail reached + root[None].append(data) + break + component = path.pop(0) + if not component in root: + root[component] = { None: [] } + root = root[component] + + def shuffle(self, root=None): + if not root: + root = self.root + + # shuffle the files of the root node + random.shuffle(root[None]) + + # build a list of directories to shuffle + subdirs = filter(None, [root[None]] + \ + [self.shuffle(root[key]) for key in root if key]) + + # check for "tail" cases + if not subdirs: + return [] + if len(subdirs) == 1: + return subdirs[0] + + # pad subdirectory list to a common length + dircount = len(subdirs) + maxlen = max(map(len, subdirs)) + subdirs = [self.fill(sd, maxlen) for sd in subdirs] + + # collect all items + res = [] + last = -1 + for i in xrange(maxlen): + # determine the directory order for this "column" + order = range(dircount) + random.shuffle(order) + if (len(order) > 1) and (order[0] == last): + order.append(order.pop(0)) + while len(order) > 1: # = if len(order) > 1: while True: + random.shuffle(order) + if last != order[0]: break + last = order[-1] + + # produce a result + res.extend(filter(lambda x: x is not None, \ + [subdirs[j][i] for j in order])) + return res + + def fill(self, data, total): + ones = len(data) + invert = (ones > (total / 2)) + if invert: + ones = total - ones + bitmap = [0] * total + remain = total + for fraction in xrange(ones, 0, -1): + bitmap[total - remain] = 1 + skip = float(remain) / fraction + skip = random.randrange(int(0.9 * skip), int(1.1 * skip) + 2) + remain -= min(max(1, skip), remain - fraction + 1) + if invert: + bitmap = [1-x for x in bitmap] + offset = random.randrange(0, total) + bitmap = bitmap[offset:] + bitmap[:offset] + def decide(x): + if x: return data.pop(0) + return None + return map(decide, bitmap) + + +################################################################################ +## Play Counts import and Scrobbling ## +################################################################################ + +def ImportPlayCounts(cache, index, scrobbler=None): + log("Updating play counts and ratings ... ", True) + + # open Play Counts file + try: + pc = iTunesDB.PlayCountsReader() + except IOError: + log("\n0 track(s) updated.\n") + return False + except iTunesDB.InvalidFormat: + log("\n-- Error in Play Counts file, import failed.\n") + return False + + # parse old iTunesDB + try: + db = iTunesDB.DatabaseReader() + files = [printable(item.get('path', u'??')[1:].replace(u':', u'/')).lower() for item in db] + db.f.close() + del db + except (IOError, iTunesDB.InvalidFormat): + log("\n-- Error in iTunesDB, import failed.\n") + return False + + # plausability check + if len(files) != pc.entry_count: + log("\n-- Mismatch between iTunesDB and Play Counts file, import failed.\n") + return False + + # walk through Play Counts file + update_count = 0 + try: + for item in pc: + path = files[item.index] + try: + track = cache[index[path]] + except (KeyError, IndexError): + continue + updated = False + if item.play_count: + track['play count'] = track.get('play count', 0) + item.play_count + updated = True + if item.last_played: + track['last played time'] = item.last_played + updated = True + if item.skip_count: + track['skip count'] = track.get('skip count', 0) + item.skip_count + updated = True + if item.last_skipped: + track['last skipped time'] = item.last_skipped + updated = True + if item.bookmark: + track['bookmark time'] = item.bookmark * 0.001 + updated = True + if item.rating: + track['rating'] = item.rating + updated = True + if updated: + update_count += 1 + if item.play_count and scrobbler: + scrobbler += track + pc.f.close() + del pc + except (IOError, iTunesDB.InvalidFormat): + log("\n-- Error in Play Counts file, import failed.\n") + return False + log("%d track(s) updated.\n" % update_count) + return update_count + + +################################################################################ +## DISSECT action ## +################################################################################ + +def Dissect(): + state, cache = load_cache((None, None)) + + if (state is not None) and not(Options['force']): + if state=="frozen": confirm(""" +WARNING: This action will put all the music files on your iPod into a completely +new directory structure. All previous file and directory names will be lost. +This also means that any iTunesDB backups you have will NOT work any longer! +""") + if state=="unfrozen": confirm(""" +WARNING: The database is currently unfrozen, so the following operations will +almost completely fail. +""") + + cache = [] + try: + db = iTunesDB.DatabaseReader() + + for info in db: + if not info.get('path', None): + log("ERROR: track lacks path attribute\n") + continue + src = printable(info['path'])[1:].replace(":", "/") + if not os.path.isfile(src): + log("ERROR: file `%s' is found in database, but doesn't exist\n" % src) + continue + if not info.get('title', None): + info.update(iTunesDB.GuessTitleAndArtist(info['path'])) + ext = os.path.splitext(src)[1] + base = DISSECT_BASE_DIR + if info.get('artist', None): + base += printable(info['artist'], "<>/\\:|?*\"") + '/' + if info.get('album', None): + base += printable(info['album'], "<>/\\:|?*\"") + '/' + if info.get('track number', None): + base += "%02d - " % info['track number'] + base += printable(info['title'], "<>/\\:|?*\"") + + # move the file, but avoid filename collisions + serial = 1 + dest = base + ext + while os.path.exists(dest): + serial += 1 + dest = base + " (%d)"%serial + ext + log("%s => %s " % (src, dest), True) + if move_file(src, dest): + continue # move failed + + # create a placeholder cache entry + cache.append({ + 'path': src, + 'original path': unicode(dest, sys.getfilesystemencoding(), 'replace') + }) + except IOError: + fatal("can't read iTunes database file") + except iTunesDB.InvalidFormat: + raise + fatal("invalid iTunes database format") + + # clear the cache + save_cache(("unfrozen", cache)) + + + +################################################################################ +## FREEZE utilities ## +################################################################################ + +g_freeze_error_count = 0 + +def check_file(base, fn): + if fn.startswith('.'): + return None # skip dot-files and -directories + key, ext = [component.lower() for component in os.path.splitext(fn)] + fullname = base + fn + try: + s = os.stat(fullname) + except OSError: + log("ERROR: directory entry `%s' is inaccessible\n" % fn) + return None + isfile = int(not(stat.S_ISDIR(s[stat.ST_MODE]))) + if isfile and not(stat.S_ISREG(s[stat.ST_MODE])): + return None # no directory and no normal file -> skip this crap + if not(isfile) and (fullname=="iPod_Control" or fullname=="iPod_Control/Music"): + isfile = -1 # trick the sort algorithm to move iPC/Music to front + return (isfile, fnrep(fn), fullname, s, ext, key) + + +def make_cache_index(cache): + index = {} + for i in xrange(len(cache)): + for path in [cache[i][f] for f in ('path', 'original path') if f in cache[i]]: + key = printable(path).lower() + if key in index: + log("ERROR: `%s' is cached multiple times\n" % printable(path)) + else: + index[key] = i + return index + + +def find_in_cache(cache, index, path, s): + i = index.get(printable(path).lower(), None) + if i is None: + return (False, None) # not found + info = cache[i] + + # check size and modification time + if info.get('size', None) != s[stat.ST_SIZE]: + return (False, info) # mismatch + if not iTunesDB.compare_mtime(info.get('mtime', 0), s[stat.ST_MTIME]): + return (False, info) # mismatch + + # all checks passed => correct file + return (True, info) + + +def move_music(src, dest, info): + global g_freeze_error_count + format = info.get('format', "mp3-cbr") + if format == "ogg": + src = printable(src) + dest = os.path.splitext(printable(dest))[0] + ".mp3" + tmp = os.tempnam(None, "repear") + ".wav" + + # generate new source filename (replace .ogg by .mp3) + newsrc = info.get('original path', src) + if type(newsrc) != types.UnicodeType: + newsrc = unicode(newsrc, sys.getfilesystemencoding(), 'replace') + newsrc = u'.'.join(newsrc.split(u'.')[:-1]) + u'.mp3' + + # decode the Ogg file + res = execute("oggdec", ["-Q", "-o", tmp, src]) + if res != 0: + g_freeze_error_count += 1 + log("[FAILED]\nERROR: cannot execute OggDec ... result '%s'\n" % res) + delete(tmp, may_fail=True) + return None + else: + log("[decoded] ", True) + + # build LAME option list + lameopts = Options['lameopts'].split(' ') + for key, optn in (('title','tt'), ('artist','ta'), ('album','tl'), ('year','ty'), ('comment','tc'), ('track number','tn')): + if key in info: + lameopts.extend(["--"+optn, printable(info[key])]) + if 'genre' in info: + ref_genre = printable(info['genre']).lower().replace(" ","") + for number, genre in mp3info.ID3v1Genres.iteritems(): + if genre.lower().replace(" ","") == ref_genre: + lameopts.extend(["--tg", str(number)]) + break + + # encode to MP3 + res = execute("lame", lameopts + [tmp, dest]) + delete(tmp) + if res != 0: + g_freeze_error_count += 1 + log("[FAILED]\nERROR: cannot execute LAME ... result code %d\n" % res) + return None + else: + log("[encoded] ", True) + + # check the resulting file + info = mp3info.GetAudioFileInfo(dest) + if not info: + g_freeze_error_count += 1 + log("[FAILED]\nERROR: generated MP3 file is invalid\n") + delete(dest) + return None + delete(src) + info['original path'] = newsrc + info['changed'] = 2 + log("[OK]\n", True) + return info + + else: # no Ogg file -> move directly + if move_file(src, dest): + g_freeze_error_count += 1 + return None # failed + else: + return info + + +def freeze_dir(cache, index, allocator, playlists=[], base="", artwork=None): + global g_freeze_error_count + try: + flist = filter(None, [check_file(base, fn) for fn in os.listdir(base or ".")]) + except KeyboardInterrupt: + raise + except: + g_freeze_error_count += 1 + log(base + "/\n" + " runtime error, traceback follows ".center(79, '-') + "\n") + traceback.print_exc(file=Logger) + log(79*'-' + "\n") + return [] + + # generate directory list + directories = filter(lambda x: x[0] < 1, flist) + directories.sort() + + # add playlist files + playlists.extend([x[2] for x in flist if (x[0] > 0) and (x[4] == ".m3u")]) + + # generate music file list + music = filter(lambda x: (x[0] > 0) and (x[4] in SUPPORTED_FILE_FORMATS), flist) + music.sort() + + # if there are no subdirs and no music files here, prune this directory + if not(directories) and not(music): + return [] + + # generate name -> artwork file associations + image_assoc = dict([(x[5], x[2]) for x in flist if (x[0] > 0) and (x[4] in (".jpg", ".png"))]) + + # find artwork files that are not associated to a file or directory + unassoc_images = image_assoc.copy() + for d0,d1,d2,d3,d4,key in directories: + if key in unassoc_images: + del unassoc_images[key] + for d0,d1,d2,d3,d4,key in music: + if key in unassoc_images: + del unassoc_images[key] + unassoc_images = unassoc_images.values() + unassoc_images.sort() + + # use one of the unassociated artwork files as this directory's artwork, + # unless the inherited artwork file name is already a perfect match (i.e. + # the directory name and the artwork name are identical) + if unassoc_images: + if not(artwork) or not(artwork.lower().startswith(base[:-1].lower())): + artwork = find_good_artwork(unassoc_images, base) + + # now that the artwork problem is solved, we start processing: + # recurse into subdirectories first + res = [] + for isfile, dummy, fullname, s, ext, key in directories: + res.extend(freeze_dir(cache, index, allocator, playlists, fullname + '/', artwork)) + + # now process the local files + locals = [] + unique_artist = None + unique_album = None + for isfile, dummy, fullname, s, ext, key in music: + try: + # we don't need to move this file if it's already in the Music directory + already_there = fullname.startswith(MUSIC_DIR) + + # is this track cached? + log(fullname + ' ', True) + valid, info = find_in_cache(cache, index, fullname, s) + if valid: + info['changed'] = 0 + log("[cached] ", True) + else: + if info: + # cache entry present, but invalid => save iPod_Control location + path = info['path'] + changed = 1 + else: + path = fullname + changed = 2 + info = mp3info.GetAudioFileInfo(fullname) + iTunesDB.FillMissingTitleAndArtist(info) + info['changed'] = changed + if not already_there: + if type(info['path']) == types.UnicodeType: + info['original path'] = info['path'] + else: + info['original path'] = unicode(info['path'], sys.getfilesystemencoding(), 'replace') + info['path'] = path + + # move the track to where it belongs + if not already_there: + path = info.get('path', None) + if not(path) or os.path.exists(path) or not(os.path.isdir(os.path.split(path)[0])): + # if anything is wrong with the path, generate a new one + path = allocator.allocate() + ext + else: + allocator.add(path) + info['path'] = path + info = move_music(fullname, path, info) + if not info: continue # something failed + else: + allocator.add(fullname) + log("[OK]\n", True) + + # associate artwork to the track + info['artwork'] = image_assoc.get(key, artwork) + + # check for unique artist and album + check = info.get('artist', None) + if not locals: + unique_artist = check + elif check != unique_artist: + unique_artist = False + check = info.get('album', None) + if not locals: + unique_album = check + elif check != unique_album: + unique_album = False + + # finally, append the track to the track list + locals.append(info) + + except KeyboardInterrupt: + log("\nInterrupted by user.\nContinue with next file or abort? [c/A] ") + try: + answer = raw_input() + except (IOError, EOFError, KeyboardInterrupt): + answer = "" + if not answer.lower().startswith("c"): + raise + + except: + g_freeze_error_count += 1 + log("\n" + " runtime error, traceback follows ".center(79, '-') + "\n") + traceback.print_exc(file=Logger) + log(79*'-' + "\n") + + # if all files in this directory share the same album title, but differ + # in the artist name, we assume it's a compilation + if unique_album and not(unique_artist): + for info in locals: + info['compilation'] = 1 + + # combine the lists and return them + res.extend(locals) + return res + + +################################################################################ +## playlist sorting ## +################################################################################ + +def cmp_lst(a, b, order, empty_pos): + a = max(a.get('last played time', 0), a.get('last skipped time', 0)) + b = max(b.get('last played time', 0), b.get('last skipped time', 0)) + if not a: + if not b: return 0 + return empty_pos + else: + if not b: return -empty_pos + return order * cmp(a, b) + +def cmp_path(a, b, order, empty_pos): + return order * trackcmp(a, b) + +class cmp_key: + def __init__(self, key): + self.key = key + def __repr__(self): + return "%s(%s)" % (self.__class__.__name__, repr(self.key)) + def __call__(self, a, b, order, empty_pos): + if self.key in a: + if self.key in b: + a = a[self.key] + if type(a) in (types.StringType, types.UnicodeType): a = a.lower() + b = b[self.key] + if type(b) in (types.StringType, types.UnicodeType): b = b.lower() + return order * cmp(a, b) + else: + return -empty_pos + else: + if self.key in b: + return empty_pos + else: + return 0 + +sort_criteria = { + 'playcount': lambda a,b,o,e: o*cmp(a.get('play count', 0), b.get('play count', 0)), + 'skipcount': lambda a,b,o,e: o*cmp(a.get('skip count', 0), b.get('skip count', 0)), + 'startcount': lambda a,b,o,e: o*cmp(a.get('play count', 0) + a.get('skip count', 0), b.get('play count', 0) + b.get('skip count', 0)), + 'artworkcount': lambda a,b,o,e: o*cmp(a.get('artwork count', 0), b.get('artwork count', 0)), + 'laststartedtime': cmp_lst, + 'laststarttime': cmp_lst, + 'lastplaytime': 'last played time', + 'lastskiptime': 'last skipped time', + 'movie': 'movie flag', + 'filesize': 'size', + 'path': cmp_path, +} +for nc in ('title', 'artist', 'album', 'compilation', 'rating', 'path', \ +'length', 'size', 'track number', 'year', 'bitrate', 'sample rate', 'volume', \ +'last played time', 'last skipped time', 'mtime', 'disc number', 'total discs', \ +'BPM', 'movie flag'): + sort_criteria[nc.replace(' ', '').lower()] = nc + + +re_sortspec = re.compile(r'^([<>+-]*)(.*?)([<>+-]*)$') +class SSParseError: pass + +class SortSpec: + def __init__(self, pattern=None): + if pattern: + self.parse(pattern) + else: + self.criteria = [] + + def parse(self, pattern): + self.criteria = filter(None, map(self._parse_criterion, pattern.split(','))) + + def _parse_criterion(self, text): + text = text.strip() + if not text: return None + m = re_sortspec.match(text) + if not m: + raise SSParseError, "invalid sort criterion `%s'" % text + text = m.group(2).strip() + key = text.lower().replace('_', '').replace(' ', '') + try: + criterion = sort_criteria[key] + except KeyError: + raise SSParseError, "unknown sort criterion `%s'" % text + if type(criterion) == types.StringType: + criterion = cmp_key(criterion) + modifiers = m.group(1) + m.group(3) + order = 1 + if '-' in modifiers: order = -1 + empty_pos = -1 + if '<' in modifiers: empty_pos = 1 + return (criterion, order, empty_pos) + + def _cmp(self, a, b): + for cmp_func, order, empty_pos in self.criteria: + res = cmp_func(self.tracks[a], self.tracks[b], order, empty_pos) + if res: return res + return cmp(a, b) + + def sort(self, tracks): + self.tracks = tracks + index = list(range(len(self.tracks))) + index.sort(self._cmp) + del self.tracks + return [tracks[i] for i in index] + + def __add__(self, other): + self.criteria += other.criteria + return self + + def __len__(self): + return len(self.criteria) + + +################################################################################ +## playlist processing ## +################################################################################ + +def add_scripted_playlist(db, tracklist, list_name, include, exclude, shuffle=False, changemask=0, sort=None): + if not(list_name) or not(include or changemask) or not(tracklist): + return + tracks = [] + log("Processing playlist `%s': " % iTunesDB.kill_unicode(list_name), True) + for track in tracklist: + if not 'original path' in track: + continue # we don't know the real name of this file, so skip it + name = track['original path'].encode(sys.getfilesystemencoding(), 'replace').lower() + ok = changemask & track.get('changed', 0) + for pattern in include: + if fnmatch.fnmatch(name, pattern): + ok = True + break + for pattern in exclude: + if fnmatch.fnmatch(name, pattern): + ok = False + break + if ok: + tracks.append(track) + log("%d tracks\n" % len(tracks)) + if not tracks: + return + if shuffle == 1: + shuffle = BalancedShuffle() + for info in tracks: + shuffle.add(info.get('original path', None) or info.get('path', "???"), info) + tracks = shuffle.shuffle() + if shuffle == 2: + random.shuffle(tracks) + if sort: + tracks = sort.sort(tracks) + db.add_playlist(tracks, list_name) + + +def process_m3u(db, tracklist, index, filename, skip_album_playlists): + if not(filename) or not(tracklist): + return + basedir, list_name = os.path.split(filename) + list_name = unicode(os.path.splitext(list_name)[0], sys.getfilesystemencoding(), 'replace') + log("Processing playlist `%s': " % iTunesDB.kill_unicode(list_name), True) + try: + f = open(filename, "r") + except IOError, e: + log("ERROR: cannot open `%s': %s\n" % (filename, e.strerror)) + tracks = [] + + # collect all tracks + for line in f: + line = line.strip() + if line.startswith('#'): + continue # comment or EXTM3U line + line = os.path.normpath(os.path.join(basedir, line)).replace("\\", "/").lower() + try: + tracks.append(tracklist[index[line]]) + except KeyError: + continue # file not found -> sad, but not fatal + f.close() + + # check if it's an album playlist + if skip_album_playlists: + ref_album = None + ok = True # "we don't know enough about this playlist, so be optimistic" + for info in tracks: + if not 'album' in info: continue + if not ref_album: + ref_album = info['album'] + elif info['album'] != ref_album: + ok = True # "this playlist is mixed-album, so it's clean" + break + else: + ok = False # "all known tracks are from the same album, how sad" + if not ok: + # now check if this playlist really covers the _whole_ album + ok = len(tracks) + for info in tracklist: + try: + if info.get('album', None) == ref_album: + ok -= 1 + if ok < 0: break + except (TypeError, UnicodeDecodeError): + # old (<0.3.0) cache files contain non-unicode information + # for ID3v1 tags which can cause trouble here, so ... + continue + if not(ok) : + log("album playlist, discarding.\n") + return + + # finish everything + log("%d tracks\n" % len(tracks)) + if not tracks: + return + db.add_playlist(tracks, list_name) + + +def make_directory_playlists(db, tracklist): + log("Processing directory playlists ...\n") + dirs = {} + for track in tracklist: + path = track.get('original path', None) + if not path: continue + for dir in path.split('/')[:-1]: + if not dir: continue + if dir in dirs: + dirs[dir].append(track) + else: + dirs[dir] = [track] + dirlist = dirs.keys() + dirlist.sort(fncmp) + + for dir in dirlist: + log("Processing playlist `%s': " % iTunesDB.kill_unicode(dir), True) + tracks = dirs[dir] + tracks.sort(trackcmp) + log("%d tracks\n" % len(tracks)) + db.add_playlist(tracks, dir) + + +shuffle_options = { + "0": 0, "no": 0, "off": 0, "false": 0, "disabled": 0 , "none": 0, + "1": 1, "yes": 1, "on": 1, "true": 0, "enabled": 1, "balanced": 1, + "2": 2, "random": 2, "standard": 2, +} + +def parse_master_playlist_file(): + # helper function + def yesno(s): + if s.lower() in ('true', 'enable', 'enabled', 'yes', 'y'): + return 1 + try: + return (int(s) != 0) + except ValueError: + return 0 + # default values + skip_album_playlists = True + directory_playlists = False + lists = [] + # now we're parsing + try: + f = open(MASTER_PLAYLIST_FILE, "r") + except IOError: + return (skip_album_playlists, directory_playlists, lists) + include = [] + exclude = [] + list_name = None + shuffle = 0 + changemask = 0 + sort = SortSpec() + lineno = 0 + for line in f: + lineno += 1 + line = line.split(';', 1)[0].strip() + if not line: continue + if (line[0] == '[') and (line[-1] == ']'): + if list_name and (include or changemask): + lists.append((list_name, include, exclude, shuffle, changemask, sort)) + include = [] + exclude = [] + list_name = line[1:-1] + shuffle = False + changemask = 0 + sort = SortSpec() + continue + try: + key, value = [x.strip().replace("\\", "/") for x in line.split('=')] + except ValueError: + continue + key = key.lower().replace(' ', '_') + if not value: + log("WARNING: In %s:%d: key `%s' without a value\n" % (MASTER_PLAYLIST_FILE, lineno, key)) + continue + if key == "skip_album_playlists": + if list_name: log("WARNING: In %s:%d: global option `%s' inside a playlist\n" % (MASTER_PLAYLIST_FILE, lineno, key)) + skip_album_playlists = yesno(value) + elif key == "directory_playlists": + if list_name: log("WARNING: In %s:%d: global option `%s' inside a playlist\n" % (MASTER_PLAYLIST_FILE, lineno, key)) + directory_playlists = yesno(value) + elif key == "shuffle": + try: + shuffle = shuffle_options[value.lower()] + except KeyError: + log("WARNING: In %s:%d: invalid value `%s' for shuffle option\n" % (MASTER_PLAYLIST_FILE, lineno, value)) + elif key == "new": + changemask = (changemask & (~2)) | (yesno(value) << 1) + elif key == "changed": + changemask = (changemask & (~1)) | yesno(value) + elif key == "sort": + try: + sort = SortSpec(value) + sort + except SSParseError, e: + log("WARNING: In %s:%d: %s\n" % (MASTER_PLAYLIST_FILE, lineno, e)) + elif key in ("include", "exclude"): + if value[0] == "/": + value = value[1:] + if os.path.isdir(value): + if value[-1] != "/": + value += "/" + value += "*" + if key == "include": + include.append(value.lower()) + else: + exclude.append(value.lower()) + else: + log("WARNING: In %s:%d: unknown key `%s'\n" % (MASTER_PLAYLIST_FILE, lineno, key)) + f.close() + if list_name and (include or changemask): + lists.append((list_name, include, exclude, shuffle, changemask, sort)) + return (skip_album_playlists, directory_playlists, lists) + + +################################################################################ +## artwork ## +################################################################################ + +re_cover = re.compile(r'[^a-z]cover[^a-z]') +re_front = re.compile(r'[^a-z]front[^a-z]') +def find_good_artwork(files, base): + if not files: + return None # sorry, no files here + dirname, basename = os.path.split(base) + if not basename: + dirname, basename = os.path.split(base) + basename = basename.strip().lower() + candidates = [] + for name in files: + ref = os.path.splitext(name)[0].strip().lower() + # if the file has the same name as the directory, we'll use that directly + if ref == basename: + return name + ref = "|%s|" % ref + score = 0 + if re_cover.search(ref): + # if the name contains the word "cover", it's a good candidate + score = -1 + if re_front.search(ref): + # if the name contains the word "front", that's even better + score = -2 + candidates.append((score, name.lower(), name)) + candidates.sort() + return candidates[0][2] # return the candidate with the best score + + +def GenerateArtwork(model, tracklist): + # step 0: check PIL availability + if not iTunesDB.PILAvailable: + log("ERROR: Python Imaging Library (PIL) isn't installed, Artwork is disabled.\n") + log(" Visit http://www.pythonware.com/products/pil/ to get PIL.\n") + return + + # step 1: generate an artwork list + artwork_list = {} + for track in tracklist: + artwork = track.get('artwork', None) + if not artwork: + continue # no artwork file + dbid = track.get('dbid', None) + if not dbid: + continue # artwork doesn't make sense without a dbid + if artwork in artwork_list: + artwork_list[artwork].append(dbid) + else: + artwork_list[artwork] = [dbid] + + # step 2: generate the artwork directory (if it doesn't exist already) + try: + os.mkdir(ARTWORK_DIR[:-1]) + except OSError: + pass # not critical (yet) + + # step 3: try to load the artwork cache + try: + try: + f = open(ARTWORK_CACHE_FILE, "rb") + except IOError: + f = open(OLDNAME(ARTWORK_CACHE_FILE), "rb") + old_cache = cPickle.load(f) + f.close() + except (IOError, EOFError, cPickle.PickleError): + old_cache = ({}, {}) + + # step 4: generate and save the ArtworkDB + artwork_db, new_cache, dbid2mhii = iTunesDB.ArtworkDB(model, artwork_list, cache_data=old_cache) + backup(ARTWORK_DB_FILE) + try: + f = open(ARTWORK_DB_FILE, "wb") + f.write(artwork_db) + f.close() + except IOError, e: + log("FAILED: %s\n" % e.strerror + + "ERROR: The ArtworkDB file could not be written. This means that the iPod will\n" + + "not show any artwork items.\n") + + # step 5: save the artwork cache + try: + f = open(ARTWORK_CACHE_FILE, "wb") + cPickle.dump(new_cache, f) + f.close() + delete(OLDNAME(ARTWORK_CACHE_FILE), True) + except (IOError, EOFError, cPickle.PickleError): + log("ERROR: can't save the artwork cache\n") + + # step 6: update the 'mhii link' field + for track in tracklist: + dbid = track.get('dbid', None) + mhii = dbid2mhii.get(dbid, None) + if mhii: + track['mhii link'] = mhii + elif 'mhii link' in track: + del track['mhii link'] + + +################################################################################ +## FREEZE and UPDATE action ## +################################################################################ + +def Freeze(CacheInfo=None, UpdateOnly=False): + global g_freeze_error_count + if not CacheInfo: CacheInfo = load_cache((None, [])) + state, cache = CacheInfo + + if UpdateOnly: + if (state != "frozen") and not(Options['force']): + confirm(""" +NOTE: The database is not frozen, the update will not work as expected! +""") + else: + if (state == "frozen") and not(Options['force']): + confirm(""" +NOTE: The database is already frozen. +""") + state = "frozen" + + # allocate the filename allocator + if not UpdateOnly: + log("Scanning for present files ...\n", True) + try: + allocator = Allocator(MUSIC_DIR[:-1]) + except (IOError, OSError): + log("FATAL: can't read or write the music directory!\n") + return + + # parse the master playlist setup file + skip_album_playlists, directory_playlists, master_playlists = parse_master_playlist_file() + + # index the track cache + log("Indexing track cache ...\n", True) + index = make_cache_index(cache) + + # allocate scrobbler + scrobbler = scrobble.Scrobbler() + if scrobbler.config(SCROBBLE_CONFIG_FILE): + if not scrobbler.load(SCROBBLE_QUEUE_FILE): + scrobbler.load(OLDNAME(SCROBBLE_QUEUE_FILE)) + else: + scrobbler = None + + # import Play Counts information + if ImportPlayCounts(cache, index, scrobbler): + # save cache and delete the play counts file afterwards + save_cache((state, cache)) + delete(CONTROL_DIR + "Play Counts", may_fail=True) + + # scrobble + if scrobbler and scrobbler.queue: + old_count = len(scrobbler.queue) + log("Scrobbling %d track(s) ... " % old_count, True) + try: + scrobbler.scrobble() + log("OK.\n") + except scrobble.ScrobbleError, e: + log("%s\n" % e) + except KeyboardInterrupt: + log("interrupted by user.\n") + new_count = len(scrobbler.queue) + log("%s track(s) scrobbled, %d track(s) still in queue.\n" % (old_count - new_count, new_count)) + if scrobbler.save(SCROBBLE_QUEUE_FILE): + delete(OLDNAME(SCROBBLE_QUEUE_FILE), True) + else: + log("Error writing scrobbler state file.\n") + + # now go for the real thing + playlists = [] + if not UpdateOnly: + log("Searching for playable files ...\n", True) + tracklist = freeze_dir(cache, index, allocator, playlists) + log("Scan complete: %d tracks found, %d error(s).\n" % (len(tracklist), g_freeze_error_count)) + + # cache save checkpoint + save_cache((state, tracklist)) + else: + # in update mode, use the cached track list directly + tracklist = cache + + # artwork processing + if not UpdateOnly: + model = Options['model'] + if not model: + try: + try: + f = open(MODEL_FILE, "r") + except IOError: + f = open(OLDNAME(MODEL_FILE), "r") + model = f.read().strip()[:10].lower() + f.close() + log("\nLoaded model name `%s' from the cache.\n" % model) + except IOError: + pass + if model: + model = model.strip().lower() + if not(model in iTunesDB.ImageFormats): + log("\nWARNING: model `%s' unrecognized, skipping Artwork generation.\n" % model) + else: + try: + f = open(MODEL_FILE, "w") + f.write(model) + f.close() + delete(OLDNAME(MODEL_FILE), True) + except IOError: + pass + else: + log("\nNo model specified, skipping Artwork generation.\n") + else: + model = None + + # generate track IDs + if not UpdateOnly: + iTunesDB.GenerateIDs(tracklist) + + # generate the artwork list + if model and not(UpdateOnly): + log("\nProcessing Artwork ...\n", True) + GenerateArtwork(model, tracklist) + + # build the database + log("\nCreating iTunesDB ...\n", True) + db = iTunesDB.iTunesDB(tracklist, name="%s %s"%(__title__, __version__)) + + # save the tracklist as the cache for the next run + save_cache((state, tracklist)) + + # add playlists according to the master playlist file + for listspec in master_playlists: + add_scripted_playlist(db, tracklist, *listspec) + + # process all m3u playlists + if playlists: + log("Updating track index ...\n", True) + index = make_cache_index(tracklist) + for plist in playlists: + process_m3u(db, tracklist, index, plist, skip_album_playlists) + + # create directory playlists + if directory_playlists: + make_directory_playlists(db, tracklist) + + # finish iTunesDB and apply hash stuff + log("Finalizing iTunesDB ...\n") + db = db.finish() + fwids = hash58.GetFWIDs() + try: + f = open(FWID_FILE, "r") + fwid = f.read().strip().upper() + f.close() + if len(fwid) != 16: + fwid = None + except IOError: + fwid = None + store_fwid = False + if fwid: + # preferred FWID stored on iPod + if fwids and not(fwid in fwids): + log("WARNING: Stored serial number doesn't match any connected iPod!\n") + else: + # auto-detect FWID + if fwids: + fwid = fwids[0] + store_fwid = (len(fwids) == 1) + if not store_fwid: + log("WARNING: Multiple iPods are connected. If the iPod you are trying to freeze is\n" + + " a recent model, it might not play anything. Please try again with the\n" + + " other iPod unplugged.\n") + else: + log("WARNING: Could not determine your iPod's serial number. If it's a recent model,\n" + + " it will likely not play anything!\n") + if fwid: + db = hash58.UpdateHash(db, fwid) + if store_fwid: + try: + f = open(FWID_FILE, "w") + f.write(fwid) + f.close() + except IOError: + pass + + # write iTunesDB + write_ok = True + backup(DB_FILE) + try: + f = open(DB_FILE, "wb") + f.write(db) + f.close() + except IOError, e: + write_ok = False + log("FAILED: %s\n" % e.strerror + + "ERROR: The iTunesDB file could not be written. This means that the iPod will\n" + + "not play anything.\n") + + # write iPod shuffle stuff (if necessary) + if os.path.exists(CONTROL_DIR + "iTunesSD"): + backup(CONTROL_DIR + "iTunesSD") + log("Creating iTunesSD ... ", True) + db = iTunesDB.iTunesSD(tracklist) + try: + f = open(CONTROL_DIR + "iTunesSD", "wb") + f.write(db) + f.close() + log("\n") + except IOError, e: + write_ok = False + log("FAILED: %s\n" % e.strerror + + "ERROR: The iTunesSD file could not be written. This means that the iPod will\n" + + "not play anything.\n") + delete(CONTROL_DIR + "iTunesShuffle") + delete(CONTROL_DIR + "iTunesPState") + + # generate statistics + if write_ok: + log("\nYou can now unmount the iPod and listen to your music.\n") + sec = int(sum([track.get('length', 0.0) for track in tracklist]) + 0.5) + log("There are %d tracks (%d:%02d:%02d" % (len(tracklist), sec/3600, (sec/60)%60, sec%60)) + if sec > 86400: log(" = %.1f days" % (sec / 86400.0)) + log(") waiting for you to be heard.\n") + + # finally, save the tracklist as the cache for the next run + save_cache((state, tracklist)) + + + +################################################################################ +## UNFREEZE action ## +################################################################################ + +def Unfreeze(CacheInfo=None): + if not CacheInfo: CacheInfo = load_cache((None, None)) + state, cache = CacheInfo + + try: + cache_len = len(cache) + except: + cache_len = None + + if not(state) or (cache_len is None): + fatal("can't unfreeze: rePear cache is missing or broken") + if state!="frozen" and not(Options['force']): + confirm(""" +NOTE: The database is already unfrozen. +""") + + log("Moving tracks back to their original locations ...\n") + success = 0 + failed = 0 + for info in cache: + src = printable(info.get('path', "")) + dest = printable(info.get('original path', "")) + if not src: + log("ERROR: track lacks path attribute\n") + continue + if not dest: + continue # no original path + log("%s " % dest) + if move_file(src, dest): + failed += 1 + else: + success += 1 + log("Operation complete: %d tracks total, %d moved back, %d failed.\n" % \ + (len(cache), success, failed)) + log("\nYou can now manage the music files on your iPod.\n") + save_cache(("unfrozen", cache)) + + +################################################################################ +## the configuration actions ## +################################################################################ + +def ConfigFWID(): + log("Determining serial number (FWID) of attached iPods ...\n") + fwids = hash58.GetFWIDs() + try: + f = open(FWID_FILE, "r") + fwid = f.read().strip().upper() + f.close() + if len(fwid) != 16: + fwid = None + except IOError: + fwid = None + if not fwids: + # no FWIDs detected + if fwid: + return log("No iPod detected, but FWID is already set up (%s).\n\n" % fwid) + else: + return log("No iPod detected, can't determine FWID.\n\n") + if len(fwids) > 1: + # multiple FWIDs detected + if fwid and (fwid in fwids): + return log("Multiple iPods detected, but FWID is already set up (%s).\n\n" % fwid) + else: + return log("Multiple iPods detected, can't determine FWID.\n" + \ + "Please unplug all iPods except the one you're configuring\n\n") + # exactly one FWID detected + log("Serial number detected: %s\n" % fwids[0]) + if fwid and (fwid != fwids[0]): + log("Warning: This serial number is different from the one that has been stored on\n" + \ + " the iPod (%s). Storing the new FWID anyway.\n" % fwid) + fwid = fwids[0] + if not fwid: + return log("\n") + try: + f = open(FWID_FILE, "w") + f.write(fwid) + f.close() + log("FWID saved.\n\n") + except IOError: + log("Error saving the FWID.\n\n") + + +models = ( + (None, "other/unspecified (no cover artwork)"), + ('photo', '4g', "iPod photo (4G)"), + ('video', '5g', "iPod video (5G)"), + ('classic', '6g', "iPod classic (6G)"), + ('nano', 'nano1g', 'nano2g', "iPod nano (1G/2G)"), + ('nano3g', "iPod nano (3G, \"fat nano\")"), + ('nano4g', "iPod nano (4G)"), +) +def is_model_ok(mod_id): + for m in models[1:]: + if mod_id in m[:-1]: + return True + return False + +def ConfigModel(): + try: + try: + f = open(MODEL_FILE, "r") + except IOError: + f = open(OLDNAME(MODEL_FILE), "r") + model = f.read().strip().lower() + f.close() + if not is_model_ok(model): + model = None + except IOError: + model = None + print "Select iPod model:" + default = 0 + for i in xrange(len(models)): + if model in models[i][:-1]: + default = i + c = "*" + else: + c = " " + print c, "%d." % i, models[i][-1] + try: + answer = int(raw_input("Which model is this iPod? [0-%d, default %d] => " % (len(models) - 1, default))) + except (IOError, EOFError, KeyboardInterrupt, ValueError): + answer = default + if (answer < 0) or (answer >= len(models)): + answer = default + if answer: + try: + f = open(MODEL_FILE, "w") + f.write(models[answer][0]) + f.close() + log("Model set to `%s'.\n\n" % models[answer][-1]) + except IOError: + log("Error: cannot set model.\n\n") + else: + delete(MODEL_FILE, True) + log("Model set to `other'.\n\n") + delete(OLDNAME(MODEL_FILE), True) + + +re_ini_key = re.compile(r'^[ \t]*(;)?[ \t]*(\w+)[ \t]*=[ \t]*(.*?)[ \t]*$', re.M) +class INIKey: + def __init__(self, key, value): + self.key = key + self.value = value + self.present = False + self.valid = False + def check(self, m): + if not m: return + if m.group(2).lower() != self.key: return + self.present = True + valid = not(not(m.group(1))) + if not(valid) and self.valid: return + self.start = m.start(3) + self.end = m.end(3) + self.comment = m.start(1) + def apply(self, s): + if not self.present: + if not s.endswith("\n"): s += "\n" + return s + "%s = %s\n" % (self.key, self.value) + s = s[:self.start] + self.value + s[self.end:] + if not(self.valid) and (self.comment >= 0): + s = s[:self.comment] + s[self.comment+1:] + return s + +def ConfigScrobble(): + print "Please enter your last.fm username, or just press ENTER if you don't want to" + try: + username = raw_input("use scrobbling => ").strip() + except (IOError, EOFError, KeyboardInterrupt): + username = "" + if username: + try: + password = getpass.getpass("password => ") + except (IOError, EOFError, KeyboardInterrupt): + password = "" + if password: + password = md5.md5(password).hexdigest() + else: + username = "" + else: + password = "" + + # import config file + try: + f = open(SCROBBLE_CONFIG_FILE, "rb") + config = f.read() + f.close() + except IOError: + config = "" + crlf = (config.find("\r\n") >= 0) + config = config.replace("\r\n", "\n") + + kuser = INIKey("username", username) + kpass = INIKey("password", password) + for m in re_ini_key.finditer(config): + kuser.check(m) + kpass.check(m) + if kuser.present and kpass.present and (kuser.start < kpass.start): + config = kpass.apply(config) + config = kuser.apply(config) + else: + config = kuser.apply(config) + config = kpass.apply(config) + + # export config file + if crlf: + config = config.replace("\n", "\r\n") + try: + f = open(SCROBBLE_CONFIG_FILE, "wb") + f.write(config) + f.close() + if username: + log("Scrobbling enabled for user `%s'.\n\n" % username) + else: + log("Scrobbling disabled.\n\n") + except IOError: + log("Error updating the scrobble config file.\n\n") + + +def ConfigAll(): + ConfigFWID() + ConfigModel() + ConfigScrobble() + + +################################################################################ +## the two minor ("also-ran") actions ## +################################################################################ + +def Auto(): + state, cache = load_cache((None, [])) + if state == 'frozen': + Unfreeze((state, cache)) + else: + Freeze((state, cache)) + +def Reset(): + state, cache = load_cache((None, [])) + if (state == 'frozen') and not(Options['force']): + confirm(""" +WARNING: The database is currently frozen. If you reset the cache now, you will + lose all file name information. This cannot be undone! +""") + return + try: + os.remove(CACHE_FILE) + except OSError: + try: + save_cache((None, [])) + except IOError: + pass + delete(OLDNAME(CACHE_FILE), True) + delete(ARTWORK_CACHE_FILE, True) + delete(OLDNAME(ARTWORK_CACHE_FILE), True) + log("\nCache reset.\n") + + + + +################################################################################ +## the main function ## +################################################################################ + +class MyOptionParser(optparse.OptionParser): + def format_help(self, formatter=None): + models = iTunesDB.ImageFormats.keys() + models.sort() + return optparse.OptionParser.format_help(self, formatter) + """ +Artwork is supported on the following models: + """ + ", ".join(models) + """ + +actions: + help show this help message and exit + freeze move all music files into the iPod's library + unfreeze move music files back to their original location + update update the frozen database without scanning for new files + dissect generate an Artist/Album/Title directory structure + reset clear rePear's metadata cache + cfg-fwid determine the iPod's serial number and save it + cfg-model interactively configure the iPod model + cfg-scrobble configure last.fm scrobbling + config run all of the configuration steps +If no action is specified, rePear automatically determines which of the +`freeze' or `unfreeze' actions should be taken. + +""" + +if __name__ == "__main__": + parser = MyOptionParser(version=__version__, + usage="%prog [options] []") + parser.add_option("-r", "--root", action="store", default=None, metavar="PATH", + help="set the iPod's root directory path") + parser.add_option("-l", "--log", action="store", default="repear.log", metavar="FILE", + help="set the output log file path") + parser.add_option("-m", "--model", action="store", default=None, metavar="MODEL", + help="specify the iPod model (REQUIRED for artwork support)") + parser.add_option("-L", "--lameopts", action="store", default=DEFAULT_LAME_OPTS, metavar="CMDLINE", + help="set the LAME encoder options (default: %s)" % DEFAULT_LAME_OPTS) + parser.add_option("-f", "--force", action="store_true", default=False, + help="skip confirmation prompts for dangerous actions") + parser.add_option("-p", "--playlist", action="store", default=None, metavar="FILE", + help="specify playlist config file") + parser.add_option("-s", "--scrobble", action="store", default=None, metavar="FILE", + help="specify scrobble config file") + if os.name == 'nt': + parser.add_option("--nowait", action="store_true", default=False, + help="don't wait for keypress when finished") + (opts, args) = parser.parse_args() + Options = opts.__dict__ + + if len(args)>1: parser.error("too many arguments") + if args: + action = args[0].strip().lower() + else: + action = "auto" + if action == "help": + parser.print_help() + sys.exit(0) + if not action in ( + 'auto', 'freeze', 'unfreeze', 'update', 'dissect', 'reset', \ + 'config', 'cfg-fwid', 'cfg-scrobble', 'cfg-model' + ): + parser.error("invalid action `%s'" % action) + + oldcwd = os.getcwd() + open_log() + log("%s\n%s\n\n" % (banner, len(banner) * '-')) + if not logfile: + log("WARNING: can't open log file `%s', logging disabled\n\n" % Options['log']) + goto_root_dir() + + if Options['playlist']: + first = Options['playlist'].replace("\\", "/").split('/', 1)[0] + if first in (".", ".."): + MASTER_PLAYLIST_FILE = os.path.normpath(os.path.join(oldcwd, Options['playlist'])) + else: + MASTER_PLAYLIST_FILE = Options['playlist'] + log("master playlist file is `%s'\n" % MASTER_PLAYLIST_FILE) + if Options['scrobble']: + first = Options['scrobble'].replace("\\", "/").split('/', 1)[0] + if first in (".", ".."): + SCROBBLE_CONFIG_FILE = os.path.normpath(os.path.join(oldcwd, Options['scrobble'])) + else: + SCROBBLE_CONFIG_FILE = Options['scrobble'] + log("scrobble configuration file is `%s'\n" % SCROBBLE_CONFIG_FILE) + + log("\n") + try: + if action=="auto": Auto() + elif action=="freeze": Freeze() + elif action=="unfreeze": Unfreeze() + elif action=="update": Freeze(UpdateOnly=True) + elif action=="dissect": Dissect() + elif action=="reset": Reset() + elif action=="config": ConfigAll() + elif action=="cfg-fwid": ConfigFWID() + elif action=="cfg-model": ConfigModel() + elif action=="cfg-scrobble": ConfigScrobble() + else: + log("Unknown action, don't know what to do.\n") + code = 0 + except SystemExit, e: + sys.exit(e.code) + except KeyboardInterrupt: + log("\n" + 79*'-' + "\n\nAction aborted by user.\n") + code = 2 + except: + log("\n" + 79*'-' + "\n\nOOPS -- rePear crashed!\n\n") + traceback.print_exc(file=Logger) + log("\nPlease inform the author of rePear about this crash by sending the\nrepear.log file.\n") + code = 1 + quit(code) diff --git a/repear_playlists.ini b/repear_playlists.ini new file mode 100644 index 0000000..f36aae4 --- /dev/null +++ b/repear_playlists.ini @@ -0,0 +1,32 @@ +; ***** rePear master playlist file ***** +; This file contains descriptions of playlists that rePear shall create +; automatically based on certain properties of the audio files. It consists of +; two parts: global options and playlist specifications. +; For more detailed information on how to use this file, read usage.html + +; ----- global options ----- + +skip album playlists = yes +; YES if .m3u files that are just a directory index shall be skipped [default] +; NO if every .m3u playlist found on the iPod shall be turned into a playlist + +directory playlists = no +; YES if rePear shall create a playlist for *every* folder found on the iPod +; NO if it shall not do that [default] + +; ----- playlist specifications ----- +; playlist specifications use the following syntax: +; [Playlist Name] ; the playlist name +; include = /path/to/directory ; include a directory into this playlist +; exclude = /another/diretory ; exclude a directory from the playlist +; new = 1 ; include all new tracks [optional] +; changed = 1 ; include all changed tracks [optional] +; shuffle = balanced ; enable balanced playlist shuffling [opt.] +; shuffle = random ; enable random playlist shuffling [opt.] +; sort = ; sort tracks [optional, see documentation] + +; an example playlist that contains all tracks that have been modified since +; the last freeze operation +[Hot New Stuff] +new = 1 +changed = 1 diff --git a/repear_scrobble.ini b/repear_scrobble.ini new file mode 100644 index 0000000..fcb34fd --- /dev/null +++ b/repear_scrobble.ini @@ -0,0 +1,12 @@ +; ***** rePear last.fm scrobbling configuration file ***** +; This file contains your login credentials for the last.fm service and +; (optionally) file or directory patterns to exclude from scrobbling. + +username = +password = +; Note: instead of the password itself, you can also specify the MD5 hash +; of the password + +; example excludes: +; exclude = /some/directory +; exclude = *.podcast.mp3 diff --git a/scrobble.py b/scrobble.py new file mode 100644 index 0000000..ddd637b --- /dev/null +++ b/scrobble.py @@ -0,0 +1,243 @@ +#!/usr/bin/env python +# +# last.fm scrobbling library for rePear, the iPod database management tool +# Copyright (C) 2008 Martin J. Fiedler +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + +import sys, urllib, urllib2, re, time, md5, types, fnmatch, os + +try: + import repear + client_id = "rpr" + client_ver = repear.__version__ +except (ImportError, NameError, AttributeError): + client_id = "tst" + client_ver = "1.0" +server = "http://post.audioscrobbler.com:80/" +protocol_version = "1.2" +max_queue = 50 + + +def utf8urlencode(x): + if type(x) != types.UnicodeType: + x = unicode(x, sys.getfilesystemencoding(), 'replace') + return urllib.quote(x.encode('utf-8', 'replace')) + + +class ScrobbleError(Exception): pass + + +class Scrobbler: + def __init__(self, user=None, password=None): + self.user = user + self.password = password + self.excludes = [] + self.queue = [] + self.index = {} + + def _add(self, item): + key = "&".join(map(str, item[:3])).lower() + if not(key in self.index): + self.queue.append(item) + self.index[key] = True + + def config(self, filename): + try: + f = file(filename, "r") + for line in f: + line = line.split(';', 1)[0] + if not(line) or not('=' in line): + continue + key, value = [x.strip() for x in line.split('=', 1)] + key = key.lower() + if key.startswith("user"): + self.user = value + elif key.startswith("pass"): + self.password = value + elif key.startswith("exclude"): + if value[0] == "/": + value = value[1:] + if os.path.isdir(value): + if value[-1] != "/": + value += "/" + value += "*" + self.excludes.append(value.lower()) + f.close() + except IOError: + return False + return not(not(self.user)) and not(not(self.password)) + + def load(self, filename): + try: + f = file(filename, "r") + for line in f: + line = line.strip().split('&') + try: + line[0] = long(line[0]) + except ValueError: + line = [] + if len(line) == 6: + self._add(tuple(line)) + f.close() + except IOError: + return False + return True + + def save(self, filename): + data = "\n".join(["&".join(map(str, item)) for item in self.queue]) + "\n" + try: + f = file(filename, "w") + f.write(data) + f.close() + except IOError: + return False + return True + + def __iadd__(self, item): + self.enqueue(item) + return self + def enqueue(self, item): + path = item.get('original path', None) or item.get('path', None) + if self.excludes and path: + if type(path) == types.UnicodeType: + path = path.encode(sys.getfilesystemencoding(), 'replace') + path = path.lower() + for pattern in self.excludes: + if fnmatch.fnmatch(path, pattern): + return + try: + artist = utf8urlencode(item['artist']) + title = utf8urlencode(item['title']) + length = str(long(item['length'])) + playtime = long(item['last played time']) + except KeyError: + return + album = utf8urlencode(item.get('album', '')) + track = str(item.get('track number', '')) + self._add((playtime, artist, title, length, album, track)) + + def scrobble(self): + if not(self.user) or not(self.password): + raise ScrobbleError, "user name or password missing" + if not(self.queue): + return + + # build authentication request + if re.match(r'^[0-9a-fA-F]{32}$', self.password): + self.password = self.password.lower() + else: + self.password = md5.md5(self.password).hexdigest() + timestamp = str(long(time.time())) + auth = md5.md5(self.password + timestamp).hexdigest() + req = urllib2.Request( + "%s?hs=true&p=%s&c=%s&v=%s&u=%s&t=%s&a=%s" % \ + (server, protocol_version, client_id, client_ver, self.user, timestamp, auth)) + + # send and read authentication request + try: + res = urllib2.urlopen(req).read() +# res = "OK\nfoobar\n\nhttp://localhost:1337/foo" + except urllib2.HTTPError, e: + raise ScrobbleError, "HTTP %d in authentication phase" % e.code + except urllib2.URLError, e: + raise ScrobbleError, "network error in authentication phase: %s" % e.reason.args[1] + except IOError: + raise ScrobbleError, "read error in authentication phase" + res = [line.strip() for line in res.split("\n")] + code = res[0].split()[0].upper() + + # check authentication response + if code == "BANNED": + raise ScrobbleError, "client banned" + elif code == "BADAUTH": + raise ScrobbleError, "invalid username or password" + elif code == "BADTIME": + raise ScrobbleError, "system clock is skewed" + elif code == "FAILED": + raise ScrobbleError, res[0].split(" ", 1)[-1] + elif code != "OK": + raise ScrobbleError, "invalid answer from server: " + res[0] + try: + sid = res[1] + url = res[3] + except IndexError: + raise ScrobbleError, "malformed authentication response" + if not url.startswith("http://"): + raise ScrobbleError, "malformed authentication response" + + # submit queued items + self.queue.sort() + while self.queue: + # build POST request string + data = "s=" + sid + for i in xrange(min(len(self.queue), 50)): + playtime, artist, title, length, album, track = self.queue[i] + data += "&a[%d]=%s&t[%d]=%s&i[%d]=%d&o[%d]=P&r[%d]=&l[%d]=%s&b[%d]=%s&n[%d]=%s&m[%d]=" % \ + (i, artist, i, title, i, playtime, i, i, i, length, i, album, i, track, i) +# print data + + # send and read submission request + try: + res = urllib2.urlopen(url, data).read() +# res = "mist" + except urllib2.HTTPError, e: + raise ScrobbleError, "HTTP %d in submission phase" % e.code + except urllib2.URLError, e: + raise ScrobbleError, "network error in submission phase: %s" % e.reason.args[1] + except IOError: + raise ScrobbleError, "read error in submission phase" + res = res.strip().split("\n", 1)[0].strip() + code = res.split()[0].upper() + + # check response + if code == "BADSESSION": + raise ScrobbleError, "invalid session while submitting" + elif code == "FAILED": + raise ScrobbleError, res.split(" ", 1)[-1] + elif code != "OK": + raise ScrobbleError, "invalid answer from server: " + res + + # finally, remove items from the queue + del self.queue[:max_queue] + + +if __name__ == "__main__": + s = Scrobbler() + s.load() + print "old queue:", len(s.queue), "items" + s += { + 'last played time': 1205685904L, + 'album': u'Story of Ohm', + 'artist': u'paniq', + 'length': 310.43918367346942, + 'title': u'Liberation', + 'track number': 4, + } + s += { + 'last played time': 1205686220L, + 'album': u'Story of Ohm', + 'artist': u'paniq', + 'length': 228.04897959183674, + 'title': u'Story of Ohm', + 'track number': 6, + } + print "new queue:", len(s.queue), "items" + try: + s.scrobble() + except ScrobbleError, e: + print e + print "after scrobbling:", len(s.queue), "items" + s.save() diff --git a/usage.html b/usage.html new file mode 100644 index 0000000..b270c55 --- /dev/null +++ b/usage.html @@ -0,0 +1,349 @@ + + +rePear User's Guide + + + +

rePear User's Guide

+ +

+Author: Martin J. Fiedler <martin.fiedler@gmx.net>
+Version: 0.4.1
+Date: 2009-02-25

+ + +

For the impatient ...

+ +

Initial Setup

+
    +
  1. initialize the iPod with iTunes
    2. +
  2. remove all tracks from the iPod with iTunes (optional)
    3. +
  3. install rePear on the iPod
      +
    • On Windows: unpack the .zip file into the iPod's root directory
      • +
    • On other systems: copy rePear (the .py files) into the iPod's root directory
    4. +
  4. run repear config or use the GUI (rePearUI.exe) to configure rePear
    5. +
  5. run repear dissect or click the »dissect« button in the GUI to have the existing iTunes music library broken down into files and folders with an /artist/album/title scheme (optional, dangerous)
    6. +
  6. copy music onto the iPod with the file manager you like best
    7. +
  7. run rePear to freeze the database
    8. +
  8. wait
    9. +
  9. safely(!) disconnect the iPod
    10. +
  10. listen to your music (optional, but recommended :)
    11. +
+ +

Regular Maintenance (i.e. Manage your Music)

+
    +
  1. connect the iPod to your computer
    2. +
  2. run rePear to unfreeze the database
    3. +
  3. do whatever you want with your music files
    4. +
  4. run rePear to freeze the database again
    5. +
  5. safely(!) disconnect the iPod
    6. +
  6. enjoy your music
    7. +
+ + +

rePear Setup – Step by Step

+

First, hold your breath. This section is awfully long, but that's only because it's written in a very detailed manner. It should not take you longer than about half an hour to get everything going (or ten minutes if you already have a decent Python installation or are working on Windows).

+ +

Now, this is what you have to do to use rePear with your iPod:

+ +

1. Prepare the iPod

+ +

If your iPod is new, you have to connect it to your computer and have it initialized by iTunes at least one time. If you used your iPod with iTunes before, then it's up to you: You may delete all tracks from it to start all over with rePear, but you may as well keep the tracks installed by iTunes. If they contain correct ID3 tags and are all unencrypted MP3 or AAC, this should work fine.

+ +

You also have to find out where the iPod's root directory is. On Windows, this is the root directory of the iPod's drive letter (e.g. L:\). On Mac OS X, it's something like /Volumes/My iPod. On other Unixes, it depends – on my Linux box, it is usually be mounted at /media/sda2, for example.

+ +

Once you figured out where you can find the root directory, you should make a backup of the directory /iPod_Control/iTunes. rePear will automatically create a backup of the most important file in there, iTunesDB, but it is always better to have a second safety net. On Windows, it may be necessary to configure Explorer so that it shows hidden files and directories.

+ + +

2. Install rePear

+ +

If you are on a Windows box, download the .zip file version of rePear and extract it into the iPod's root directory. You should get a rePear subdirectory and some files in the root directory. The two most importand of these are repear.bat, which is a small launcher script if you want to run the command-line version of rePear, and rePearUI.exe, the graphical user interface. This is all you need, everything else is already included in the ZIP file.

+ +

On UNIX-like operating systems, download the .tar.gz version and put all the .py and .ini files from the downloaded archive into the iPod root directory (this is a recommendation; if you don't put rePear there, you will have to use the –r switch described below).
+You will also need a Python runtime to use rePear. On most Unix distributions (including OS X), it is installed by default. If it isn't, use the package manager to install it or look at the official download page to see what to do. Every version of Python between 2.3 and 2.6 should work.
+If you intend to make use of rePear's Ogg Vorbis transcoding feature, you also need to install OggDec (from the vorbis-tools package) and LAME. On Unix-like systems, a vorbis-tools package should be available from your package manager, but you may have to compile and install LAME for yourself.
+Furthermore, the cover artwork features require the Python Imaging Library (PIL). Again, this should be available through the package manager.

+ +

The »direct« way to start the command-line version rePear is to open a console window (on Windows: Start Menu, Run, cmd), go to the iPod root directory (e.g. with L: or cd "/Volumes/My iPod") and run repear [options here] (Windows) or python repear.py [options here] (Unix). This method is needed if you want to specify options or tell rePear which action to perform. If you don't need to do this (and you won't for most of the time), you can as well double-click repear.bat on Windows or use the graphical user interface, which has buttons for the most commonly used operations. On Mac OS X, after double-clicking repear.py, the system will ask whether to execute the program in the Terminal when you double-click it the first time. Confirm this message box. If it tells you that »no default application is specified«, click »Choose Application« and select »Terminal«.

+ + +

3. Configure rePear

+ +

Some of rePear's features require a moderate amount of configuration – namely, the exact iPod model must be known to generate cover artwork and to use the last.fm scrobbling interface, you need to specify your last.fm login data.

+ +

These configuration steps can be done on the command line by running repear config. This will start a small text-based wizard that asks you to pick your model from a list and specify your last.fm credentials. If you don't use last.fm or don't want to use that feature, just press ENTER when asked for the login name.

+ +

In the Windows GUI, there's a »config« button that opens a small dialog window with the same options: a drop-down menu for the model selection and two fields for the last.fm login data. Just make the appropriate settings here and click »OK«.

+ +

Another very important step to follow is synchronizing the clocks of the iPod and the computer: If you intend to use last.fm scrobbling, a precise record of when which track has beed played is essential. So please double-check that the iPod's and the computer's clock agree and are set to the same timezome. This is important, because last.fm might reject your scrobbled tracks if there are any inconsistencies in the play times.

+ + +

4. Import tracks (optional)

+ +

If you still have some tracks in the iPod database that were previously installed with iTunes, you can either keep them in place or have rePear sort them back into directories. If your iPod is empty, just skip this step completely.

+ +

If you choose to keep them inside the iPod's private directory structures, it will be fine. But note that you will not be able to modify them any further (not even delete them) without iTunes.

+ +

The other option is to have rePear dissect the iPod's music database. To do this, start rePear with the dissect parameter (like in repear dissect) or click the »dissect« button in the GUI. After asking you to confirm your command, rePear will create a new folder Dissected Tracks on the iPod. Inside this folder, rePear will create a typical artist/album/track directory structure. For example, you will get files like /Dissected Tracks/Some Artist/Nice Album/07 – Some Track.mp3. Tracks with album information missing will be put directly in the artist's folder. Tracks without even an artist name will be put directly into Dissected Tracks.

+ + +

5. Copy your tracks onto the iPod

+ +

Now it's time to make the USB port glow! You can now copy, delete, move or rename tracks on your iPod. You can create virtually any directory structure you want. It's up to you how to organize the files. In particular, you don't need to put them in any special location. iTunes stores them with cryptic names in /iPod_Control/Music, but you don't need to do that. In fact, it's stronly suggested not to put your self-managed files here – just create a folder elsewhere on the iPod. For example, all my music files go directly into /Music.

+ + +

6. Run rePear

+ +

Now it's time to run rePear without parameters or select the highlighted »freeze« option in the GUI. It will do what I refer to as freezing the iPod's database: It will scan the whole filesystem of the iPod for MP3, Ogg and AAC files and move them into the hidden area where the playable tracks are kept, transcoding Ogg to MP3 as needed. While doing this, rePear analyzes each file to get the ID3 metadata information to finally generate a new music database on the iPod.

+ +

I'm afraid that this process is quite a lengthy one – rePear has a »metadata cache« that remembers all information about the tracks so that they don't need to be re-scanned the next time when new files are added, but moving the files is very time-consuming in itself. You should make sure to enable the operating system's write cache to get best performance. As a rule of thumb, freezing a 8 GB flash-based iPod should take 2 to 5 minutes.

+ + +

7. Disconnect the iPod and listen to your music

+ +

After the freezing process is complete, you can disconnect the iPod. You should always do this in a safe manner (»safe removal of devices« in the Taskbar Notification Area on Windows; the eject icon next to the iPod volume's icon in Finder on OS X; umount on other Unixes).

+ +

You should now be able to listen to your music. If not, please send me a bug report with a precise description of what exactly went wrong, including the log file generated by rePear and a (compressed!) copy of the /iPod_Control/iTunes/iTunesDB file in both versions: The one created by rePear and the one from your previous backup. (And now guess what that backup was good for ... not only for sending me bug reports, but also for fixing your broken iPod again!)

+ + + +

Managing the rePear tracks – Step by Step

+ +

Let's imagine that everything went OK (and I hope it did!) and now you want to put new music onto the iPod, remove old tracks or do other maintaining tasks. Here is what you have to do:

+ + +

1. Plug in the iPod and run rePear

+ +

If you start rePear without any parameters, it will remember that the databaze is frozen and automatically do the right thing: unfreeze it. The GUI doesn't do this completely automatically, but at least it will recognize that »unfreeze« is the most logical thing to do and thus, it will highlight the according button.

+ +

Unfreezing means that all the files you put on the iPod that mysteriously disappeared while freezing (because they were moved to the iPod's sected music hideout) will be put back again. This will not take as long as freezing it, but on iPods that are slow with I/O (most newer models), it will be close.

+ + +

2. Manage your Music

+ +

If there were no errors, all the files will be back at their original locations and you can now start to copy, move and rename files or directories at will.

+ + +

3. Run rePear again

+ +

Before disconnecting the iPod, you have to run rePear once again so that it can freeze the iPod's music database and move the music files away again. All files that you left untouched will be processed in an instant; new or modified files will take a little bit longer to process.

+ +

If you enabled the last.fm scrobbling feature, this will also be the point where the information about when you listened to which tracks will be submitted to last.fm. If a track has been played more than once since the last freeze or update, only the last time it has been played will be submitted. This isn't rePear's fault – the iPod simply only saves the last playback time.
+If any error occurs while uploading the information to last.fm (like a broken network connection, or just because you're offline), the information will not be lost. Instead, it will be stored and re-submitted the next time the freeze or update action is executed.

+ + +

4. Disconnect the iPod and listen to your music

+ +

Just like in the setup procedure described above, you may now safely (I mean it!) disconnect the iPod and enjoy your music.

+ + + +

Details on playlists

+ +

There are two ways to generate playlists with rePear: You can put normal M3U playlist files anywhere on the iPod and rePear will add them to the Playlist menu. These playlists will have the same name as the .m3u file they're created from. The directory name won't be included, so please make sure you don't have a dozen .m3u files with same names on your iPod.

+ +

The other, more advanced method of playlist generation are »automatic« playlists that are created from the main database with some user-specified rules. In the default configuration, there's already one such playlist: »Hot New Stuff« is where rePear puts all new and changed tracks it found into. If you don't like that, you can disable that, of course. This chapter describes how to do that.

+ +

The master playlist file

+ +

All automatic playlists and other playlist options are specified in a single file, which is repear_playlists.ini in the root directory of the iPod. If this file is not present, rePear assumes default options and doesn't generate any automatic playlists (not even »Hot New Stuff«).

+ +

The master playlist file looks like a normal Windows INI file, which means it is a text file with lines in the »key = value« format. It is subdivided into sections that start with a line with the section name in square brackets, like »[this]«. Comments start with a semicolon (;).

+ +

Each section in the master playlist file describes one playlist. The playlist name is derived from the section title. There's one exception, though: The first part of the file, before the first section header, is used for general, global playlist options.

+ +

Some options require boolean values. In this case, you can use either 1/0, yes/no, y/n, on/off, true/false or enable/disable to specify whether that option shall be used or not.

+ +

Global playlist options

+ +

The following options are available in the global options (pseudo-)section at the beginning of the repear_playlists.ini file:

+ +
+ +
skip album playlists (boolean, default: enabled)
+
This option specifies whether playlists that cover exactly one album will be included or not. Normally, these playlists are pointless: The album appears under »Albums«, there's no reason why it should be under »Playlists«, too. This means that you can keep the .m3u files that usually come with album downloads, without having them clutter your Playlists menu on the iPod. That's why this option is enabled by default – if you don't like it, you can disable it, though.
+ +
directory playlists (boolean, default: disabled)
+
If this option is enabled, rePear will create a playlist for every folder on the iPod filesystem it finds playable files in. Note that the playlist name will only contain the last component of the path name. The files in /Music/foo/bar/*.mp3 go into a playlist called »bar«, for example.
+ +
+ +

Automatic playlist options

+ +

The following options can be used to specify automatic playlists:

+ +
+ +
include = <path>
+
Specifies which files shall be included in the playlist, either using a filename pattern like »*.mp3« or a directory name like »/Music«. In the latter case, the specified directory and all subdirectories will be included.
+By default, no files or directories are included in a playlist. This means that a playlist without any include (or new or changed) statements will be empty. So, to make a usable playlists, there needs to be at least one include statement. The number of includes per playlists is unlimited, so it's perfectly possible to include multiple directories.
+ +
exclude = <path>
+
This works like include, but it specifies which files shall not be included in the playlist. exclude is »stronger« than include, so you can exclude subdirectories of other directories. The number of excludes per playlists is also unlimited.
+ +
new = <boolean>
+
If this option is enabled, the playlist will contain all files that reTune found that were not present during the last freeze operation. Note that new is even stronger than include or exclude.
+ +
changed = <boolean>
+
If this option is enabled, the playlist will contain all files whose metadata changed since the last freeze operation. Like new, changed is even »stronger« than include or exclude.
+ +
shuffle = <mode>
+
Selects whether or not the playlist shall be shuffled, and which shuffle algorithm shall be used. The default (0, no, off, false, disabled or none) will keep the tracks in their original order. When this option is enabled (using 1, yes, on, true, enabled or balanced), an advanced shuffle algorithm will be used that creates a not completely random, but very homogenous order of the tracks. The detailed algorithm is described on this web page. Alternatively, a normal random shuffle can be selected with 2, random or standard.
+ +
sort = <criteria>
+
This option specifies the criteria after which the tracks in the playlist shall be sorted. For a detailed explanation of the syntax of these criteria, read below. If there is neither a sort nor a shuffle statement in a playlist definition, the files will be sorted by path and filename. If sorting is enabled, it will take place after shuffling. This means that sorting is »stronger« than shuffling. If there are multiple sort options in a playlist definition, the sort operations will be performed in the same order as defined, so the last sort will be the strongest one.
+ +
+ +

Sort criteria

+ +

The following sort criteria are defined:

+ + + + + + + + + + + + + + + + + + + + + + + + + +
titleTrack title.
artistTrack artist.
albumAlbum title.
yearYear the track has been published in.
compilationWhether the track is part of a compilation or not.
ratingRating, as set up in iTunes or the iPod itself.
pathPath of the file.
lengthLength of the track.
file sizeSize of the file.
mtimeTime of the last modification on the file.
bitrateTrack bitrate.
sample rateTrack sample rate.
track numberThe number of the track on the disc.
disc numberThe number of the disc in the set.
total discsThe total number of discs in the set.
artwork countThe number of cover artworks associated with the track.
BPMThe BPM speed of the track.
movie flagWhether the track is a movie or not.
play countThe number of times the track has been played completely.
skip countThe number of times the track has been skipped over. Note that not every iPod stores this value.
start countThe sum of play count and skip count, i.e. the number of times the track has been started.
last played timeThe time the track has last been played completely.
last skipped timeThe time the track has last been skipped over. Note that not every iPod stores this value.
last started timeEither last played time or last skipped time, whichever is later. In other words, this is the last time the track has been started.
+ +

Multiple criteria can be combined with commas to form complex sort orders. For example »artist, year, album, disc number, track number, title« will sort the tracks by artist, the tracks of each artist will then be sorted by year, the tracks of each year will then be sorted by album, disc number, track number and finally by title. Note that sorting text values will always take place in a case-insensitive manner.

+ +

There are also various modifies for each subcriterion which are added as prefixes: A plus sign (»+«) will sort in ascending order, which is also the default. A minus sign (»-«) will sort in descending order. Angle brackets specify where tracks will be sorted that are missing the value associated with the sort criterion: With »<«, empty values will be sorted to the front, with »>« (the default), empty values will be sorted to the back of the list. The ordering of the empty values is always independent from the main sort order. »-year«, for example, will sort the tracks in descending order by time, but tracks that are missing the »year« metadata field will still appear at the end of the sorted list. If this is not desired, »<-year« needs to be written.

+ +

Playlist examples

+ +

Finally, here's an example of how a repear_playlists.ini file could look like:

+
[Heavy Metal]
+include = /Music/Metal
+
+[Random Soft Songs]
+include = /Music
+exclude = /Music/Metal
+exclude = /Music/Rock
+shuffle = 1
+sort = artist, album, title
+
+[Audiobooks]
+include = *.book.mp3
+
+[Hot New Stuff]
+new = 1
+changed = 1
+
+[Randomized]
+include = /Music
+shuffle = 1
+sort = start count
+

Each playlist is specified by a header, containing the playlist name in square brackets, and a number of statements in a key = value form. In this example, we define three playlists: »Heavy Metal«, »Random Soft Songs« and »Audiobooks«.

+ +

The »Heavy Metal« playlist is a very simple one: It consists only of one include statement that tells rePear to put everything in the /Music/Metal directory (and its subdirectories) into that playlist. You could add more include statements if you like.

+ +

The »Random Soft Songs« playlist basically contains the whole /Music folder, ordered by Artist/Album/Title, but the two exclude statements remove the Metal and Rock subdirectories from that selection. So, what you get would be a collection of all your songs except Metal and Rock ones (given that you organize your music that way, of course). The shuffle = 1 statement makes rePear shuffle all the songs in that playlist. If this statement is not present, rePear would add the songs in alphabetic order, subdirectories first – just like you see them in your file manager if »sort by filename« is selected.

+ +

The »Audiobooks« playlist makes use of a generic filename pattern instead of a directory name. It selects all files whose names end in .book.mp3, regardless of how scattered they are across the iPod's directory structure, and puts them into a common playlist.

+ +

The »Hot New Stuff» playlist is a special one: The new and changed statements act like special include statements, except that they don't match filenames, but the »freshness« of a file. If the new statement is active, rePear will include every file in the playlist that has not been there when the database was last frozen (note that this will include moved files as well!). Likewise, the changed statements instructs rePear to include every file that has been changed, but not added or renamed. In combination, these two statements build a playlist that mirrors every change since the last freeze operation.

+ +

Finally, there's the »Randomized» playlist, which simply contains the whole /Music folder in random order – or almost random order, because it's then sorted by start count. This has the effect that tracks that have been played less frequently occur early in the playlist, while tracks that are played often will be put at the end of the list.

+ + +

Details on artwork

+ +

rePear's artwork support tries to make sensible assumptions on what image to display for every title. Consider, for example, a track like /MyMusic/TheAlbum/TheTrack.mp3. The image (JPEG or PNG format) shown for this track will be the first match from this list (sorted from highest to lowest priority):

    +
  1. an image file with the same name as the music file, but with the filename extension .jpg or .png, like /MyMusic/TheAlbum/TheTrack.jpg
    2. +
  2. an image file with the same name as the directory of the music file (except the extension, of course), like /MyMusic/TheAlbum/TheAlbum.jpg
    3. +
  3. an image file in the same directory as the music file that contains the word »front« somewhere in its name, like /MyMusic/TheAlbum/front.jpg or /MyMusic/TheAlbum/Image_Front.jpg (this rule is there to ensure that »front« covers are prioritzed over »back« covers, even though the latter ones come first in the alphabet)
    4. +
  4. an image file in the same directory that contains the word »cover«, like /MyMusic/TheAlbum/cover_image.jpg
    5. +
  5. the first image file (in alphabetic order) in the directory, like /MyMusic/TheAlbum/SomeImage.jpg
    6. +
  6. an image file with the same name as the directory, but located in the parent directory, like /MyMusic/TheAlbum.jpg
    7. +
  7. the directoy-specific rules are inherited to subdirectories, i.e. if there is any directory-wide image file for one of the parent directories, it will be used for subdirectories, too; consider, for example, /MyMusic/MyMusic.jpg
    8. +
+ +

Please read rule #7 carefully: If you have some totally music-unrelated image files in the root directory, the first of these images will be assigned to every track of the iPod (except those that have higher-priority images, of course). You probably don't want that, so keep your folders clean :)

+ + +

Details on last.fm scrobbling

+ +

Some options regarding the last.fm scrobble feature can be set up in another .ini file in the iPod's root directory: repear_scrobble.ini is located right next to repear_playlists.ini and is syntactically similar. It also looks like a Windows INI file, but it lacks the sections.

+ +

The most important things to set up in this file are username and password – these are your last.fm credentials. The password can be specified either in plaintext or as a MD5 sum. Neither method is really secure, but the MD5 sum at least makes it impossible to guess the plaintext password, so this is the recommended method. Normally, you don't need to set these values anyway, as the setup options in rePear and the GUI launcher already take care of this.

+ +

However, there are additional options in this file that need to be put there by hand if they're needed: The exclude options can specify directories or filename patterns for which scrobbling shall not take place. This is useful if you don't want some of your tracks appear in your last.fm profile, like audiobooks.

+ + + +

A more detailed look at rePear's options and actions

+ +

As stated above, rePear automatically chooses between the freeze and unfreeze actions if it is started without parameters. However, you can override this by specifying the action as a command-line parameter (like in the dissect example: repear dissect). The following actions are available:

+ +
+ +
help
Shows the brief help message and exits the program after that.
+ +
auto
Automatically choose between freeze and unfreeze, based on the current state of the cache. This is the default if no action is specified.
+ +
freeze
Scans the iPod for playable music files, moves them into the /iPod_Control/Music directory and generates an iTunesDB from it.
+ +
unfreeze
Moves music files that have previously been moved into /iPod_Control/Music by the freeze action back to their original locations.
+ +
update
Rebuilds iTunesDB based on rePear's internal cache with the data from the last freeze. In principle, this is identical to the freeze action, except that it doesn't search for new files. However, it will update play counts, scrobble tracks to last.fm and rebuild the automatic playlists specified in the repear_playlists.ini file.
+ +
dissect
Parses the current iTunesDB and moves all tracks found there into a directory following a /Dissected Tracks/<artist>/<album>/<title> scheme.
+ +
reset
Deletes rePear's metadata cache. All information about the tracks installed on the iPod will be erased, but the music files themselves will remain. Note that if this is done while the database is in the frozen state, the information about the original filenames will be lost, too, so the files will be »trapped« in /iPod_Control/Music.
+ +
cfg-fwid
Tries to determine the serial number (FWID) of the currently attached iPod. This will also be done automatically during the first freeze operaton, but the auto-detection can also be started manually with this option.
+The serial number needs to be known because some newer iPods (nano 3G, classic 6G and newer) require a checksum in the iTunesDB file that depends on the serial number.
+ +
cfg-model
Runs the (text-based) model selection wizard.
+ +
cfg-scrobble
Runs the (text-based) scrobble configuration wizard.
+ +
config
Runs all of the cfg options, one after another.
+ +
+ + +

Command-line options

+ +

To get command-line help, run repear –h. This will also tell you about some other options that are available. These should be placed between repear and the action name:

+ +
    +
  • Use –r [some path] to tell rePear where the iPod's root directory is. This is useful if rePear can't determine from what directory it was called, or if you deliberately don't want to keep rePear in the iPod's root directory.
    • +
  • –l [some filename] specifies where the rePear logfile should be written to.
    • +
  • –L [options] can be used to override the LAME encoding options that are used when transcoding Ogg files.
    • +
  • Usage of the option –m [model] is required for the cover artwork feature. This option is only used for the freeze action, and its value is saved for upcoming freeze actions that won't need this option again. It can also be set using the configuration wizard or the GUI. Valid models are: + + + + + +
    nano, nano1g or nano2giPod nano, first or second generation
    4g or photoiPod photo (4th generation)
    5g or videoiPod video (5th generation)
    6g, classic or nano3giPod classic (6th generation) or iPod nano third generation (»fat nano«)
    nano4giPod nano 4th generation
    • +
  • –f deactivates the confirmation prompts that are shown when doing »uncommon« things.
    • +
  • –p [some filename] specifies the location of the master playlist file.
    • +
  • –s [some filename] specifies the location of the scrobble configuration file.
    • +
  • On Windows systems, rePear will wait for a keypress after it is done. The ––nowait option deactivates this behavior.
    • +
+ + + +