commit 663add9246006d0ae503d94ebeb48473561becaf Author: KubaPro010 Date: Mon Dec 1 13:16:08 2025 +0100 My edits, initial commit diff --git a/License.txt b/License.txt new file mode 100644 index 0000000..e72bfdd --- /dev/null +++ b/License.txt @@ -0,0 +1,674 @@ + GNU GENERAL PUBLIC LICENSE + Version 3, 29 June 2007 + + Copyright (C) 2007 Free Software Foundation, Inc. + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The GNU General Public License is a free, copyleft license for +software and other kinds of works. + + The licenses for most software and other practical works are designed +to take away your freedom to share and change the works. By contrast, +the GNU General Public License is intended to guarantee your freedom to +share and change all versions of a program--to make sure it remains free +software for all its users. We, the Free Software Foundation, use the +GNU General Public License for most of our software; it applies also to +any other work released this way by its authors. You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +them if you wish), that you receive source code or can get it if you +want it, that you can change the software or use pieces of it in new +free programs, and that you know you can do these things. + + To protect your rights, we need to prevent others from denying you +these rights or asking you to surrender the rights. Therefore, you have +certain responsibilities if you distribute copies of the software, or if +you modify it: responsibilities to respect the freedom of others. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must pass on to the recipients the same +freedoms that you received. You must make sure that they, too, receive +or can get the source code. And you must show them these terms so they +know their rights. + + Developers that use the GNU GPL protect your rights with two steps: +(1) assert copyright on the software, and (2) offer you this License +giving you legal permission to copy, distribute and/or modify it. + + For the developers' and authors' protection, the GPL clearly explains +that there is no warranty for this free software. For both users' and +authors' sake, the GPL requires that modified versions be marked as +changed, so that their problems will not be attributed erroneously to +authors of previous versions. + + Some devices are designed to deny users access to install or run +modified versions of the software inside them, although the manufacturer +can do so. This is fundamentally incompatible with the aim of +protecting users' freedom to change the software. The systematic +pattern of such abuse occurs in the area of products for individuals to +use, which is precisely where it is most unacceptable. Therefore, we +have designed this version of the GPL to prohibit the practice for those +products. If such problems arise substantially in other domains, we +stand ready to extend this provision to those domains in future versions +of the GPL, as needed to protect the freedom of users. + + Finally, every program is threatened constantly by software patents. +States should not allow patents to restrict development and use of +software on general-purpose computers, but in those that do, we wish to +avoid the special danger that patents applied to a free program could +make it effectively proprietary. To prevent this, the GPL assures that +patents cannot be used to render the program non-free. + + The precise terms and conditions for copying, distribution and +modification follow. + + TERMS AND CONDITIONS + + 0. Definitions. + + "This License" refers to version 3 of the GNU General Public License. + + "Copyright" also means copyright-like laws that apply to other kinds of +works, such as semiconductor masks. + + "The Program" refers to any copyrightable work licensed under this +License. Each licensee is addressed as "you". "Licensees" and +"recipients" may be individuals or organizations. + + To "modify" a work means to copy from or adapt all or part of the work +in a fashion requiring copyright permission, other than the making of an +exact copy. The resulting work is called a "modified version" of the +earlier work or a work "based on" the earlier work. + + A "covered work" means either the unmodified Program or a work based +on the Program. + + To "propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification), making available to the +public, and in some countries other activities as well. + + To "convey" a work means any kind of propagation that enables other +parties to make or receive copies. Mere interaction with a user through +a computer network, with no transfer of a copy, is not conveying. + + An interactive user interface displays "Appropriate Legal Notices" +to the extent that it includes a convenient and prominently visible +feature that (1) displays an appropriate copyright notice, and (2) +tells the user that there is no warranty for the work (except to the +extent that warranties are provided), that licensees may convey the +work under this License, and how to view a copy of this License. If +the interface presents a list of user commands or options, such as a +menu, a prominent item in the list meets this criterion. + + 1. Source Code. + + The "source code" for a work means the preferred form of the work +for making modifications to it. "Object code" means any non-source +form of a work. + + A "Standard Interface" means an interface that either is an official +standard defined by a recognized standards body, or, in the case of +interfaces specified for a particular programming language, one that +is widely used among developers working in that language. + + The "System Libraries" of an executable work include anything, other +than the work as a whole, that (a) is included in the normal form of +packaging a Major Component, but which is not part of that Major +Component, and (b) serves only to enable use of the work with that +Major Component, or to implement a Standard Interface for which an +implementation is available to the public in source code form. A +"Major Component", in this context, means a major essential component +(kernel, window system, and so on) of the specific operating system +(if any) on which the executable work runs, or a compiler used to +produce the work, or an object code interpreter used to run it. + + The "Corresponding Source" for a work in object code form means all +the source code needed to generate, install, and (for an executable +work) run the object code and to modify the work, including scripts to +control those activities. However, it does not include the work's +System Libraries, or general-purpose tools or generally available free +programs which are used unmodified in performing those activities but +which are not part of the work. For example, Corresponding Source +includes interface definition files associated with source files for +the work, and the source code for shared libraries and dynamically +linked subprograms that the work is specifically designed to require, +such as by intimate data communication or control flow between those +subprograms and other parts of the work. + + The Corresponding Source need not include anything that users +can regenerate automatically from other parts of the Corresponding +Source. + + The Corresponding Source for a work in source code form is that +same work. + + 2. Basic Permissions. + + All rights granted under this License are granted for the term of +copyright on the Program, and are irrevocable provided the stated +conditions are met. This License explicitly affirms your unlimited +permission to run the unmodified Program. The output from running a +covered work is covered by this License only if the output, given its +content, constitutes a covered work. This License acknowledges your +rights of fair use or other equivalent, as provided by copyright law. + + You may make, run and propagate covered works that you do not +convey, without conditions so long as your license otherwise remains +in force. You may convey covered works to others for the sole purpose +of having them make modifications exclusively for you, or provide you +with facilities for running those works, provided that you comply with +the terms of this License in conveying all material for which you do +not control copyright. Those thus making or running the covered works +for you must do so exclusively on your behalf, under your direction +and control, on terms that prohibit them from making any copies of +your copyrighted material outside their relationship with you. + + Conveying under any other circumstances is permitted solely under +the conditions stated below. Sublicensing is not allowed; section 10 +makes it unnecessary. + + 3. Protecting Users' Legal Rights From Anti-Circumvention Law. + + No covered work shall be deemed part of an effective technological +measure under any applicable law fulfilling obligations under article +11 of the WIPO copyright treaty adopted on 20 December 1996, or +similar laws prohibiting or restricting circumvention of such +measures. + + When you convey a covered work, you waive any legal power to forbid +circumvention of technological measures to the extent such circumvention +is effected by exercising rights under this License with respect to +the covered work, and you disclaim any intention to limit operation or +modification of the work as a means of enforcing, against the work's +users, your or third parties' legal rights to forbid circumvention of +technological measures. + + 4. Conveying Verbatim Copies. + + You may convey verbatim copies of the Program's source code as you +receive it, in any medium, provided that you conspicuously and +appropriately publish on each copy an appropriate copyright notice; +keep intact all notices stating that this License and any +non-permissive terms added in accord with section 7 apply to the code; +keep intact all notices of the absence of any warranty; and give all +recipients a copy of this License along with the Program. + + You may charge any price or no price for each copy that you convey, +and you may offer support or warranty protection for a fee. + + 5. Conveying Modified Source Versions. + + You may convey a work based on the Program, or the modifications to +produce it from the Program, in the form of source code under the +terms of section 4, provided that you also meet all of these conditions: + + a) The work must carry prominent notices stating that you modified + it, and giving a relevant date. + + b) The work must carry prominent notices stating that it is + released under this License and any conditions added under section + 7. This requirement modifies the requirement in section 4 to + "keep intact all notices". + + c) You must license the entire work, as a whole, under this + License to anyone who comes into possession of a copy. This + License will therefore apply, along with any applicable section 7 + additional terms, to the whole of the work, and all its parts, + regardless of how they are packaged. This License gives no + permission to license the work in any other way, but it does not + invalidate such permission if you have separately received it. + + d) If the work has interactive user interfaces, each must display + Appropriate Legal Notices; however, if the Program has interactive + interfaces that do not display Appropriate Legal Notices, your + work need not make them do so. + + A compilation of a covered work with other separate and independent +works, which are not by their nature extensions of the covered work, +and which are not combined with it such as to form a larger program, +in or on a volume of a storage or distribution medium, is called an +"aggregate" if the compilation and its resulting copyright are not +used to limit the access or legal rights of the compilation's users +beyond what the individual works permit. Inclusion of a covered work +in an aggregate does not cause this License to apply to the other +parts of the aggregate. + + 6. Conveying Non-Source Forms. + + You may convey a covered work in object code form under the terms +of sections 4 and 5, provided that you also convey the +machine-readable Corresponding Source under the terms of this License, +in one of these ways: + + a) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by the + Corresponding Source fixed on a durable physical medium + customarily used for software interchange. + + b) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by a + written offer, valid for at least three years and valid for as + long as you offer spare parts or customer support for that product + model, to give anyone who possesses the object code either (1) a + copy of the Corresponding Source for all the software in the + product that is covered by this License, on a durable physical + medium customarily used for software interchange, for a price no + more than your reasonable cost of physically performing this + conveying of source, or (2) access to copy the + Corresponding Source from a network server at no charge. + + c) Convey individual copies of the object code with a copy of the + written offer to provide the Corresponding Source. This + alternative is allowed only occasionally and noncommercially, and + only if you received the object code with such an offer, in accord + with subsection 6b. + + d) Convey the object code by offering access from a designated + place (gratis or for a charge), and offer equivalent access to the + Corresponding Source in the same way through the same place at no + further charge. You need not require recipients to copy the + Corresponding Source along with the object code. If the place to + copy the object code is a network server, the Corresponding Source + may be on a different server (operated by you or a third party) + that supports equivalent copying facilities, provided you maintain + clear directions next to the object code saying where to find the + Corresponding Source. Regardless of what server hosts the + Corresponding Source, you remain obligated to ensure that it is + available for as long as needed to satisfy these requirements. + + e) Convey the object code using peer-to-peer transmission, provided + you inform other peers where the object code and Corresponding + Source of the work are being offered to the general public at no + charge under subsection 6d. + + A separable portion of the object code, whose source code is excluded +from the Corresponding Source as a System Library, need not be +included in conveying the object code work. + + A "User Product" is either (1) a "consumer product", which means any +tangible personal property which is normally used for personal, family, +or household purposes, or (2) anything designed or sold for incorporation +into a dwelling. In determining whether a product is a consumer product, +doubtful cases shall be resolved in favor of coverage. For a particular +product received by a particular user, "normally used" refers to a +typical or common use of that class of product, regardless of the status +of the particular user or of the way in which the particular user +actually uses, or expects or is expected to use, the product. A product +is a consumer product regardless of whether the product has substantial +commercial, industrial or non-consumer uses, unless such uses represent +the only significant mode of use of the product. + + "Installation Information" for a User Product means any methods, +procedures, authorization keys, or other information required to install +and execute modified versions of a covered work in that User Product from +a modified version of its Corresponding Source. The information must +suffice to ensure that the continued functioning of the modified object +code is in no case prevented or interfered with solely because +modification has been made. + + If you convey an object code work under this section in, or with, or +specifically for use in, a User Product, and the conveying occurs as +part of a transaction in which the right of possession and use of the +User Product is transferred to the recipient in perpetuity or for a +fixed term (regardless of how the transaction is characterized), the +Corresponding Source conveyed under this section must be accompanied +by the Installation Information. But this requirement does not apply +if neither you nor any third party retains the ability to install +modified object code on the User Product (for example, the work has +been installed in ROM). + + The requirement to provide Installation Information does not include a +requirement to continue to provide support service, warranty, or updates +for a work that has been modified or installed by the recipient, or for +the User Product in which it has been modified or installed. Access to a +network may be denied when the modification itself materially and +adversely affects the operation of the network or violates the rules and +protocols for communication across the network. + + Corresponding Source conveyed, and Installation Information provided, +in accord with this section must be in a format that is publicly +documented (and with an implementation available to the public in +source code form), and must require no special password or key for +unpacking, reading or copying. + + 7. Additional Terms. + + "Additional permissions" are terms that supplement the terms of this +License by making exceptions from one or more of its conditions. +Additional permissions that are applicable to the entire Program shall +be treated as though they were included in this License, to the extent +that they are valid under applicable law. If additional permissions +apply only to part of the Program, that part may be used separately +under those permissions, but the entire Program remains governed by +this License without regard to the additional permissions. + + When you convey a copy of a covered work, you may at your option +remove any additional permissions from that copy, or from any part of +it. (Additional permissions may be written to require their own +removal in certain cases when you modify the work.) You may place +additional permissions on material, added by you to a covered work, +for which you have or can give appropriate copyright permission. + + Notwithstanding any other provision of this License, for material you +add to a covered work, you may (if authorized by the copyright holders of +that material) supplement the terms of this License with terms: + + a) Disclaiming warranty or limiting liability differently from the + terms of sections 15 and 16 of this License; or + + b) Requiring preservation of specified reasonable legal notices or + author attributions in that material or in the Appropriate Legal + Notices displayed by works containing it; or + + c) Prohibiting misrepresentation of the origin of that material, or + requiring that modified versions of such material be marked in + reasonable ways as different from the original version; or + + d) Limiting the use for publicity purposes of names of licensors or + authors of the material; or + + e) Declining to grant rights under trademark law for use of some + trade names, trademarks, or service marks; or + + f) Requiring indemnification of licensors and authors of that + material by anyone who conveys the material (or modified versions of + it) with contractual assumptions of liability to the recipient, for + any liability that these contractual assumptions directly impose on + those licensors and authors. + + All other non-permissive additional terms are considered "further +restrictions" within the meaning of section 10. If the Program as you +received it, or any part of it, contains a notice stating that it is +governed by this License along with a term that is a further +restriction, you may remove that term. If a license document contains +a further restriction but permits relicensing or conveying under this +License, you may add to a covered work material governed by the terms +of that license document, provided that the further restriction does +not survive such relicensing or conveying. + + If you add terms to a covered work in accord with this section, you +must place, in the relevant source files, a statement of the +additional terms that apply to those files, or a notice indicating +where to find the applicable terms. + + Additional terms, permissive or non-permissive, may be stated in the +form of a separately written license, or stated as exceptions; +the above requirements apply either way. + + 8. Termination. + + You may not propagate or modify a covered work except as expressly +provided under this License. Any attempt otherwise to propagate or +modify it is void, and will automatically terminate your rights under +this License (including any patent licenses granted under the third +paragraph of section 11). + + However, if you cease all violation of this License, then your +license from a particular copyright holder is reinstated (a) +provisionally, unless and until the copyright holder explicitly and +finally terminates your license, and (b) permanently, if the copyright +holder fails to notify you of the violation by some reasonable means +prior to 60 days after the cessation. + + Moreover, your license from a particular copyright holder is +reinstated permanently if the copyright holder notifies you of the +violation by some reasonable means, this is the first time you have +received notice of violation of this License (for any work) from that +copyright holder, and you cure the violation prior to 30 days after +your receipt of the notice. + + Termination of your rights under this section does not terminate the +licenses of parties who have received copies or rights from you under +this License. If your rights have been terminated and not permanently +reinstated, you do not qualify to receive new licenses for the same +material under section 10. + + 9. Acceptance Not Required for Having Copies. + + You are not required to accept this License in order to receive or +run a copy of the Program. Ancillary propagation of a covered work +occurring solely as a consequence of using peer-to-peer transmission +to receive a copy likewise does not require acceptance. However, +nothing other than this License grants you permission to propagate or +modify any covered work. These actions infringe copyright if you do +not accept this License. Therefore, by modifying or propagating a +covered work, you indicate your acceptance of this License to do so. + + 10. Automatic Licensing of Downstream Recipients. + + Each time you convey a covered work, the recipient automatically +receives a license from the original licensors, to run, modify and +propagate that work, subject to this License. You are not responsible +for enforcing compliance by third parties with this License. + + An "entity transaction" is a transaction transferring control of an +organization, or substantially all assets of one, or subdividing an +organization, or merging organizations. If propagation of a covered +work results from an entity transaction, each party to that +transaction who receives a copy of the work also receives whatever +licenses to the work the party's predecessor in interest had or could +give under the previous paragraph, plus a right to possession of the +Corresponding Source of the work from the predecessor in interest, if +the predecessor has it or can get it with reasonable efforts. + + You may not impose any further restrictions on the exercise of the +rights granted or affirmed under this License. For example, you may +not impose a license fee, royalty, or other charge for exercise of +rights granted under this License, and you may not initiate litigation +(including a cross-claim or counterclaim in a lawsuit) alleging that +any patent claim is infringed by making, using, selling, offering for +sale, or importing the Program or any portion of it. + + 11. Patents. + + A "contributor" is a copyright holder who authorizes use under this +License of the Program or a work on which the Program is based. The +work thus licensed is called the contributor's "contributor version". + + A contributor's "essential patent claims" are all patent claims +owned or controlled by the contributor, whether already acquired or +hereafter acquired, that would be infringed by some manner, permitted +by this License, of making, using, or selling its contributor version, +but do not include claims that would be infringed only as a +consequence of further modification of the contributor version. For +purposes of this definition, "control" includes the right to grant +patent sublicenses in a manner consistent with the requirements of +this License. + + Each contributor grants you a non-exclusive, worldwide, royalty-free +patent license under the contributor's essential patent claims, to +make, use, sell, offer for sale, import and otherwise run, modify and +propagate the contents of its contributor version. + + In the following three paragraphs, a "patent license" is any express +agreement or commitment, however denominated, not to enforce a patent +(such as an express permission to practice a patent or covenant not to +sue for patent infringement). To "grant" such a patent license to a +party means to make such an agreement or commitment not to enforce a +patent against the party. + + If you convey a covered work, knowingly relying on a patent license, +and the Corresponding Source of the work is not available for anyone +to copy, free of charge and under the terms of this License, through a +publicly available network server or other readily accessible means, +then you must either (1) cause the Corresponding Source to be so +available, or (2) arrange to deprive yourself of the benefit of the +patent license for this particular work, or (3) arrange, in a manner +consistent with the requirements of this License, to extend the patent +license to downstream recipients. "Knowingly relying" means you have +actual knowledge that, but for the patent license, your conveying the +covered work in a country, or your recipient's use of the covered work +in a country, would infringe one or more identifiable patents in that +country that you have reason to believe are valid. + + If, pursuant to or in connection with a single transaction or +arrangement, you convey, or propagate by procuring conveyance of, a +covered work, and grant a patent license to some of the parties +receiving the covered work authorizing them to use, propagate, modify +or convey a specific copy of the covered work, then the patent license +you grant is automatically extended to all recipients of the covered +work and works based on it. + + A patent license is "discriminatory" if it does not include within +the scope of its coverage, prohibits the exercise of, or is +conditioned on the non-exercise of one or more of the rights that are +specifically granted under this License. You may not convey a covered +work if you are a party to an arrangement with a third party that is +in the business of distributing software, under which you make payment +to the third party based on the extent of your activity of conveying +the work, and under which the third party grants, to any of the +parties who would receive the covered work from you, a discriminatory +patent license (a) in connection with copies of the covered work +conveyed by you (or copies made from those copies), or (b) primarily +for and in connection with specific products or compilations that +contain the covered work, unless you entered into that arrangement, +or that patent license was granted, prior to 28 March 2007. + + Nothing in this License shall be construed as excluding or limiting +any implied license or other defenses to infringement that may +otherwise be available to you under applicable patent law. + + 12. No Surrender of Others' Freedom. + + If conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot convey a +covered work so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you may +not convey it at all. For example, if you agree to terms that obligate you +to collect a royalty for further conveying from those to whom you convey +the Program, the only way you could satisfy both those terms and this +License would be to refrain entirely from conveying the Program. + + 13. Use with the GNU Affero General Public License. + + Notwithstanding any other provision of this License, you have +permission to link or combine any covered work with a work licensed +under version 3 of the GNU Affero General Public License into a single +combined work, and to convey the resulting work. The terms of this +License will continue to apply to the part which is the covered work, +but the special requirements of the GNU Affero General Public License, +section 13, concerning interaction through a network will apply to the +combination as such. + + 14. Revised Versions of this License. + + The Free Software Foundation may publish revised and/or new versions of +the GNU General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + + Each version is given a distinguishing version number. If the +Program specifies that a certain numbered version of the GNU General +Public License "or any later version" applies to it, you have the +option of following the terms and conditions either of that numbered +version or of any later version published by the Free Software +Foundation. If the Program does not specify a version number of the +GNU General Public License, you may choose any version ever published +by the Free Software Foundation. + + If the Program specifies that a proxy can decide which future +versions of the GNU General Public License can be used, that proxy's +public statement of acceptance of a version permanently authorizes you +to choose that version for the Program. + + Later license versions may give you additional or different +permissions. However, no additional obligations are imposed on any +author or copyright holder as a result of your choosing to follow a +later version. + + 15. Disclaimer of Warranty. + + THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT +HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY +OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, +THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM +IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF +ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. Limitation of Liability. + + IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS +THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY +GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE +USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF +DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD +PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), +EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF +SUCH DAMAGES. + + 17. Interpretation of Sections 15 and 16. + + If the disclaimer of warranty and limitation of liability provided +above cannot be given local legal effect according to their terms, +reviewing courts shall apply local law that most closely approximates +an absolute waiver of all civil liability in connection with the +Program, unless a warranty or assumption of liability accompanies a +copy of the Program in return for a fee. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +state the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + + Copyright (C) + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see . + +Also add information on how to contact you by electronic and paper mail. + + If the program does terminal interaction, make it output a short +notice like this when it starts in an interactive mode: + + Copyright (C) + This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, your program's commands +might be different; for a GUI interface, you would use an "about box". + + You should also get your employer (if you work as a programmer) or school, +if any, to sign a "copyright disclaimer" for the program, if necessary. +For more information on this, and how to apply and follow the GNU GPL, see +. + + The GNU General Public License does not permit incorporating your program +into proprietary programs. If your program is a subroutine library, you +may consider it more useful to permit linking proprietary applications with +the library. If this is what you want to do, use the GNU Lesser General +Public License instead of this License. But first, please read +. \ No newline at end of file diff --git a/__pycache__/hash58.cpython-313.pyc b/__pycache__/hash58.cpython-313.pyc new file mode 100644 index 0000000..d55d6c4 Binary files /dev/null and b/__pycache__/hash58.cpython-313.pyc differ diff --git a/__pycache__/iTunesDB.cpython-313.pyc b/__pycache__/iTunesDB.cpython-313.pyc new file mode 100644 index 0000000..9972479 Binary files /dev/null and b/__pycache__/iTunesDB.cpython-313.pyc differ diff --git a/__pycache__/mp3info.cpython-313.pyc b/__pycache__/mp3info.cpython-313.pyc new file mode 100644 index 0000000..8cb4bca Binary files /dev/null and b/__pycache__/mp3info.cpython-313.pyc differ diff --git a/__pycache__/qtparse.cpython-313.pyc b/__pycache__/qtparse.cpython-313.pyc new file mode 100644 index 0000000..fef7902 Binary files /dev/null and b/__pycache__/qtparse.cpython-313.pyc differ diff --git a/hash58.py b/hash58.py new file mode 100644 index 0000000..2518663 --- /dev/null +++ b/hash58.py @@ -0,0 +1,276 @@ +#!/usr/bin/env python +# +# hash generation library for rePear, the iPod database management tool +# Copyright (C) 2008 Martin J. Fiedler +# based on original code by William Whistler (wtbw) +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + +import os, re, hashlib + +try: + import winreg + HaveWin32 = True +except ImportError: + HaveWin32 = False + +def GetFWIDs_Win32(): + # phase 1: enumerate all mass storage devices + if not HaveWin32: return [] + try: + key = winreg.OpenKey(winreg.HKEY_LOCAL_MACHINE, "SYSTEM\\CurrentControlSet\\Services\\Disk\\Enum", 0, winreg.KEY_QUERY_VALUE) + except: + raise + return [] + devs = [] + devid = 0 + try: + while True: + dev = winreg.QueryValueEx(key, str(devid))[0] + devs.append(dev.upper()) + devid += 1 + except: + pass + key.Close() + + # phase 2: find iPods and their FWIDs there (I'm being very careful here) + fwids = [] + for dev in devs: + dev = dev.upper().replace("\\", "&").split("&") + info = dict([x.split('_', 1) for x in dev if '_' in x]) + vendor = info.get('VEN', None) or info.get('VID', None) + product = info.get('DEV', None) or info.get('PROD', None) or info.get('PID', None) + if not(vendor in ('APPLE', '05AC')): continue + if product and (product != 'IPOD') and not(product[:2] in ('12', '13')): continue + fwid = dev[-2].upper() + for item in dev: + if item.startswith("000A27"): fwid = item + if len(fwid) == 16: fwids.append(fwid) + return fwids + +def GetFWIDs_Linux26(): + LINUX_DEVDIR_BASE = "/sys/bus/usb/devices" + try: + devs = os.listdir(LINUX_DEVDIR_BASE) + except OSError: + return [] + def sysfile(filename): + try: + f = open(os.path.join(devdir, filename), "rb") + data = f.read() + f.close() + return data.split(b"\0", 1)[0].strip().upper() + except IOError: + return "" + fwids = [] + re_devdir = re.compile(r'^\d+-\d+$') + for dev in devs: + if not re_devdir.match(dev): continue + devdir = os.path.join(LINUX_DEVDIR_BASE, dev) + if not os.path.isdir(devdir): continue + if not((sysfile("idVendor") == "05AC") \ + or (sysfile("manufacturer") == "APPLE")): continue + if not((sysfile("idProduct")[:2] in ("12", "13")) \ + or (sysfile("product") == "IPOD")): continue + fwid = sysfile("serial") + if fwid and fwid.startswith(b"000A27"): + fwids.append(fwid) + return fwids + +re_ioreg = re.compile(r'"(.*?)"\s+=\s+"?(.*?)"?$') +def GetFWIDs_Darwin(): + devs = [] + try: + f = os.popen("/usr/sbin/ioreg -l", 'r') + valid = False + for line in f: + line = line.strip("\r\n\t |+-") + if not line: + continue + if line.startswith('o'): + valid = (line[1:].strip().split()[0].split('@', 1)[0] == "iPod") + elif valid and line.startswith('"'): + m = re_ioreg.match(line) + if m: + key, value = m.groups() + if (key.lower() == "usb serial number") and (len(value) == 16): + devs.append(value.upper()) + f.close() + except (OSError, IOError, EOFError): + pass + return devs + +def GetFWIDs(): + if os.name == 'nt': + return GetFWIDs_Win32() + elif os.name == 'posix': + try: + uname = os.uname()[0].lower() + except (AttributeError, OSError): + return [] + if uname == 'linux': + return GetFWIDs_Linux26() + elif uname == 'darwin': + return GetFWIDs_Darwin() + return [] + + +################################################################################ + + +inv = [ + 0x74, 0x85, 0x96, 0xA7, 0xB8, 0xC9, 0xDA, 0xEB, 0xFC, 0x0D, 0x1E, 0x2F, 0x40, 0x51, 0x62, 0x73, + 0x84, 0x95, 0xA6, 0xB7, 0xC8, 0xD9, 0xEA, 0xFB, 0x0C, 0x1D, 0x2E, 0x3F, 0x50, 0x61, 0x72, 0x83, + 0x94, 0xA5, 0xB6, 0xC7, 0xD8, 0xE9, 0xFA, 0x0B, 0x1C, 0x2D, 0x3E, 0x4F, 0x60, 0x71, 0x82, 0x93, + 0xA4, 0xB5, 0xC6, 0xD7, 0xE8, 0xF9, 0x0A, 0x1B, 0x2C, 0x3D, 0x4E, 0x5F, 0x70, 0x81, 0x92, 0xA3, + 0xB4, 0xC5, 0xD6, 0xE7, 0xF8, 0x09, 0x1A, 0x2B, 0x3C, 0x4D, 0x5E, 0x6F, 0x80, 0x91, 0xA2, 0xB3, + 0xC4, 0xD5, 0xE6, 0xF7, 0x08, 0x19, 0x2A, 0x3B, 0x4C, 0x5D, 0x6E, 0x7F, 0x90, 0xA1, 0xB2, 0xC3, + 0xD4, 0xE5, 0xF6, 0x07, 0x18, 0x29, 0x3A, 0x4B, 0x5C, 0x6D, 0x7E, 0x8F, 0xA0, 0xB1, 0xC2, 0xD3, + 0xE4, 0xF5, 0x06, 0x17, 0x28, 0x39, 0x4A, 0x5B, 0x6C, 0x7D, 0x8E, 0x9F, 0xB0, 0xC1, 0xD2, 0xE3, + 0xF4, 0x05, 0x16, 0x27, 0x38, 0x49, 0x5A, 0x6B, 0x7C, 0x8D, 0x9E, 0xAF, 0xC0, 0xD1, 0xE2, 0xF3, + 0x04, 0x15, 0x26, 0x37, 0x48, 0x59, 0x6A, 0x7B, 0x8C, 0x9D, 0xAE, 0xBF, 0xD0, 0xE1, 0xF2, 0x03, + 0x14, 0x25, 0x36, 0x47, 0x58, 0x69, 0x7A, 0x8B, 0x9C, 0xAD, 0xBE, 0xCF, 0xE0, 0xF1, 0x02, 0x13, + 0x24, 0x35, 0x46, 0x57, 0x68, 0x79, 0x8A, 0x9B, 0xAC, 0xBD, 0xCE, 0xDF, 0xF0, 0x01, 0x12, 0x23, + 0x34, 0x45, 0x56, 0x67, 0x78, 0x89, 0x9A, 0xAB, 0xBC, 0xCD, 0xDE, 0xEF, 0x00, 0x11, 0x22, 0x33, + 0x44, 0x55, 0x66, 0x77, 0x88, 0x99, 0xAA, 0xBB, 0xCC, 0xDD, 0xEE, 0xFF, 0x10, 0x21, 0x32, 0x43, + 0x54, 0x65, 0x76, 0x87, 0x98, 0xA9, 0xBA, 0xCB, 0xDC, 0xED, 0xFE, 0x0F, 0x20, 0x31, 0x42, 0x53, + 0x64, 0x75, 0x86, 0x97, 0xA8, 0xB9, 0xCA, 0xDB, 0xEC, 0xFD, 0x0E, 0x1F, 0x30, 0x41, 0x52, 0x63 +] + +table1 = [ + 0x3A, 0x3F, 0x3E, 0x72, 0xBD, 0xA2, 0xD6, 0xB4, 0x63, 0xC0, 0x6E, 0x62, 0x59, 0x1E, 0xE2, 0x71, + 0xB5, 0x0D, 0xE8, 0x0C, 0x25, 0x38, 0xCE, 0x23, 0x7C, 0xB7, 0xAD, 0x16, 0xDF, 0x47, 0x3D, 0xB3, + 0x7E, 0x8C, 0xAA, 0x61, 0x31, 0x66, 0xBE, 0x4F, 0x97, 0x14, 0x54, 0xF0, 0x70, 0xEB, 0x30, 0xC4, + 0x27, 0x4E, 0xFA, 0x1A, 0x2B, 0x11, 0xF4, 0x45, 0x8E, 0x5D, 0x73, 0xED, 0x22, 0x2E, 0x7D, 0xA4, + 0x28, 0xDA, 0x2F, 0xC5, 0x92, 0x09, 0x05, 0x13, 0x9D, 0x32, 0x51, 0x4A, 0xC8, 0xBA, 0x96, 0xA7, + 0x6A, 0x50, 0xF3, 0xBC, 0x93, 0xBF, 0xB0, 0xD2, 0xD5, 0x82, 0x19, 0x98, 0x35, 0xCF, 0x6B, 0xB6, + 0x83, 0x56, 0x15, 0xF2, 0x9A, 0x9C, 0xCA, 0x74, 0x34, 0x58, 0x8D, 0xA6, 0x03, 0xFF, 0x46, 0x7B, + 0xD0, 0x7A, 0x33, 0x76, 0xDD, 0xAC, 0xCB, 0x24, 0x7F, 0xB1, 0x85, 0x60, 0xC3, 0x26, 0x8A, 0x1D, + 0x1C, 0x8F, 0x2A, 0xEF, 0x06, 0xDE, 0x67, 0x5E, 0xE7, 0xAE, 0xD9, 0xCC, 0x07, 0x6C, 0xF8, 0x0A, + 0xD3, 0x40, 0x36, 0x1F, 0x2D, 0x95, 0x43, 0xDB, 0x01, 0x89, 0x4B, 0xF7, 0xB9, 0x39, 0xC2, 0x52, + 0x53, 0xFD, 0x65, 0xF5, 0x68, 0xC1, 0xC7, 0x9F, 0x4D, 0xEA, 0xAF, 0x6D, 0x10, 0x44, 0x87, 0xD8, + 0xEE, 0x1B, 0xFE, 0x3C, 0xDC, 0x84, 0x69, 0x48, 0x6F, 0xD1, 0x57, 0x55, 0xD4, 0xA5, 0x49, 0x5B, + 0xE5, 0x0B, 0x94, 0xC9, 0x5F, 0xE1, 0x17, 0x81, 0xBB, 0xEC, 0xD7, 0xC6, 0x02, 0x4C, 0x42, 0x75, + 0xA3, 0x99, 0xE4, 0xA1, 0x9B, 0x5A, 0xF1, 0x29, 0xA0, 0x64, 0x9E, 0x18, 0x41, 0x80, 0x2C, 0x79, + 0x20, 0x8B, 0xAB, 0x90, 0x08, 0xB8, 0xA9, 0x77, 0x12, 0xF9, 0x0E, 0x88, 0xE9, 0x04, 0xFB, 0x86, + 0x0F, 0xE0, 0xA8, 0x5C, 0xE6, 0x21, 0xCD, 0x3B, 0x00, 0x78, 0xFC, 0xF6, 0xE3, 0x37, 0xB2, 0x91 +] + +table2 = [ + 0xF3, 0xE4, 0x1B, 0x38, 0xE5, 0x6F, 0xE8, 0x9D, 0x3E, 0x55, 0xBA, 0xC7, 0xAC, 0xEA, 0x66, 0xA2, + 0xB9, 0x7A, 0x34, 0x43, 0x02, 0x4E, 0xFE, 0x36, 0x41, 0x57, 0x1A, 0xB1, 0x31, 0x87, 0x04, 0x52, + 0x21, 0x22, 0xE1, 0x13, 0x7F, 0x03, 0x3A, 0x90, 0xF7, 0x69, 0x78, 0x12, 0x83, 0x0B, 0x9A, 0x97, + 0x4D, 0xB7, 0x8C, 0xBF, 0x2D, 0x94, 0xD1, 0x93, 0x2F, 0x42, 0x23, 0xA4, 0xE0, 0x92, 0xDC, 0x68, + 0xD3, 0xDD, 0xAF, 0x91, 0x9F, 0xED, 0x3D, 0x8F, 0xA1, 0x51, 0xD9, 0xE9, 0x70, 0x28, 0xEF, 0xB3, + 0x49, 0xA5, 0x0D, 0xC5, 0xD0, 0x60, 0xB4, 0x2B, 0x07, 0xF8, 0xDF, 0xE6, 0x16, 0xC0, 0x30, 0x71, + 0x85, 0xFD, 0x72, 0x95, 0x29, 0x79, 0x0A, 0x7B, 0x46, 0x11, 0x7D, 0x88, 0x1D, 0x2A, 0x48, 0x1F, + 0x45, 0x89, 0x47, 0xEE, 0xBB, 0xBE, 0x6E, 0xC3, 0x6C, 0xCE, 0x10, 0x5A, 0x2C, 0xCA, 0xFB, 0xB2, + 0xCB, 0x1C, 0x9C, 0xEC, 0x2E, 0x56, 0x59, 0x9B, 0xA6, 0x53, 0xAE, 0x17, 0x25, 0xC1, 0x3F, 0x6A, + 0x0F, 0x09, 0x01, 0xA3, 0xD6, 0xA0, 0xD8, 0x08, 0xE3, 0x74, 0x06, 0x6D, 0x19, 0x98, 0x1E, 0x77, + 0x76, 0xBC, 0xEB, 0x3C, 0xB0, 0xC4, 0xC8, 0x64, 0x0E, 0x86, 0x63, 0xD7, 0xDB, 0xBD, 0xA7, 0x82, + 0x39, 0x4F, 0x27, 0xD2, 0x5F, 0x73, 0xF4, 0x75, 0x6B, 0xC2, 0xD5, 0x67, 0x5D, 0x80, 0xAB, 0x81, + 0xDE, 0xF0, 0xAD, 0xAA, 0xCD, 0xB6, 0xF6, 0x7C, 0xFC, 0x33, 0x05, 0x14, 0x96, 0x15, 0xC9, 0x9E, + 0x35, 0x5C, 0x7E, 0x44, 0x54, 0x58, 0x3B, 0x40, 0x20, 0xA8, 0x8B, 0x5E, 0x4A, 0x24, 0x99, 0x8E, + 0xF5, 0xB5, 0x62, 0x00, 0x37, 0x5B, 0x18, 0x65, 0x8D, 0x32, 0xE2, 0xF9, 0xDA, 0x8A, 0xD4, 0xCC, + 0x26, 0xF2, 0xF1, 0xE7, 0x4B, 0xC6, 0xCF, 0xFF, 0x4C, 0x84, 0x61, 0xFA, 0xB8, 0x0C, 0xA9, 0x50 +] + +fixed = [ + 0x67, 0x23, 0xFE, 0x30, 0x45, 0x33, 0xF8, 0x90, 0x99, 0x21, 0x07, 0xC1, 0xD0, 0x12, 0xB2, 0xA1, 0x07, 0x81 +] + + +def gcd(a, b): + while b > 0: + a, b = b, a % b + return a + +def lcm(a, b): + if not(a) or not(b): + return 1 + return a * b / gcd(a, b) + + +def UpdateHash(db, fwid): + # extract dbid, zero out all hash stuff and add hash indicator + dbid = db[24:32] + hash2 = db[50:70] + z = 20 * b"\0" + db = db[:24] + (8 * b"\0") + db[32:48] + b"\x01\x00" + z + db[70:88] + z + db[108:] + + # convert fwid to byte array + fwid = [int(fwid[i:i+2], 16) for i in range(0, 16, 2)] + + # key generation, step 1: take LCM of each two bytes in the FWID in turn + key = 16 * [0] + for i in (0, 2, 4, 6): + l = int(lcm(*fwid[i:i+2])) + hi = (l & 0xFF00) >> 8 + lo = l & 0x00FF + j = i << 1 + key[j] = ((table1[hi] * 0xB5) - 0x03) & 0xFF + key[j|1] = ((table2[hi] * 0xB7) + 0x49) & 0xFF + key[j|2] = ((table1[lo] * 0xB5) - 0x03) & 0xFF + key[j|3] = ((table2[lo] * 0xB7) + 0x49) & 0xFF + # step 2: invert key + key = [inv[x] for x in key] + # step 3: create hash key + key = fixed + key + key = list(hashlib.sha1(bytes(key)).digest()) + + # first XOR + key = [(x ^ 0x36) for x in key] + 44 * [0x36] + + # first SHA + h = hashlib.sha1(bytes(key) + db).digest() + + # second XOR + key = [(x ^ (0x36 ^ 0x5C)) for x in key] + + # second SHA + h = hashlib.sha1(bytes(key) + h).digest() + + # reassemble database + return db[:24] + dbid + db[32:50] + hash2 + db[70:88] + h + db[108:] + + +################################################################################ + + +if __name__ == "__main__": + import sys + fwids = GetFWIDs() + print("detected FWIDs:", fwids) + if fwids: + fwid = fwids[0] + else: + fwid = "000A27001B3EAD37" + print("no FWID detected, using default FWID for BIST") + + try: + old = open(sys.argv[1], "rb").read() + except IndexError: + sys.exit(0) + new = UpdateHash(old, "000A27001B3EAD37") + print("old =>", " ".join(["%02X" % c for c in old[88:108]])) + print("new =>", " ".join(["%02X" % c for c in new[88:108]])) + if old == new: + print("MATCH!") + else: + print("no match :(") + try: + open(sys.argv[2], "wb").write(new) + except IndexError: + pass diff --git a/iTunesDB.py b/iTunesDB.py new file mode 100644 index 0000000..e6c361c --- /dev/null +++ b/iTunesDB.py @@ -0,0 +1,1027 @@ +#!/usr/bin/env python +# +# iTunesDB generator library for rePear, the iPod database management tool +# Copyright (C) 2006-2008 Martin J. Fiedler +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + +import struct, random, array, sys, os, stat, time +from dataclasses import dataclass +from functools import cmp_to_key +try: + from PIL import Image + PILAvailable = True +except ImportError: + PILAvailable = False + + +def DefaultLoggingFunction(text, force_flush=True): + sys.stdout.write(text) + if force_flush: sys.stdout.flush() +log = DefaultLoggingFunction + +################################################################################ +## some helper classes to represent ITDB records, and some helper functions ## +################################################################################ + +class Field: + def __bytes__(self): raise Exception("abstract function call") + def __len__(self): raise Exception("abstract function call") + +class F_Tag(Field): + def __init__(self, tag: bytes): + self.tag = tag + assert isinstance(tag, bytes) + def __bytes__(self): return self.tag + def __len__(self): return len(self.tag) + +class F_Formatable(Field): + def __init__(self, format, value): + self.format = format + self.value = int(value) + def __bytes__(self): return struct.pack("<"+self.format, self.value) + def __len__(self): return struct.calcsize(self.format) + +class F_Int64(F_Formatable): + def __init__(self, value): F_Formatable.__init__(self, "Q", value) +class F_Int32(F_Formatable): + def __init__(self, value): F_Formatable.__init__(self, "L", value) +class F_Int16(F_Formatable): + def __init__(self, value): F_Formatable.__init__(self, "H", value) +class F_Int8(F_Formatable): + def __init__(self, value): F_Formatable.__init__(self, "B", value) + +class F_HeaderLength(F_Int32): + def __init__(self): F_Int32.__init__(self, 0) +class F_TotalLength(F_Int32): + def __init__(self): F_Int32.__init__(self, 0) +class F_ChildCount(F_Int32): + def __init__(self): F_Int32.__init__(self, 0) + +class F_Padding(Field): + def __init__(self, length): self.length = length + def __bytes__(self): return self.length * b"\0" + def __len__(self): return self.length + +class Record: + def __init__(self, header): + self.header_length_at = None + self.total_length_at = None + self.child_count_at = None + data = b"" + for field in header: + if field.__class__ == F_HeaderLength: self.header_length_at = len(data) + if field.__class__ == F_TotalLength: self.total_length_at = len(data) + if field.__class__ == F_ChildCount: self.child_count_at = len(data) + d = field + if isinstance(d, str): d = d.encode() + elif not isinstance(d, bytes): d = bytes(d) + data += d + if self.header_length_at: data = data[:self.header_length_at] + struct.pack(" val_b: # type: ignore + return 1 + return 0 + +def ifelse(condition, then_val, else_val=None): + if condition: return then_val + else: return else_val + +MAC_TIME_OFFSET = 2082844800 +if time.daylight: tzoffset = time.altzone +else: tzoffset = time.timezone +def unixtime2mactime(t): + if not t: return t + return t + MAC_TIME_OFFSET - tzoffset +def mactime2unixtime(t): + if not t: return t + return t - MAC_TIME_OFFSET + tzoffset + + +# "fuzzy" mtime comparison, allows for two types of slight deviations: +# 1. differences of exact multiples of one hour (usually time zome problems) +# 2. differences of less than 2 seconds (FAT timestamps are imprecise) +def compare_mtime(a, b): + diff = abs(a - b) + if diff > 86402: return False + return ((diff % 3600) in (0, 1, 2, 3598, 3599)) + + +################################################################################ +## some higher-level ITDB record classes ## +################################################################################ + +class StringDataObject(Record): + def __init__(self, mhod_type, content): + if isinstance(content, bytes): encoded = content + else: encoded = content.encode('utf_16_le', 'replace') + Record.__init__(self, ( + F_Tag(b"mhod"), + F_Int32(0x18), + F_TotalLength(), + F_Int32(mhod_type), + F_Padding(8), + F_Int32(1), + F_Int32(len(encoded)), + F_Int32(1), + F_Padding(4) + )) + self.add(encoded) + +class OrderDataObject(Record): + def __init__(self, order): + Record.__init__(self, ( + F_Tag(b"mhod"), + F_Int32(0x18), + F_Int32(0x2C), + F_Int32(100), + F_Padding(8), + F_Int32(order), + F_Padding(16) + )) + + +class TrackItemRecord(Record): + def __init__(self, info): + if not 'id' in info: + raise KeyError("no track ID set") + format = info.get('format', "mp3-cbr") + if info.get('artwork', None): + default_has_artwork = True + default_artwork_size = 1 + else: + default_has_artwork = False + default_artwork_size = 0 + if 'video format' in info: + media_type = 2 + else: + media_type = 1 + Record.__init__(self, ( + F_Tag(b"mhit"), + F_HeaderLength(), + F_TotalLength(), + F_ChildCount(), + F_Int32(info.get('id', 0)), + F_Int32(info.get('visible', 1)), # visible + F_Tag({"mp3": " 3PM", "aac": " CAA", "mp4a": "A4PM"}.get(format[:3], "\0\0\0\0").encode()), + F_Int16({"mp3-cbr": 0x100, "mp3-vbr": 0x101, "aac": 0, "mp4a": 0}.get(format, 0)), + F_Int8(info.get('compilation', 0)), + F_Int8(info.get('rating', 0)), + F_Int32(unixtime2mactime(info.get('mtime', 0))), + F_Int32(info.get('size', 0)), + F_Int32(int(info.get('length', 0) * 1000)), + F_Int32(info.get('track number', 0)), + F_Int32(info.get('total tracks', 0)), + F_Int32(info.get('year', 0)), + F_Int32(info.get('bitrate', 0)), + F_Int16(0), + F_Int16(info.get('sample rate', 0)), + F_Int32(info.get('volume', 0)), + F_Int32(info.get('start time', 0)), + F_Int32(info.get('stop time', 0)), + F_Int32(info.get('soundcheck', 0)), + F_Int32(info.get('play count', 0)), + F_Int32(0), + F_Int32(unixtime2mactime(info.get('last played time', 0))), + F_Int32(info.get('disc number', 0)), + F_Int32(info.get('total discs', 0)), + F_Int32(info.get('user id', 0)), + F_Int32(info.get('date added', 0)), + F_Int32(int(info.get('bookmark time', 0) * 1000)), + F_Int64(info.get('dbid', 0)), + F_Int8(info.get('checked', 0)), + F_Int8(info.get('application rating', 0)), + F_Int16(info.get('BPM', 0)), + F_Int16(info.get('artwork count', 1)), + F_Int16({"wave": 0, "audible": 1}.get(format, 0xFFFF)), + F_Int32(info.get('artwork size', default_artwork_size)), + F_Int32(0), + F_Formatable("f", info.get('sample rate', 0)), + F_Int32(info.get('release date', 0)), + F_Int16({"aac": 0x0033, "mp4a": 0x0033, "audible": 0x0029, "wave:": 0}.get(format, 0x0C)), + F_Int16(info.get('explicit flag', 0)), + F_Padding(8), + F_Int32(info.get('skip count', 0)), + F_Int32(unixtime2mactime(info.get('last skipped time', 0))), + F_Int8(2 - int(info.get('has artwork', default_has_artwork))), + F_Int8(not info.get('shuffle flag', 1)), + F_Int8(info.get('bookmark flag', 0)), + F_Int8(info.get('podcast flag', 0)), + F_Int64(info.get('dbid', 0)), + F_Int8(info.get('lyrics flag', 0)), + F_Int8(info.get('movie flag', 0)), + F_Int8(info.get('played mark', 1)), + F_Padding(9), + F_Int32(ifelse(format[:3]=="mp3", 0, info.get('sample count', 0))), + F_Padding(16), + F_Int32(media_type), + F_Int32(0), # season number + F_Int32(0), # episode number + F_Padding(28), + F_Int32(info.get('gapless data', 0)), + F_Int32(0), + F_Int16(info.get('gapless track flag', 0)), + F_Int16(info.get('gapless album flag', 0)), + F_Padding(20), # hash + F_Padding(18), # misc unknowns + F_Int16(info.get('album id', 0)), + F_Padding(52), # padding before mhii link + F_Int32(info.get('mhii link', 0)) + )) + for mhod_type, key in ((1,'title'), (4,'artist'), (3,'album'), (5,'genre'), (6,'filetype'), (2,'path')): + if key in info: + value = info[key] + if key=="path": + value = ":" + value.replace("/", ":").replace("\\", ":") + self.add(StringDataObject(mhod_type, value)) + + +class PlaylistItemRecord(Record): + def __init__(self, order, trackid, timestamp=0): + Record.__init__(self, ( + F_Tag(b"mhip"), + F_HeaderLength(), + F_TotalLength(), + F_ChildCount(), + F_Int32(0), + F_Int32((trackid + 0x1337) & 0xFFFF), + F_Int32(trackid), + F_Int32(timestamp), + F_Int32(0), + F_Padding(40) + )) + self.add(OrderDataObject(order)) + + +class PlaylistRecord(Record): + def __init__(self, name, track_count, order=0, master=0, timestamp=0, plid=None, sort_order=1): + if not plid: plid = random.randrange(0, 18446744073709551615) + Record.__init__(self, ( + F_Tag(b"mhyp"), + F_HeaderLength(), + F_TotalLength(), + F_ChildCount(), + F_Int32(track_count), + F_Int32(master), + F_Int32(timestamp), + F_Int64(plid), + F_Int32(0), + F_Int16(1), + F_Int16(0), + F_Int32(sort_order), + F_Padding(60) + )) + self.add(StringDataObject(1, name)) + self.add(OrderDataObject(order)) + + def add_index(self, tracklist, index_type, fields): + order = list(range(len(tracklist))) + order.sort(key=cmp_to_key(lambda a, b: compare_dict(tracklist[a], tracklist[b], fields))) + mhod = Record(( + F_Tag(b"mhod"), + F_Int32(24), + F_TotalLength(), + F_Int32(52), + F_Padding(8), + F_Int32(index_type), + F_Int32(len(order)), + F_Padding(40) + )) + arr = array.array('L', order) + # the array module doesn't directly support endianness, so we detect + # the machine's endianness and swap if it is big-endian + if array.array('L', [1]).tobytes()[3] == 1: + arr.byteswap() + data = bytes(arr) + mhod.add(data) + self.add(mhod) + + def set_playlist(self, track_ids): + for i in range(len(track_ids)): + self.add(PlaylistItemRecord(i+1, track_ids[i]), 0) + + + +################################################################################ +## the toplevel ITDB class ## +################################################################################ + +class iTunesDB: + def __init__(self, tracklist, name="Unnamed", dbid=None, dbversion=0x19): + if not dbid: dbid = random.randrange(0, 18446744073709551615) + + self.mhbd = Record(( + F_Tag(b"mhbd"), + F_HeaderLength(), + F_TotalLength(), + F_Int32(0), + F_Int32(dbversion), + F_ChildCount(), + F_Int64(dbid), + F_Int16(2), + F_Padding(14), + F_Int16(0), # hash indicator (set later by hash58) + F_Padding(20), # first hash + F_Tag(b"en"), # language = 'en' + F_Tag(b"\0rePear!"), # library persistent ID + F_Padding(20), # hash58 + F_Padding(80) + )) + + self.mhsd = Record(( + F_Tag(b"mhsd"), + F_HeaderLength(), + F_TotalLength(), + F_Int32(1), + F_Padding(80) + )) + self.mhlt = Record(( + F_Tag(b"mhlt"), + F_HeaderLength(), + F_ChildCount(), + F_Padding(80) + )) + + for track in tracklist: + self.mhlt.add(TrackItemRecord(track)) + + self.mhsd.add(self.mhlt) + del self.mhlt + self.mhbd.add(self.mhsd) + + self.mhsd = Record(( + F_Tag(b"mhsd"), + F_HeaderLength(), + F_TotalLength(), + F_Int32(2), + F_Padding(80) + )) + self.mhlp = Record(( + F_Tag(b"mhlp"), + F_HeaderLength(), + F_ChildCount(), + F_Padding(80) + )) + + mhyp = PlaylistRecord(name, len(tracklist), master=1, sort_order=10) + mhyp.add_index(tracklist, 0x03, ('title',)) + mhyp.add_index(tracklist, 0x04, ('album','disc number','track number','title')) + mhyp.add_index(tracklist, 0x05, ('artist','album','disc number','track number','title')) + mhyp.add_index(tracklist, 0x07, ('genre','artist','album','disc number','track number','title')) + mhyp.add_index(tracklist, 0x12, ('composer','title')) + mhyp.set_playlist([track['id'] for track in tracklist]) + self.mhlp.add(mhyp) + + def add_playlist(self, tracks, name="Unnamed"): + mhyp = PlaylistRecord(name, len(tracks), sort_order=1) + mhyp.set_playlist([track['id'] for track in tracks]) + self.mhlp.add(mhyp) + + def finish(self): + self.mhsd.add(self.mhlp) + del self.mhlp + self.mhbd.add(self.mhsd) + del self.mhsd + result = self.mhbd.__bytes__() + del self.mhbd + return result + + + +################################################################################ +## ArtworkDB / PhotoDB record classes ## +################################################################################ + +class RGB565_LE: + bpp = 16 + @staticmethod + def convert(data): + res = array.array('B', [0 for x in range(len(data)//3*2)]) + io = 0 + for ii in range(0, len(data), 3): + g = ord(data[ii+1]) >> 2 + res[io] = ((g & 7) << 5) | (ord(data[ii+2]) >> 3) + res[io|1] = (ord(data[ii]) & 0xF8) | (g >> 3) + io += 2 + return str(res) + convert = staticmethod(convert) + +ImageFormats = { + 'nano': ((1027, 100, 100, RGB565_LE), + (1031, 42, 42, RGB565_LE)), + 'photo': ((1016, 140, 140, RGB565_LE), + (1017, 56, 56, RGB565_LE)), + 'video': ((1028, 100, 100, RGB565_LE), + (1029, 200, 200, RGB565_LE)), + 'nano3g': ((1055, 128, 128, RGB565_LE), + (1060, 320, 320, RGB565_LE), + (1061, 55, 56, RGB565_LE)), + 'nano4g': ((1055, 128, 128, RGB565_LE), + (1078, 80, 80, RGB565_LE), + (1071, 240, 240, RGB565_LE), + (1074, 50, 50, RGB565_LE)), + '4g': 'photo', + '5g': 'video', + '6g': 'nano3g', + 'classic': 'nano3g', + 'nano1g': 'nano', + 'nano2g': 'nano', +} + +@dataclass +class ImageInfo: + format: object = None + index: int = 0 + sx: int = 0 + sy: int = 0 + mx: int = 0 + my: int = 0 + +class ArtworkFormat: + def __init__(self, descriptor, cache_info=(0,0)): + self.fid, self.height, self.width, self.format = descriptor + self.filename = "F%04d_1.ithmb" % self.fid + self.size = self.width * self.height * self.format.bpp/8 + self.fullname = "iPod_Control/Artwork/" + self.filename + + # check if the cache file can be used + try: + s = os.stat(self.fullname) + use_cache = stat.S_ISREG(s[stat.ST_MODE]) \ + and compare_mtime(cache_info[0], s[stat.ST_MTIME]) \ + and (s[stat.ST_SIZE] == cache_info[1]) + except OSError: + use_cache = False + + # load the cache + if use_cache: + try: + f = open(self.fullname, "rb") + self.cache = f.read() + f.close() + except IOError: + use_cache = False + if not use_cache: + self.cache = None + + # open the destination file + try: + self.f = open(self.fullname, "wb") + except IOError as e: + log("WARNING: Error opening the artwork data file `%s'\n", self.filename) + self.f = None + + def close(self): + if self.f: + self.f.close() + try: + s = os.stat(self.fullname) + cache_info = (s[stat.ST_MTIME], s[stat.ST_SIZE]) + except OSError: + cache_info = (0, 0) + return (self.fid, cache_info) + + def GenerateImage(self, image, index, cache_entry=None): + if cache_entry and self.cache: + offset = self.size * cache_entry['index'] + data = self.cache[offset : offset+self.size] + sx = cache_entry['dim'][self.fid]['sx'] + sy = cache_entry['dim'][self.fid]['sy'] + mx = cache_entry['dim'][self.fid]['mx'] + my = cache_entry['dim'][self.fid]['my'] + else: + log(" [%dx%d]" % (self.width, self.height), True) + + # sx/sy = resulting image size + sx = self.width + sy = image.size[1] * sx / image.size[0] + if sy > self.height: + sy = self.height + sx = image.size[0] * sy / image.size[1] + # mx/my = margin size + mx = self.width - sx + my = self.height - sy + + # process the image + temp = image.resize((sx, sy), Image.Resampling.LANCZOS) + thumb = Image.new('RGB', (self.width, self.height), (255, 255, 255)) + thumb.paste(temp, (mx/2, my/2)) + del temp + data = self.format.convert(thumb.tobytes()) + del thumb + + # save the image + try: + assert self.f + self.f.seek(self.size * index) + self.f.write(data) + except IOError: + log(" [WRITE ERROR]", True) + + # return image metadata + iinfo = ImageInfo() + iinfo.format = self + iinfo.index = index + iinfo.sx = sx + iinfo.sy = sy + iinfo.mx = mx + iinfo.my = my + return iinfo + + + +class ArtworkDBStringDataObject(Record): + def __init__(self, mhod_type, content): + if isinstance(content, bytes): + content = content.decode(sys.getfilesystemencoding(), 'replace') + elif not isinstance(content, str): + content = str(content) + content = content.encode('utf_16_le', 'replace') + padding = len(content) % 4 + if padding: padding = 4 - padding + Record.__init__(self, ( + F_Tag(b"mhod"), + F_Int32(0x18), + F_TotalLength(), + F_Int16(mhod_type), + F_Int16(padding), + F_Padding(8), + F_Int32(len(content)), + F_Int32(2), + F_Int32(0) + )) + self.add(content) + if padding: + self.add("\0" * padding) + + +class ImageDataObject(Record): + def __init__(self, iinfo): + Record.__init__(self, ( + F_Tag(b"mhod"), + F_Int32(0x18), + F_TotalLength(), + F_Int32(2), + F_Padding(8) + )) + + mhni = Record(( + F_Tag(b"mhni"), + F_Int32(0x4C), + F_TotalLength(), + F_ChildCount(), + F_Int32(iinfo.format.fid), + F_Int32(iinfo.format.size * iinfo.index), + F_Int32(iinfo.format.size), + F_Int16(iinfo.my), + F_Int16(iinfo.mx), + F_Int16(iinfo.sy), + F_Int16(iinfo.sx), + F_Padding(4), + F_Int32(iinfo.format.size), + F_Padding(32) + )) + + mhod = ArtworkDBStringDataObject(3, ":" + iinfo.format.filename) + mhni.add(mhod) + self.add(mhni) + + +class ImageItemRecord(Record): + def __init__(self, img_id, dbid, iinfo_list, orig_size=0): + Record.__init__(self, ( + F_Tag(b"mhii"), + F_Int32(0x98), + F_TotalLength(), + F_ChildCount(), + F_Int32(img_id), + F_Int64(dbid), + F_Padding(20), + F_Int32(orig_size), + F_Padding(100) + )) + + for iinfo in iinfo_list: + self.add(ImageDataObject(iinfo)) + + +def ArtworkDB(model, imagelist, base_id=0x40, cache_data=({}, {})): + while isinstance(ImageFormats.get(model, None), str): + model = ImageFormats[model] + if not model in ImageFormats: + return None + + format_cache, image_cache = cache_data + formats = [] + for descriptor in ImageFormats[model]: + formats.append(ArtworkFormat(descriptor, + cache_info = format_cache.get(descriptor[0], (0,0)))) + # if there's at least one format whose image file isn't cache-clean, + # invalidate the cache + if not formats[-1].cache: + image_cache = {} + + # Image List + mhsd = Record(( + F_Tag(b"mhsd"), + F_HeaderLength(), + F_TotalLength(), + F_Int32(1), + F_Padding(80) + )) + mhli = Record(( + F_Tag(b"mhli"), + F_HeaderLength(), + F_ChildCount(), + F_Padding(80) + )) + + img_id = base_id + index = 0 + output_image_cache = {} + image_count = 0 + dbid2mhii = {} + for source, dbid_list in imagelist.items(): + log(source, False) + + # stat this image + try: + s = os.stat(source) + except OSError as e: + log(" [Error: %s]\n" % e.strerror, True) + continue + + # check if the image is cacheworthy + cache_entry = image_cache.get(source, None) + if cache_entry: + if (cache_entry['size'] != s[stat.ST_SIZE]) \ + or not(compare_mtime(cache_entry['mtime'], s[stat.ST_MTIME])): + cache_entry = None + + # if it's not cached, open the image + if not cache_entry: + try: + image = Image.open(source) + image.tobytes() + except IOError as e: + log(" [Error: %s]\n" % e, True) + continue + else: + log(" [cached]", True) + image = None + + # generate the image data and ArtworkDB records + iinfo_list = [format.GenerateImage(image, index, cache_entry) for format in formats] + for dbid in dbid_list: + mhli.add(ImageItemRecord(img_id, dbid, iinfo_list, s[stat.ST_SIZE])) + dbid2mhii[dbid] = img_id + img_id += 1 + del image + + # add the image into the new cache + dim = {} + for iinfo in iinfo_list: + dim[iinfo.format.fid] = { + 'sx': iinfo.sx, + 'sy': iinfo.sy, + 'mx': iinfo.mx, + 'my': iinfo.my + } + output_image_cache[source] = { + 'index': index, + 'size': s[stat.ST_SIZE], + 'mtime': s[stat.ST_MTIME], + 'dim': dim + } + + # done with this image + del iinfo_list + index += 1 + image_count += len(dbid_list) + log(" [OK]\n", True) + + # Date File Header + mhfd = Record(( + F_Tag(b"mhfd"), + F_HeaderLength(), + F_TotalLength(), + F_Int32(0), + F_Int32(2), + F_Int32(3), + F_Int32(0), + F_Int32(base_id + image_count), + F_Padding(16), + F_Int32(2), + F_Padding(80) + )) + + mhsd.add(mhli) + mhfd.add(mhsd) + + # Album List (dummy) + mhsd = Record(( + F_Tag(b"mhsd"), + F_HeaderLength(), + F_TotalLength(), + F_Int32(2), + F_Padding(80) + )) + mhsd.add(Record(( + F_Tag(b"mhla"), + F_HeaderLength(), + F_Int32(0), + F_Padding(80) + ))) + mhfd.add(mhsd) + + # File List + mhsd = Record(( + F_Tag(b"mhsd"), + F_HeaderLength(), + F_TotalLength(), + F_Int32(3), + F_Padding(80) + )) + + mhlf = Record(( + F_Tag(b"mhlf"), + F_HeaderLength(), + F_Int32(len(formats)), + F_Padding(80) + )) + + for format in formats: + mhlf.add(Record(( + F_Tag(b"mhif"), + F_HeaderLength(), + F_TotalLength(), + F_Int32(0), + F_Int32(format.fid), + F_Int32(format.size), + F_Padding(100) + ))) + + # finalize ArtworkDB + mhsd.add(mhlf) + mhfd.add(mhsd) + output_format_cache = dict([format.close() for format in formats]) + del formats + output_cache_data = (output_format_cache, output_image_cache) + return (str(mhfd), output_cache_data, dbid2mhii) + + +################################################################################ +## a rudimentary ITDB reader (only reads titles, no playlists, and isn't very ## +## fault-tolerant) for the "dissect" action ## +################################################################################ + +mhod_type_map = { + 1: 'title', + 2: 'path', + 3: 'album', + 4: 'artist', + 5: 'genre', + 6: 'filetype', + 8: 'comment', + 12: 'composer' +} + +class InvalidFormat(Exception): pass + +class DatabaseReader: + def __init__(self, f="iPod_Control/iTunes/iTunesDB"): + if isinstance(f, str): + f = open(f, "rb") + self.f = f + self._skip_header("mhbd") + while True: + h = self._skip_header("mhsd") + if len(h) < 16: + raise InvalidFormat + size, mhsd_type = struct.unpack(' yeah! + if size < len(h): + raise InvalidFormat + self.f.seek(size - len(h), 1) + self._skip_header("mhlt") + + def _skip_header(self, tag): # a little helper function + hh = self.f.read(8) + if (len(hh) != 8) or (hh[:4] != tag): + raise InvalidFormat + size = struct.unpack(' 40) and (data[:4] == "mhod"): + size, mhod_type = struct.unpack('> 16, (x >> 8) & 0xFF, x & 0xFF) + +SD_type_map = { "aac": 2, "mp4a": 2, "wave": 4} + +def MakeSDEntry(info): + path = info['path'] + + if isinstance(path, bytes): + path = path.decode(sys.getfilesystemencoding(), 'replace') + elif not isinstance(path, str): + path = str(path) + + path_bytes = ('/' + path).encode("utf_16_le", 'replace') + + return b"\0\x02\x2E\x5A\xA5\x01" + (20 * b"\0") + b"\x64\0\0" + bytes([SD_type_map.get(info.get('type', None), 1)]) + b"\0\x02\0" + \ + path_bytes + (261 * 2 - len(path_bytes)) * b"\0" + bytes([info.get('shuffle flag', 1), info.get('bookmark flag', 0), 0]) + + +def iTunesSD(tracklist): + header = b"\0\x02\x2E\x5A\xA5\x01" + (20*b"\0") + b"\x64\0\0\0x01\0\0x02\0" + return be3(len(tracklist)) + b"\x01\x06\0\0\0\x12" + (9*b"\0") + \ + b"".join(map(MakeSDEntry, tracklist)) + + +################################################################################ +## some useful helper functions for "fine tuning" of track lists ## +################################################################################ + +def GenerateIDs(tracklist): + trackid = random.randint(0, (0xFFFF-0x1337) - len(tracklist)) + dbid = random.randrange(0, 18446744073709551615 - len(tracklist)) + for track in tracklist: + track['id'] = trackid + track['dbid'] = dbid + trackid += 1 + dbid += 1 + + +def GuessTitleAndArtist(filename): + info = {} + filename = os.path.split(filename)[1] + filename = os.path.splitext(filename)[0] + filename = filename.replace('_', ' ') + n = "" + for i in range(len(filename)): + c = filename[i] + if c in "0123456789": + n += c + continue + if c in " -": + if n: info['track number'] = int(n) + filename = filename[i+1:] + break + parts = filename.split(' - ', 1) + if len(parts)==2: + info['artist'] = parts[0].strip() + info['title'] = parts[1].strip(" -\r\n\t\v") + else: + info['title'] = filename.strip() + return info + +def FillMissingTitleAndArtist(track_or_list): + if isinstance(track_or_list, list): + for track in track_or_list: + FillMissingTitleAndArtist(track) + else: + if track_or_list.get('title',None) and track_or_list.get('artist',None): + return # no need to do something, it's fine already + guess = GuessTitleAndArtist(track_or_list['path']) + for key in ('title', 'artist', 'track number'): + if not(track_or_list.get(key,None)) and guess.get(key,None): + track_or_list[key] = guess[key] + + +################################################################################ +## some additional general purpose helper functions ## +################################################################################ + +def ASCIIMap(c): + if ord(c) < 32: return "." + if ord(c) == 127: return "." + return c + +def DisplayTitle(info): + s = info.get('title', "") + if 'album' in info: s = "%s -> %s" % ((info['album']), s) + if 'artist' in info: s = "%s: %s" % ((info['artist']), s) + q = [str((info[key])) for key in ('genre','year') if key in info] + if q: s = "%s [%s]" % (s, ", ".join(q)) + return s \ No newline at end of file diff --git a/mp3info.py b/mp3info.py new file mode 100644 index 0000000..b052c45 --- /dev/null +++ b/mp3info.py @@ -0,0 +1,582 @@ +#!/usr/bin/env python +# +# audio file information library for rePear, the iPod database management tool +# Copyright (C) 2006-2008 Martin J. Fiedler +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + +import sys, re, zlib, struct, os, stat +import qtparse + + +################################################################################ +## a sh*tload of constants ## +################################################################################ + +ID3v1Genres = { 0:"Blues", 1:"Classic Rock", 2:"Country", 3:"Dance", 4:"Disco", +5:"Funk", 6:"Grunge", 7:"Hip-Hop", 8:"Jazz", 9:"Metal", 10:"New Age", +11:"Oldies", 12:"Other", 13:"Pop", 14:"R&B", 15:"Rap", 16:"Reggae", 17:"Rock", +18:"Techno", 19:"Industrial", 20:"Alternative", 21:"Ska", 22:"Death Metal", +23:"Pranks", 24:"Soundtrack", 25:"Euro-Techno", 26:"Ambient", 27:"Trip-Hop", +28:"Vocal", 29:"Jazz&Funk", 30:"Fusion", 31:"Trance", 32:"Classical", +33:"Instrumental", 34:"Acid", 35:"House", 36:"Game", 37:"Sound Clip", +38:"Gospel", 39:"Noise", 40:"Alternative Rock", 41:"Bass", 42:"Soul", +43:"Punk", 44:"Space", 45:"Meditative", 46:"Instrumental Pop", +47:"Instrumental Rock", 48:"Ethnic", 49:"Gothic", 50:"Darkwave", +51:"Techno-Industrial", 52:"Electronic", 53:"Pop-Folk", 54:"Eurodance", +55:"Dream", 56:"Southern Rock", 57:"Comedy", 58:"Cult", 59:"Gangsta", +60:"Top 40", 61:"Christian Rap", 62:"Pop/Funk", 63:"Jungle", 64:"Native US", +65:"Cabaret", 66:"New Wave", 67:"Psychedelic", 68:"Rave", 69:"Showtunes", +70:"Trailer", 71:"Lo-Fi", 72:"Tribal", 73:"Acid Punk", 74:"Acid Jazz", +75:"Polka", 76:"Retro", 77:"Musical", 78:"Rock & Roll", 79:"Hard Rock", +80:"Folk", 81:"Folk-Rock", 82:"National Folk", 83:"Swing", 84:"Fast Fusion", +85:"Bebop", 86:"Latin", 87:"Revival", 88:"Celtic", 89:"Bluegrass", +90:"Avantgarde", 91:"Gothic Rock", 92:"Progressive Rock", +93:"Psychedelic Rock", 94:"Symphonic Rock", 95:"Slow Rock", 96:"Big Band", +97:"Chorus", 98:"Easy Listening", 99:"Acoustic", 100:"Humour", 101:"Speech", +102:"Chanson", 103:"Opera", 104:"Chamber Music", 105:"Sonata", 106:"Symphony", +107:"Booty Bass", 108:"Primus", 109:"Porn Groove", 110:"Satire", +111:"Slow Jam", 112:"Club", 113:"Tango", 114:"Samba", 115:"Folklore", +116:"Ballad", 117:"Power Ballad", 118:"Rhytmic Soul", 119:"Freestyle", +120:"Duet", 121:"Punk Rock", 122:"Drum Solo", 123:"Acapella", 124:"Euro-House", +125:"Dance Hall", 126:"Goa", 127:"Drum & Bass", 128:"Club-House", +129:"Hardcore", 130:"Terror", 131:"Indie", 132:"BritPop", 133:"Negerpunk", +134:"Polsk Punk", 135:"Beat", 136:"Christian Gangsta", 137:"Heavy Metal", +138:"Black Metal", 139:"Crossover", 140:"Contemporary Christian", +141:"Christian Rock", 142:"Merengue", 143:"Salsa", 144:"Thrash Metal", +145:"Anime", 146:"JPop", 147:"SynthPop" } + + +ID3v2FrameMap = { + "TIT1": "content group", + "TIT2": "title", + "TIT3": "subtitle", + "TALB": "album", + "TOAL": "original album", + "TRCK": "/track number/total tracks", + "TPOS": "/disc number/total discs", + "TPE1": "artist", + "TPE2": "band", + "TPE3": "conductor", + "TPE4": "interpreted by", + "TOPE": "original artist", + "TEXT": "lyrics", + "TOLY": "original lyrics", + "TCOM": "composer", + "TMCL": "musician credits", + "TIPL": "involved people", + "TENC": "encoded by", + "TBPM": "#BPM", + "TYER": "#year", +# "TLEN": "length", # unreliable, rather use Xing/FhG tags or scan the file + "TKEY": "initial key", + "TLAN": "language", + "TCON": "genre", + "TFLT": "file type", + "TMED": "media type", + "TMOO": "mood", + "TCOP": "copyright", + "TPRO": "produced", + "TPUB": "publisher", + "TOWN": "owner", + "TRSN": "station name", + "TRSO": "station owner", + "TOFN": "original file name", + "TDLY": "playlist delay", + "TDEN": "encoding time", + "TDOR": "original release time", + "TDRC": "recording time", + "TDRL": "release time", + "TDTG": "tagging time", + "TSSE": "encoding settings", + "TSOA": "album sort order", + "TSOP": "performer sort order", + "TSOT": "title sort order", + "WCOM": "commercial information URL", + "WCOP": "copyright URL", + "WOAF": "audio file URL", + "WOAR": "artist URL", + "WOAS": "audio source URL", + "WORS": "station URL", + "WPAY": "payment URL", + "WPUB": "publisher URL", + "COMM": "comment" +} + +RE_ID3v2_Frame_Type = re.compile(r'[A-Z0-9]{4}') +RE_ID3v2_Strip_Genre = re.compile(r'\([0-9]+\)(.*)') + + +################################################################################ +## ID3v1 decoder ## +################################################################################ + +def GetID3v1(f, info): + try: + f.seek(-128, 2) + data = f.read(128) + except IOError: + return 0 + if len(data) != 128 or data[:3] != b"TAG": + return 0 + info['tag'] = "id3v1" + field = data[3:33].split(b"\0", 1)[0].strip() + if field: + info['title'] = field.decode('latin-1', 'replace') + field = data[33:63].split(b"\0", 1)[0].strip() + if field: + info['artist'] = field.decode('latin-1', 'replace') + field = data[63:93].split(b"\0", 1)[0].strip() + if field: + info['album'] = field.decode('latin-1', 'replace') + field = data[93:97].split(b"\0", 1)[0].strip() + if field: + try: + info['year'] = int(field.decode('latin-1')) + except ValueError: + pass + field = data[97:127].split(b"\0", 1)[0].strip() + if field: + info['comment'] = field.decode('latin-1', 'replace') + if data[125] == 0 and data[126] != 0: + info['track number'] = data[126] + try: + info['genre'] = ID3v1Genres[data[127]] + except KeyError: + pass + return -128 + + +################################################################################ +## ID3v2 decoder ## +################################################################################ + +def DecodeInteger(s): + res = 0 + for c in s: + res = (res << 8) | ord(c) + return res + +def DecodeSyncsafeInteger(s): + res = 0 + for c in s: + res = (res << 7) | (ord(c) & 0x7F) + return res + + +def GetCharset(encoding): + if encoding=="\1": return "utf_16" + if encoding=="\2": return "utf_16_be" + if encoding=="\3": return "utf_8" + else: return "iso-8859-1" + + +def GetEndID3v2(f, offset=0): + try: + f.seek(offset-10, 2) + marker = f.read(10) + if len(marker)!=10 or marker[:3]!="3DI": + return None + size = DecodeSyncsafeInteger(marker[-4:]) + 10 + f.seek(offset-10-size, 2) + data = f.read(size) + if len(data)!=size or data[:3]!="ID3": + return None + return data + except IOError: + return None + + +def GetStartID3v2(f): + try: + f.seek(0) + marker = f.read(10) + if len(marker)!=10 or marker[:3]!="ID3": + return None + size = DecodeSyncsafeInteger(marker[-4:]) + payload = f.read(size) + if len(payload)!=size: + return None + return marker+payload + except IOError: + return None + + +def DecodeID3v2(data, info): + info['tag'] = "id3v2.%d.%d" % (ord(data[3]), ord(data[4])) + if ord(data[3]) >= 4: + decode_size = DecodeSyncsafeInteger + else: + decode_size = DecodeInteger + + # parse header flags, strip header(s) + flags = ord(data[5]) + data = data[10:] + if flags & 0x40: # extended header + size = decode_size(data[:4]) + data = data[size:] + + # parse frames + while len(data)>=10: + frame = data[:4] + if not RE_ID3v2_Frame_Type.match(frame): + break # invalid frame name or start of padding => bail out + size = decode_size(data[4:8]) + payload = data[10:size+10] + flags = ord(data[9]) + if flags & 0x02: + payload = payload.replace("\xff\0", "\xff") + if flags & 0x04: + try: + payload = zlib.decompress(payload) + except zlib.error: + continue # this frame is broken + HandleID3v2Frame(frame, payload, flags, info) + data = data[size+10:] + +def HandleID3v2Frame(frame, payload, flags, info): + text = None + if not payload: return # empty payload + + if frame[0] == 'T' and frame != "TXXX": + # text frame + charset = GetCharset(payload[0]) + text = payload[1:].decode(charset, 'replace').split('\0', 1)[0] + + elif frame[0] == 'W' and frame != "WXXX": + # URL + text = payload.split(b"\0", 1)[0].decode("iso-8859-1", 'replace') + + elif frame == "COMM": + # comment + charset = GetCharset(payload[0]) + lang = payload[1:4].split(b"\0", 1)[0] + parts = payload[4:].decode(charset, 'replace').split('\0', 2) + if len(parts) < 2: return # broken frame + text = parts[1] + + if text: ##### apply the current textual frame #### + key = ID3v2FrameMap.get(frame, frame) + text = text.strip() + + if frame == "TCON": # strip crappy numerical genre comment + m = RE_ID3v2_Strip_Genre.match(text.encode('iso-8859-1', 'replace')) + if m: text = m.group(1) + + if key[0] == "#": # numerical key + try: + text = str(int(text.strip())) + except ValueError: + return # broken frame + key = key[1:] + + if key[0] == "/": # multipart numerical key + keys = key[1:].split("/") + values = text.split("/") + for key, value in zip(keys, values): + try: + info[key] = int(value) + except: + pass + return # already done here + + info[key] = text + + +################################################################################ +## ultra-simple (and not very fault-tolerant) Ogg Vorbis metadata decoder ## +################################################################################ + +def DecodeVorbisHeader(f, info): + try: + f.seek(0) + data = f.read(4096) # almost one page, should be enough + except IOError: + return False + if data[:4] != b"OggS": return False # no Ogg -- don't bother + data = data.split(b"vorbis", 3) + if len(data) != 4: return False # no Vorbis packets + info['format'] = "ogg" # at this point, we can assume the stream is valid + info['filetype'] = "Ogg Vorbis" + data = data[2] + if len(data) < 8: return True # comment packet too short + + # encoder version + size = struct.unpack("> 3) & 1 + samples = 576 * (version + 1) + b2 = header[2] + bitrate = mp3_bitrates[version][b2 >> 4] + samplerate = mp3_samplerates[version][(b2 >> 2) & 3] + padding = (b2 >> 1) & 1 + framesize = 72000 * (version + 1) * bitrate // samplerate + padding + + # skip frame data + try: + frame = f.read(framesize - 4) + # accumulate the data of the first 10 frames + if total_frames < 10: + data += frame + except IOError: + break + + # fix statistics + total_samples += samples + total_frames += 1 + total_bytes += framesize + used_bitrates[bitrate] = None + + # after 10 frames, check for Xing/LAME/FhG headers + if total_frames == 10: + valid = False + # check for Xing/LAME VBR header + p2 = data.find(b"Xing\0\0\0") + if (p2 > 0) and (data[p2 + 7] & 1): + force_vbr = True + # check for LAME CBR header + p = data.find(b"Info\0\0\0") + if force_vbr or ((p > 0) and (data[p + 7] & 1)): + if force_vbr: p = p2 + total_frames, total_bytes = struct.unpack(">ii", data[p+8:p+16]) + if not(data[p + 7] & 2): + total_bytes = info['size'] # size not specified, estimate + total_samples = total_frames * samples + valid = True + # check for FhG header + else: + p = data.find(b"VBRI\0\1") + if p > 0: + force_vbr = True + total_bytes, total_frames = struct.unpack(">ii", data[p+10:p+18]) + total_samples = total_frames * samples + valid = True + # final sanity check + if valid: + if (total_frames < 10) or (total_bytes < 1000) or (total_bytes > info['size']): + valid = False + if valid: + # verify computed bitrate + check_bitrate = total_bytes * 8 * 0.001 / total_frames / samples * samplerate + if force_vbr: + # valid range for VBR files: all the way through + min_rate = 30.0 + max_rate = 330.0 + else: + # valid range for CBR files: current bitrate +/- 10% + min_rate = bitrate * 0.9 + max_rate = bitrate * 1.1 + valid = (check_bitrate > min_rate) and (check_bitrate < max_rate) + if valid: + break + else: + # this didn't work out, continue conventionally + total_samples = 10 * samples + total_frames = 10 + total_bytes = 0 + force_vbr = False + + # scan complete, finish things + if total_frames < 10: + return False # less than 10 frames? that's a little bit short ... + info['filetype'] = "MPEG-%d Audio Layer 3" % (2 - version) + info['sample rate'] = samplerate + info['sample count'] = total_samples + info['length'] = total_samples * 1.0 / samplerate + if force_vbr or (len(used_bitrates) > 1): + info['format'] = "mp3-vbr" + info['bitrate'] = int(total_bytes * 8 * 0.001 / info['length']) + else: + info['format'] = "mp3-cbr" + info['bitrate'] = bitrate + return True + + +################################################################################ +## MP4 decoder wrapper ## +################################################################################ + +def DecodeMP4(f, info): + try: + f.seek(0) + first_atom = f.read(8)[4:] + except IOError: + return False + if not(first_atom in ('moov', 'ftyp')): + return False # no MP4 file + try: + qt = qtparse.QTParser(f) + except IOError: + return False + info.update(qt.get_repear_info()) + del qt + return True + + +################################################################################ +## toplevel GetAudioFileInfo() function ## +################################################################################ + +def GetAudioFileInfo(filename, stat_only=False): + try: + s = os.stat(filename) + except OSError: + return None + + if not stat.S_ISREG(s[stat.ST_MODE]): + return None + info = {'path': filename, 'size':s[stat.ST_SIZE], 'mtime':s[stat.ST_MTIME]} + if stat_only: return info + + # try to extract a track number from the file name + track = 0 + for c in os.path.split(filename)[-1]: + if c in "0123456789": + track = (10 * track) + ord(c) - 48 + else: + break + if track: + info['track number'] = track + + # open the file + try: + f = open(filename, "rb") + except IOError: + return None + + # MP4 probing + if DecodeMP4(f, info): + return info + + # Ogg Vorbis probing + if DecodeVorbisHeader(f, info): + return info + + # some ID3 probing + end_offset = GetID3v1(f, info) + id3v2_data = GetEndID3v2(f, end_offset) + if id3v2_data: DecodeID3v2(id3v2_data, info) + id3v2_data = GetStartID3v2(f) + if id3v2_data: + start_offset = len(id3v2_data) + DecodeID3v2(id3v2_data, info) + else: start_offset = 0 + ScanMP3(f, info, start_offset) + f.close() + return info \ No newline at end of file diff --git a/qtparse.py b/qtparse.py new file mode 100644 index 0000000..aef3d0f --- /dev/null +++ b/qtparse.py @@ -0,0 +1,582 @@ +#!/usr/bin/env python +# +# QuickTime parser library for rePear, the iPod database management tool +# Copyright (C) 2006-2008 Martin J. Fiedler +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + +import sys, struct + +ID3v1Genres = { 0:"Blues", 1:"Classic Rock", 2:"Country", 3:"Dance", 4:"Disco", +5:"Funk", 6:"Grunge", 7:"Hip-Hop", 8:"Jazz", 9:"Metal", 10:"New Age", +11:"Oldies", 12:"Other", 13:"Pop", 14:"R&B", 15:"Rap", 16:"Reggae", 17:"Rock", +18:"Techno", 19:"Industrial", 20:"Alternative", 21:"Ska", 22:"Death Metal", +23:"Pranks", 24:"Soundtrack", 25:"Euro-Techno", 26:"Ambient", 27:"Trip-Hop", +28:"Vocal", 29:"Jazz&Funk", 30:"Fusion", 31:"Trance", 32:"Classical", +33:"Instrumental", 34:"Acid", 35:"House", 36:"Game", 37:"Sound Clip", +38:"Gospel", 39:"Noise", 40:"Alternative Rock", 41:"Bass", 42:"Soul", +43:"Punk", 44:"Space", 45:"Meditative", 46:"Instrumental Pop", +47:"Instrumental Rock", 48:"Ethnic", 49:"Gothic", 50:"Darkwave", +51:"Techno-Industrial", 52:"Electronic", 53:"Pop-Folk", 54:"Eurodance", +55:"Dream", 56:"Southern Rock", 57:"Comedy", 58:"Cult", 59:"Gangsta", +60:"Top 40", 61:"Christian Rap", 62:"Pop/Funk", 63:"Jungle", 64:"Native US", +65:"Cabaret", 66:"New Wave", 67:"Psychedelic", 68:"Rave", 69:"Showtunes", +70:"Trailer", 71:"Lo-Fi", 72:"Tribal", 73:"Acid Punk", 74:"Acid Jazz", +75:"Polka", 76:"Retro", 77:"Musical", 78:"Rock & Roll", 79:"Hard Rock", +80:"Folk", 81:"Folk-Rock", 82:"National Folk", 83:"Swing", 84:"Fast Fusion", +85:"Bebop", 86:"Latin", 87:"Revival", 88:"Celtic", 89:"Bluegrass", +90:"Avantgarde", 91:"Gothic Rock", 92:"Progressive Rock", +93:"Psychedelic Rock", 94:"Symphonic Rock", 95:"Slow Rock", 96:"Big Band", +97:"Chorus", 98:"Easy Listening", 99:"Acoustic", 100:"Humour", 101:"Speech", +102:"Chanson", 103:"Opera", 104:"Chamber Music", 105:"Sonata", 106:"Symphony", +107:"Booty Bass", 108:"Primus", 109:"Porn Groove", 110:"Satire", +111:"Slow Jam", 112:"Club", 113:"Tango", 114:"Samba", 115:"Folklore", +116:"Ballad", 117:"Power Ballad", 118:"Rhytmic Soul", 119:"Freestyle", +120:"Duet", 121:"Punk Rock", 122:"Drum Solo", 123:"Acapella", 124:"Euro-House", +125:"Dance Hall", 126:"Goa", 127:"Drum & Bass", 128:"Club-House", +129:"Hardcore", 130:"Terror", 131:"Indie", 132:"BritPop", 133:"Negerpunk", +134:"Polsk Punk", 135:"Beat", 136:"Christian Gangsta", 137:"Heavy Metal", +138:"Black Metal", 139:"Crossover", 140:"Contemporary Christian", +141:"Christian Rock", 142:"Merengue", 143:"Salsa", 144:"Thrash Metal", +145:"Anime", 146:"JPop", 147:"SynthPop" } + + + +QTAtomTypeMap = { + 'moov': 'container', + 'udta': 'container', + 'trak': 'container', + 'mdia': 'container', + 'minf': 'container', + 'stbl': 'container', + 'pinf': 'container', + 'schi': 'container', + 'ilst': 'container', +} + +QTTrackTypeMap = { + 'vide': 'video', + 'soun': 'audio', +} + +QTMetaDataMap = { + '$nam': ('text', 'title'), + '$alb': ('text', 'album'), + '$art': ('text', 'artist'), + '$ART': ('text', 'artist'), # FAAC writes this tag in captital letters + 'aART': ('text', 'album artist'), + '$cmt': ('text', 'comment'), + '$day': ('year', 'year'), + '$gen': ('text', 'genre'), + '$wrt': ('text', 'composer'), + '$too': ('text', 'encoder'), + 'cprt': ('text', 'copyright'), + 'trkn': ('track', None), + 'disk': ('disc', None), + 'covr': ('artwork', None), + 'cpil': ('flag', 'compilation'), + '$lyr': ('text', 'lyrics'), + 'desc': ('text', 'description'), + 'purl': ('text', 'podcast url'), + 'egid': ('text', 'episode id'), + 'catg': ('text', 'category'), + 'keyw': ('text', 'keyword'), + 'gnre': ('genre', None), +# gnre Genre 1 | 0 text | uint8 iTunes 4.0 +# tmpo BPM 21 uint8 iTunes 4.0 +# rtng Rating/Advisory 21 uint8 iTunes 4.0 +# stik ?? (stik) 21 uint8 ?? +# pcst Podcast 21 uint8 iTunes 4.9 +# tvnn TV Network Name 1 text iTunes 6.0 +# tvsh TV Show Name 1 text iTunes 6.0 +# tven TV Episode No 1 text iTunes 6.0 +# tvsn TV Season 21 uint8 iTunes 6.0 +# tves TV Episode 21 uint8 iTunes 6.0 +# pgap Gapless Play 21 uin8 iTunes 7.0 +} + +MP4DescriptorMap = { + 0x03: 'MP4ESDescr', + 0x04: 'MP4DecConfigDescr', + 0x05: 'MP4DecSpecificDescr', + 0x06: 'MP4SLConfigDescr', +} + +MP4ObjectTypeMap = { + 0x20: 'MPEG4Visual', + 0x40: 'MPEG4Audio', +} + +MP4ProfileMap = { + 1: "AAC Main", + 2: "LC-AAC", + 3: "AAC SSR", + 4: "AAC LTP", + 5: "HE-AAC", + 6: "Scalable", + 7: "TwinVQ", + 8: "CELP", + 9: "HVXC", + 12: "TTSI", + 13: "Main Synthetic Profile", + 14: "Wavetable synthesis", + 15: "General MIDI", + 16: "Algorithmic Synthesis and Audio FX", + 17: "LC-AAC with error recovery", + 19: "AAC LTP with error recovery", + 20: "AAC SSR with error recovery", + 21: "TwinVQ with error recovery", + 22: "BSAC with error recovery", + 23: "AAC LD with error recovery", + 24: "CELP with error recovery", + 25: "HXVC with error recovery", + 26: "HILN with error recovery", + 27: "Parametric with error recovery", +} + +H264ProfileMap = { + 66: "BP", + 77: "MP", + 88: "EP", + 100: "HP", + 110: "H10P", + 144: "H444P", +} + + +def chop(s): + if s: return (ord(s[0]), s[1:]) + return (0, "") + +def dictremove(d, rlist): + for r in rlist: + if r in d: + del d[r] + + +class QTParser: + def __init__(self, f, verbose=False): + self.f = f + self.verbose = verbose + self.info = {} + self.time_scale = 1 + self.tracks = {} + self.trackid = None + self.artwork = [] + self.errors = [] + + self.f.seek(0, 2) + self.info['size'] = self.f.tell() + self.parse_container(0, self.info['size']) + + def log_path(self, path, atom, size, start=None): + if not self.verbose: return + if start is None: + print("%s%s (%d bytes)" % (" " * len(path), atom, size)) + else: + print("%s%s (%d bytes @ %d)" % (" " * len(path), atom, size, start)) + + def err(self, path, message): + self.errors.append((self.repr_path(path), message)) + + def reject(self, path, size, minsize, need_track=True): + atom = path[-1] + if need_track: + if self.trackid is None: + return self.err(path, "%s outside of a track" % atom) + if not(self.trackid in self.tracks): + return True + if size < minsize: + return self.err(path, "atom too small") + return False + + def gettrack(self, prop, default=None): + return self.tracks[self.trackid].get(prop, default) + def settrack(self, prop, value): + self.tracks[self.trackid][prop] = value + + def repr_path(self, path): + if not path: return "" + return ".".join(path) + + def parse_container(self, start=0, size=0, path=[]): + end = start + size + while (start + 8) < end: + self.f.seek(start) + head = self.f.read(8) + start += 8 + size = struct.unpack(">L", head[:4])[0] - 8 + if size < 0: + return self.err(path, "invalid sub-atom size") + atom = head[4:].strip("\0 ").replace('\xa9', '$') + if not atom: + break + self.log_path(path, atom, size, start) + if atom in QTMetaDataMap: + alias = 'container' + else: + alias = QTAtomTypeMap.get(atom, atom) + try: + assert alias + parser = getattr(self, "parse_" + alias) + except (AttributeError, AssertionError): + parser = None + if parser: + parser(start, min(size, end - start), path + [atom]) + start += size + if start < end: + return self.err(path, "%d orphaned bytes" % (end - start)) + if start > end: + return self.err(path, "%d missing bytes" % (start - end)) + + def parse_trak(self, start, size, path): + self.track = None + self.parse_container(start, size, path) + self.track = None + + def parse_mvhd(self, start, size, path): + if self.reject(path, size, 20, False): return + data = self.f.read(20) + self.time_scale, length = struct.unpack(">LL", data[12:]) + self.info['length'] = float(length) / self.time_scale + + def parse_tkhd(self, start, size, path): + if self.reject(path, size, 24, False): return + data = self.f.read(min(size, 84)) + self.trackid, dummy, length = struct.unpack(">LLL", data[12:24]) + if not self.trackid in self.tracks: + self.tracks[self.trackid] = {} + self.settrack('length', float(length) / self.time_scale) + if len(data) >= 84: + w, h = struct.unpack(">LL", data[76:84]) + self.settrack('width', w >> 16) + self.settrack('height', h >> 16) + + def parse_mdhd(self, start, size, path): + if self.reject(path, size, 20): return + data = self.f.read(20) + time_scale, length = struct.unpack(">LL", data[12:]) + self.settrack('length', float(length) / time_scale) + + def parse_hdlr(self, start, size, path): + if 'udta' in path: + return + if self.reject(path, size, 12): return + data = self.f.read(12) + try: + self.tracks[self.trackid]['type'] = QTTrackTypeMap[data[8:]] + except KeyError: + del self.tracks[self.trackid] + + def parse_stsd(self, start, size, path): + if self.reject(path, size, 8): return + data = self.f.read(8) + count = struct.unpack(">L", data[4:8])[0] + end = start + size + start += 8 + media_type = self.gettrack('type') + for i in range(count): + if start > (end - 16): + return self.err(path, "description #%d too small" % (i+1)) + self.f.seek(start) + data = self.f.read(16) + start += 16 + size = struct.unpack(">L", data[:4])[0] - 16 + format = data[4:8].strip("\0 ") + refidx = struct.unpack(">H", data[14:])[0] + self.log_path(path, format, size, start) + try: + parser = getattr(self, "parse_stsd_" + media_type) + except KeyError: + if not i: self.err(path, "descriptions found, but no handler defined") + parser = None + except AttributeError: + parser = None + if parser: + self.settrack('format', format) + parser(start, min(size, end - start), path + [format]) + start += size + + def parse_stsd_audio(self, start, size, path): + if self.reject(path, size, 20): return + data = self.f.read(20) + version, rev, ven, chan, res, compid, packsize, rate_hi, rate_lo = \ + struct.unpack(">HHLHHHHHH", data) + if version == 0: + hsize = 20 + elif version == 1: + hsize = 24 + else: + return self.err(path, "unknown audio stream description version") + if size < hsize: + return self.err(path, "stream description too small") + start += hsize + size -= hsize + self.settrack('channels', chan) + self.settrack('bits per sample', res) + self.settrack('sample rate', rate_hi) + if self.gettrack('length'): + self.settrack('sample count', int(rate_hi * self.gettrack('length') + 0.5)) + self.parse_container(start, size, path) + + def parse_stsd_video(self, start, size, path): + if self.reject(path, size, 70): return + version, rev, ven, tq, sq, w, h, hres, vres, zero, frames = \ + struct.unpack(">HHLLLHHLLLH", self.f.read(34)) + if (w != self.gettrack('width')) or (h != self.gettrack('height')): + self.err(path, "video size doesn't match track header value") + data = self.f.read(32) + clen = ord(data[0]) + if clen > 31: + self.err(path, "invalid compressor name length") + elif clen: + self.settrack('compressor', data[1:clen+1]) + self.parse_container(start + 70, size - 70, path) + + def parse_avcC(self, start, size, path): + if self.reject(path, size, 4): return + data = self.f.read(4) + profile = H264ProfileMap.get(ord(data[1]), None) + level = ord(data[3]) + if level % 10: + level = "%d.%d" % (level / 10, level % 10) + else: + level = str(level / 10) + if (level == "1.1") and (ord(data[2]) & 0x10): + level = "1b" + format = "H.264" + if profile: format += " " + profile + self.settrack('video format', format + "@L" + level) + + def parse_esds(self, start, size, path): + try: + if not(path[-2] in ('mp4a', 'mp4v')): + return # unknown format, ignore it + except IndexError: + return self.err(path, "esds atom found at root level") + if self.reject(path, size, 4, False): return + self.f.seek(start + 4) + self.parse_mp4desc(path, self.f.read(size - 4)) + + def parse_mp4desc(self, path, data): + while data: + tag, data = chop(data) + size = 0 + while True: + if not data: + return self.err(path, "descriptor ends while decoding length") + byte, data = chop(data) + size = (size << 7) | (byte & 0x7F) + if not(byte & 0x80): break + if size > len(data): + self.err(path, "%d missing bytes in descriptor" % (size - len(data))) + size = len(data) + tag = MP4DescriptorMap.get(tag, "0x%02X" % tag) + self.log_path(path, tag, size) + try: + parser = getattr(self, "parse_" + tag) + except AttributeError: + parser = None + if parser: + parser(path + [tag], data[:size]) + data = data[size:] + + def parse_MP4ESDescr(self, path, data): + if self.reject(path, len(data), 3, False): return + esid, flags = struct.unpack(">BH", data[:3]) + data = data[3:] + if flags & 0x80: # stream_dependence + if self.reject(path, len(data), 2, False): return + data = data[2:] + if flags & 0x40: # URL + if self.reject(path, len(data), 1, False): return + size, data = chop(data) + if self.reject(path, len(data), size, False): return + data = data[size:] + if flags & 0x20: # ocr_stream + if self.reject(path, len(data), 2, False): return + data = data[2:] + self.parse_mp4desc(path, data) + + def parse_MP4DecConfigDescr(self, path, data): + if self.reject(path, len(data), 13, False): return + otid, flags, buf_hi, buf_lo, rate_max, rate_avg = \ + struct.unpack(">BBBHLL", data[:13]) + self.settrack('bitrate', int(rate_avg / 1000)) + objtype = MP4ObjectTypeMap.get(otid, None) + if not objtype: return # some unknown format + self.parse_mp4desc(path + [objtype], data[13:]) + + def parse_MP4DecSpecificDescr(self, path, data): + try: + parser = getattr(self, "parse_MP4DecSpecificDescr_" + path[-2]) + except AttributeError: + return # unknown format + except IndexError: + raise + parser(path, data) + + def parse_MP4DecSpecificDescr_MPEG4Audio(self, path, data): + if self.reject(path, len(data), 2, False): return + a, data = chop(data) + b, data = chop(data) + profile = (a >> 3) & 0x1F + freq = ((a << 1) | (b >> 7)) & 0x0F + try: + self.settrack('filetype', "MPEG-4 " + MP4ProfileMap[profile]) + except KeyError: + pass + if freq == 15: + if self.reject(path, len(data), 3, False): return + freq = b & 0x7F + b, data = chop(data) + freq = (freq << 7) | b + b, data = chop(data) + freq = (freq << 7) | b + b, data = chop(data) + freq = (freq << 1) | (b >> 7) + else: + try: + freq = (96000, 88200, 64000, 48000, 44100, 32000, 24000, 22050, 16000, 12000, 11025, 8000, 7350)[freq] + except IndexError: + self.err(path, "invalid sampling rate code %d" % freq) + freq = 0 + if freq: + ref = self.gettrack('sample rate') + if not ref: + self.settrack('sample rate', freq) + elif freq != ref: + self.err(path, "sample rate in AAC descriptor (%d) doesn't match sample rate in stream description (%d)" % (freq, ref)) + if data: + return self.err(path, "descriptor is longer than expected") + + def parse_MP4DecSpecificDescr_MPEG4Visual(self, path, data): + self.settrack('video format', 'MPEG-4 ASP') + + def parse_meta(self, start, size, path): + self.parse_container(start + 4, size - 4, path) + + def parse_data(self, start, size, path): + if self.reject(path, size, 8, False): return + try: + parser, key = QTMetaDataMap[path[-2]] + except IndexError: + return self.err(path, "data atom found at root level") + except KeyError: + return self.err(path, "no parser defined for this atom") + format = struct.unpack(">L", self.f.read(8)[:4])[0] & 0x00FFFFFF + try: + parser = getattr(self, "format_" + parser) + except AttributeError: + return self.err(path, "format parser `%s' doesn't exist" % parser) + res = parser(path, self.f.read(size - 8)) + if key: + if res is None: + return self.err(path, "decoding failed, no value assigned") + self.info[key] = res + + def format_text(self, path, data): + data = data.strip("\0") + if data.startswith("\xfeff"): + return str(data, 'utf_16') + else: + return str(data, 'utf_8') + + def format_year(self, path, data): + data = data.strip("\0").split('-', 1)[0] + try: + return int(data) + except ValueError: + return self.err(path, "invalid date format") + + def format_byte(self, path, data): + if not data: + return self.err(path, "zero-length data block") + return ord(data[0]) + + def format_genre(self, path, data): + if not data: + return self.err(path, "zero-length data block") + genre = ID3v1Genres.get(ord(data[-1]) - 1, None) + if genre: self.info["genre"] = genre + + def format_track(self, path, data, item='track'): + if self.reject(path, len(data), 6, False): return + current, total = struct.unpack(">HH", data[2:6]) + if current: self.info["%s number" % item] = current + if total: self.info["total %ss" % item] = total + + def format_disc(self, path, data): + return self.format_track(path, data, 'disc') + + def format_artwork(self, path, data): + self.artwork.append(data) + + def format_flag(self, path, data): + return len(data.strip("\0")) != 0 + + def get_repear_info(self): + info = {} + have_video = False + have_audio = False + for track in self.tracks.values(): + ttype = track.get('type', '?') + if not(have_audio) and (ttype == 'audio'): + ainfo = track.copy() + dictremove(ainfo, ('type', 'width', 'height')) + info.update(ainfo) + have_audio = True + if not(have_video) and (ttype == 'video'): + vinfo = track.copy() + dictremove(vinfo, ('type',)) + if have_audio: dictremove(vinfo, ('format', )) + info.update(vinfo) + have_video = True + info.update(self.info) + if ('album artist' in info) and not('artist' in info): + info['artist'] = info['album artist'] + if have_video: + info['filetype'] = "MPEG-4 Video file" + elif not('filetype' in info): + info['filetype'] = "MPEG-4 Audio File" + return info + +################################################################################ + +def dump_dict(d): + keys = d.keys() + keys.sort() + for key in keys: + print(" %s = %s" % (key, repr(d[key]))) + +if __name__ == "__main__": + qt = QTParser(open(sys.argv[1], "rb"), True) + print() + + print("Raw file information:") + dump_dict(qt.info) + for track in qt.tracks: + print("Raw track information (id %s):" % track) + dump_dict(qt.tracks[track]) + print() + + print("rePear-compliant information:") + dump_dict(qt.get_repear_info()) + print() + + if qt.errors: + print("Errors:") + for e in qt.errors: print(" %s: %s" % e) + print() diff --git a/repear.log b/repear.log new file mode 100644 index 0000000..787bea3 --- /dev/null +++ b/repear.log @@ -0,0 +1,10 @@ +Welcome to rePear, version 0.4.1 +-------------------------------- + +iPod root directory is `F:/' + +Moving tracks back to their original locations ... +s2/Sade - Your Love Is King.mp3 [OK] +Operation complete: 1 tracks total, 1 moved back, 0 failed. + +You can now manage the music files on your iPod. diff --git a/repear.py b/repear.py new file mode 100644 index 0000000..956754f --- /dev/null +++ b/repear.py @@ -0,0 +1,1686 @@ +#!/usr/bin/env python +# +# rePear, the iPod database management tool +# Copyright (C) 2006-2008 Martin J. Fiedler +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + +__title__ = "rePear" +__version__ = "0.4.1" +banner = "Welcome to %s, version %s" % (__title__, __version__) + +DISSECT_BASE_DIR = "Dissected Tracks/" +DEFAULT_LAME_OPTS = "--quiet -h -V 5" +MASTER_PLAYLIST_FILE = "repear_playlists.ini" +SUPPORTED_FILE_FORMATS = (".mp3", ".ogg", ".m4a", ".m4b", ".mp4") +MUSIC_DIR = "iPod_Control/Music/" +CONTROL_DIR = "iPod_Control/iTunes/" +ARTWORK_DIR = "iPod_Control/Artwork/" +DB_FILE = CONTROL_DIR + "iTunesDB" +CACHE_FILE = CONTROL_DIR + "repear.cache" +MODEL_FILE = CONTROL_DIR + "repear.model" +FWID_FILE = CONTROL_DIR + "fwid" +ARTWORK_CACHE_FILE = ARTWORK_DIR + "repear.artwork_cache" +ARTWORK_DB_FILE = ARTWORK_DIR + "ArtworkDB" + +import sys, optparse, os, fnmatch, stat, string, types, pickle, random +import re, traceback, tempfile +from pathlib import Path + +import iTunesDB, mp3info, hash58 +Options = {} + +################################################################################ +## Some internal management functions ## +################################################################################ + +broken_log = False +homedir = "" +logfile = None + +def open_log(): + global logfile + Options['log'] = os.path.abspath(Options['log']) + try: logfile = open(Options['log'], "w") + except IOError: logfile = None + +def log(line, flush=True): + global logfile, broken_log + sys.stdout.write(line) + if flush: sys.stdout.flush() + if logfile: + try: + logfile.write(line) + if flush: logfile.flush() + except IOError: broken_log = True +iTunesDB.log = log + +def quit(code=1): + global logfile, broken_log + if logfile: + try: logfile.close() + except IOError: broken_log = True + logfile = None + log("\nLog written to `%s'\n" % Options['log']) + if broken_log: log("WARNING: there were errors while writing the log file\n") + sys.exit(code) + +def fatal(line): + log("FATAL: %s\n" % line) + quit() + +def confirm(prompt): + sys.stdout.write("%sDo you really want to continue? (y/N) " % prompt) + sys.stdout.flush() + try: answer = input() + except (IOError, EOFError, KeyboardInterrupt): answer = "" + if answer.strip().lower() in ("y", "yes"): return + log("Action aborted by user.\n") + quit() + +def goto_root_dir(): + global homedir + homedir = os.path.abspath(os.path.split(sys.argv[0])[0]).replace("\\", "/") + if homedir[-1] != '/': homedir += '/' + if Options['root']: + rootdir = Options['root'].replace("\\", "/") + if rootdir[-1] != '/': rootdir += '/' + else: + # no root directory specified -- try the current directory + rootdir = os.getcwd().replace("\\", "/") + if rootdir[-1] != '/': rootdir += '/' + if not os.path.isfile(rootdir + "iPod_Control/iTunes/iTunesDB"): + # not found? then try the executable's directory + rootdir = homedir + # special case on Windows: if the current directory doesn't contain + # a valid iPod directory structure, reduce the pathname to the first + # three characters, as in 'X:/', which is usually the root directory + if (os.name == 'nt') and not(os.path.isfile(rootdir + DB_FILE)): + rootdir = rootdir[:3] + + if os.path.isfile(rootdir + DB_FILE): log("iPod root directory is `%s'\n" % rootdir) + else: fatal("root directory `%s' contains no iPod database" % rootdir) + + try: os.chdir(rootdir) + except OSError as e: fatal("can't change to the iPod root directory: %s" % e.strerror) + +def load_cache(return_on_error=(None,None)): + try: f = open(CACHE_FILE, "rb") + except IOError: return return_on_error + try: + content = pickle.load(f) + f.close() + except (IOError, EOFError, pickle.PickleError): return return_on_error + return content + +def save_cache(content=None): + try: + with open(CACHE_FILE, "wb") as f: pickle.dump(content, f) + except (IOError, EOFError, pickle.PickleError): log("ERROR: can't save the rePear cache\n") + +def execute(program, args): + global homedir + if os.name == "nt": + spawn = os.spawnv + path = homedir + program + ".exe" + args = ["\"%s\"" % arg for arg in args] + else: + spawn = os.spawnvp + path = program + try: return spawn(os.P_WAIT, path, [program] + args) + except OSError as e: log("ERROR: can't execute %s: %s\n" % (program, e.strerror)) + except KeyboardInterrupt: return -2 + +def printable(x, kill_chars=""): + x = str(x) + for c in kill_chars: x = x.replace(c, "_") + return x + +def move_file(src: str | Path, dest: str | Path): + src = Path(src) + dest = Path(dest) + + # check if source file exists + if not src.is_file(): + log(f"[FAILED]\nERROR: source file `{printable(src)}' doesn't exist\n", True) + return 'missing' + + # don't clobber files (wouldn't work on Windows anyway) + if dest.is_file(): + log(f"[FAILED]\nERROR: destination file `{printable(dest)}' already exists\n", True) + return 'exists' + + # create parent directories if necessary + dest_dir = dest.parent + if not dest_dir.is_dir(): + try: dest_dir.mkdir() + except OSError as e: + log(f"[FAILED]\nERROR: can't create destination directory `{printable(dest_dir)}': {e.strerror}\n", True) + return 'mkdir' + + # finally rename it + try: + src.rename(dest) + except OSError as e: + log(f" [FAILED]\nERROR: can't move `{printable(src)}' to `{printable(dest)}': {e.strerror}\n", True) + return 'move' + log("[OK]\n", True) + return None + +def backup(file: str | Path): + file = Path(file) + dest = file.parent / f"{file.name}.repear_backup" + if dest.exists(): return + try: + file.rename(dest) + return True + except OSError as e: + log(f"WARNING: Cannot backup `{file.name}': {e.strerror}\n") + return False + +def delete(file: str | Path, may_fail=False): + file = Path(file) + if not file.exists(): return + try: + file.unlink() + return True + except OSError as e: + if not may_fail: log(f"ERROR: Cannot delete `{file.name}': {e.strerror}\n") + return False + +class Logger: + @staticmethod + def write(s): log(s) + +# path and file name sorting routines +re_digit = re.compile(r'(\d+)') +def tryint(s): + try: return int(s) + except ValueError: return s.lower() +def fnrep(fn): return tuple(map(tryint, re_digit.split(fn))) +def fncmp(a, b): + a_val = fnrep(a) + b_val = fnrep(b) + return (a_val > b_val) - (a_val < b_val) +def pathcmp(a, b): + a = a.split(u'/') + b = b.split(u'/') + # compare base directories + for i in range(min(len(a), len(b)) - 1): + r = fncmp(a[i], b[i]) + if r: return r + # subdirectories first + r = len(b) - len(a) + if r: return r + # finally, compare leaf file name + return fncmp(a[-1], b[-1]) +def trackcmp(a, b): return pathcmp(a.get('original path', None) or a.get('path', '???'), b.get('original path', None) or b.get('path', '???')) + +class Allocator: + def __init__(self, root: Path | str, files_per_dir=100, max_dirs=100): + self.root = Path(root) + self.files_per_dir = files_per_dir + self.max_dirs = max_dirs + self.names = {} + self.files = {} + digits = [] + try: dirs = self.root.iterdir() + except OSError: + self.root.mkdir() + dirs = [] + for elem in dirs: + try: index = self.getindex(elem) + except ValueError: continue + self.names[index] = elem.name + self.files[index] = self.scandir(root / elem) + digits.append(len(elem.name) - 1) + if digits: + digits.sort() + self.fmt = "F%%0%dd" % (digits[len(digits) // 2]) + else: self.fmt = "F%02d" + if not self.files: self.mkdir(0) + self.current_dir = min(self.files.keys()) + + def getindex(self, name): + if not name: raise ValueError + name = Path(name) + if name.name[0].upper() != 'F': raise ValueError + return int(name.name[1:], 10) + + def scandir(self, root: str | Path): + try: dir_contents = Path(root).iterdir() + except OSError: return {} + dir_contents = [os.path.splitext(x.name)[0].upper() for x in dir_contents if x.name[0] != '.'] + return dict(zip(dir_contents, [None] * len(dir_contents))) + + def __len__(self): return sum(map(len, self.files.values())) + def __repr__(self): return "" % (len(self), len(self.files)) + + def allocate_ex(self, index): + while True: + name = "".join([random.choice(string.ascii_uppercase) for x in range(4)]) + if not(name in self.files[index]): break + self.files[index][name] = None + return self.names[index] + '/' + name + + def mkdir(self, index): + if index in self.files: return + name = self.fmt % index + try: Path(self.root, name).mkdir() + except OSError: pass + self.names[index] = name + self.files[index] = {} + + def allocate(self): + count, index = min([(len(d[1]), d[0]) for d in self.files.items()]) + # need to allocate a new directory + if (count >= self.files_per_dir) and (len(self.files) < self.max_dirs): + available = [i for i in range(self.max_dirs) if not i in self.files] + index = available[0] + self.mkdir(index) + # generate a file name + while True: + name = "".join([random.choice(string.ascii_uppercase) for x in range(4)]) + if not(name in self.files[index]): break + self.files[index][name] = None + return str(self.root) + '/' + self.names[index] + '/' + name + + def add(self, fullname): + try: + dirname, filename = fullname.split('/')[-2:] + index = self.getindex(dirname) + except ValueError: return + filename = os.path.splitext(filename)[0] + if not index in self.files: + self.names[index] = dirname + self.files[index] = {} + self.files[index][filename] = None + +class BalancedShuffle: + def __init__(self): + self.root: dict[None | bytes, list] = { None: [] } + + def add(self, path, data): + if isinstance(path, str): + path = path.encode('ascii', 'replace') + path = path.replace(b"\\", b"/").lower().split(b"/") + if path and not path[0]: + path.pop(0) + if not path: + return # broken path + root = self.root + while True: + if len(path) == 1: + # tail reached + root[None].append(data) + break + component = path.pop(0) + if component not in root: + root[component] = { None: [] } + root = root[component] + + def shuffle(self, root=None): + if not root: + root = self.root + + # shuffle the files of the root node + random.shuffle(root[None]) + + # build a list of directories to shuffle + subdirs = filter(None, [root[None]] + \ + [self.shuffle(root[key]) for key in root if key]) + + # check for "tail" cases + if not subdirs: + return [] + if len(subdirs) == 1: + return subdirs[0] + + # pad subdirectory list to a common length + dircount = len(subdirs) + maxlen = max(map(len, subdirs)) + subdirs = [self.fill(sd, maxlen) for sd in subdirs] + + # collect all items + res = [] + last = -1 + for i in range(maxlen): + # determine the directory order for this "column" + order = range(dircount) + random.shuffle(order) + if (len(order) > 1) and (order[0] == last): + order.append(order.pop(0)) + while len(order) > 1: # = if len(order) > 1: while True: + random.shuffle(order) + if last != order[0]: break + last = order[-1] + + # produce a result + res.extend(filter(lambda x: x is not None, \ + [subdirs[j][i] for j in order])) + return res + + def fill(self, data, total): + ones = len(data) + invert = (ones > (total / 2)) + if invert: + ones = total - ones + bitmap = [0] * total + remain = total + for fraction in range(ones, 0, -1): + bitmap[total - remain] = 1 + skip = float(remain) / fraction + skip = random.randrange(int(0.9 * skip), int(1.1 * skip) + 2) + remain -= min(max(1, skip), remain - fraction + 1) + if invert: + bitmap = [1-x for x in bitmap] + offset = random.randrange(0, total) + bitmap = bitmap[offset:] + bitmap[:offset] + def decide(x): + if x: return data.pop(0) + return None + return map(decide, bitmap) + + +def ImportPlayCounts(cache, index): + log("Updating play counts and ratings ... ", True) + + # open Play Counts file + try: + pc = iTunesDB.PlayCountsReader() + except IOError: + log("\n0 track(s) updated.\n") + return False + except iTunesDB.InvalidFormat: + log("\n-- Error in Play Counts file, import failed.\n") + return False + + # parse old iTunesDB + try: + db = iTunesDB.DatabaseReader() + files = [printable(item.get('path', u'??')[1:].replace(u':', u'/')).lower() for item in db] + db.f.close() + del db + except (IOError, iTunesDB.InvalidFormat): + log("\n-- Error in iTunesDB, import failed.\n") + return False + + # plausability check + if len(files) != pc.entry_count: + log("\n-- Mismatch between iTunesDB and Play Counts file, import failed.\n") + return False + + # walk through Play Counts file + update_count = 0 + try: + for item in pc: + path = files[item.index] + try: + track = cache[index[path]] + except (KeyError, IndexError): + continue + updated = False + if item.play_count: + track['play count'] = track.get('play count', 0) + item.play_count + updated = True + if item.last_played: + track['last played time'] = item.last_played + updated = True + if item.skip_count: + track['skip count'] = track.get('skip count', 0) + item.skip_count + updated = True + if item.last_skipped: + track['last skipped time'] = item.last_skipped + updated = True + if item.bookmark: + track['bookmark time'] = item.bookmark * 0.001 + updated = True + if item.rating: + track['rating'] = item.rating + updated = True + if updated: + update_count += 1 + pc.f.close() + del pc + except (IOError, iTunesDB.InvalidFormat): + log("\n-- Error in Play Counts file, import failed.\n") + return False + log("%d track(s) updated.\n" % update_count) + return update_count + + +################################################################################ +## DISSECT action ## +################################################################################ + +def Dissect(): + state, cache = load_cache((None, None)) + + if (state is not None) and not(Options['force']): + if state=="frozen": confirm(""" +WARNING: This action will put all the music files on your iPod into a completely +new directory structure. All previous file and directory names will be lost. +This also means that any iTunesDB backups you have will NOT work any longer! +""") + if state=="unfrozen": confirm(""" +WARNING: The database is currently unfrozen, so the following operations will +almost completely fail. +""") + + cache = [] + try: + db = iTunesDB.DatabaseReader() + + for info in db: + if not info.get('path', None): + log("ERROR: track lacks path attribute\n") + continue + src = printable(info['path'])[1:].replace(":", "/") + if not os.path.isfile(src): + log("ERROR: file `%s' is found in database, but doesn't exist\n" % src) + continue + if not info.get('title', None): + info.update(iTunesDB.GuessTitleAndArtist(info['path'])) + ext = os.path.splitext(src)[1] + base = DISSECT_BASE_DIR + if info.get('artist', None): + base += printable(info['artist'], "<>/\\:|?*\"") + '/' + if info.get('album', None): + base += printable(info['album'], "<>/\\:|?*\"") + '/' + if info.get('track number', None): + base += "%02d - " % info['track number'] + base += printable(info['title'], "<>/\\:|?*\"") + + # move the file, but avoid filename collisions + serial = 1 + dest = base + ext + while os.path.exists(dest): + serial += 1 + dest = base + " (%d)"%serial + ext + log("%s => %s " % (src, dest), True) + if move_file(src, dest): + continue # move failed + + # create a placeholder cache entry + cache.append({ + 'path': src, + 'original path': str(dest, sys.getfilesystemencoding(), 'replace') + }) + except IOError: + fatal("can't read iTunes database file") + except iTunesDB.InvalidFormat: + raise + fatal("invalid iTunes database format") + + # clear the cache + save_cache(("unfrozen", cache)) + + + +################################################################################ +## FREEZE utilities ## +################################################################################ + +g_freeze_error_count = 0 + +def check_file(base, fn): + if fn.startswith('.'): + return None # skip dot-files and -directories + key, ext = [component.lower() for component in os.path.splitext(fn)] + fullname = base + fn + try: + s = os.stat(fullname) + except OSError: + log("ERROR: directory entry `%s' is inaccessible\n" % fn) + return None + isfile = int(not(stat.S_ISDIR(s[stat.ST_MODE]))) + if isfile and not(stat.S_ISREG(s[stat.ST_MODE])): + return None # no directory and no normal file -> skip this crap + if not(isfile) and (fullname=="iPod_Control" or fullname=="iPod_Control/Music"): + isfile = -1 # trick the sort algorithm to move iPC/Music to front + + return (isfile, fnrep(fn), fullname, s, ext, key) + + +def make_cache_index(cache): + index = {} + for i in range(len(cache)): + for path in [cache[i][f] for f in ('path', 'original path') if f in cache[i]]: + key = printable(path).lower() + if key in index: + log("ERROR: `%s' is cached multiple times\n" % printable(path)) + else: + index[key] = i + return index + + +def find_in_cache(cache, index, path, s): + i = index.get(printable(path).lower(), None) + if i is None: + return (False, None) # not found + info = cache[i] + + # check size and modification time + if info.get('size', None) != s[stat.ST_SIZE]: + return (False, info) # mismatch + if not iTunesDB.compare_mtime(info.get('mtime', 0), s[stat.ST_MTIME]): + return (False, info) # mismatch + + # all checks passed => correct file + return (True, info) + + +def move_music(src, dest, info): + global g_freeze_error_count + format = info.get('format', "mp3-cbr") + if format == "ogg": + src = printable(src) + dest = os.path.splitext(printable(dest))[0] + ".mp3" + with tempfile.NamedTemporaryFile(delete=True, suffix=".wav") as f: + # generate new source filename (replace .ogg by .mp3) + newsrc = info.get('original path', src) + if not isinstance(newsrc, str): + newsrc = str(newsrc, sys.getfilesystemencoding(), 'replace') + newsrc = u'.'.join(newsrc.split(u'.')[:-1]) + u'.mp3' + + # decode the Ogg file + res = execute("oggdec", ["-Q", "-o", f.name, src]) + if res != 0: + g_freeze_error_count += 1 + log("[FAILED]\nERROR: cannot execute OggDec ... result '%s'\n" % res) + delete(f.name, may_fail=True) + return None + else: + log("[decoded] ", True) + + # build LAME option list + lameopts = Options['lameopts'].split(' ') + for key, optn in (('title','tt'), ('artist','ta'), ('album','tl'), ('year','ty'), ('comment','tc'), ('track number','tn')): + if key in info: + lameopts.extend(["--"+optn, printable(info[key])]) + if 'genre' in info: + ref_genre = printable(info['genre']).lower().replace(" ","") + for number, genre in mp3info.ID3v1Genres.items(): + if genre.lower().replace(" ","") == ref_genre: + lameopts.extend(["--tg", str(number)]) + break + + # encode to MP3 + res = execute("lame", lameopts + [f.name, dest]) + delete(f.name) + if res != 0: + g_freeze_error_count += 1 + log("[FAILED]\nERROR: cannot execute LAME ... result code %d\n" % res) + return None + else: + log("[encoded] ", True) + + # check the resulting file + info = mp3info.GetAudioFileInfo(dest) + if not info: + g_freeze_error_count += 1 + log("[FAILED]\nERROR: generated MP3 file is invalid\n") + delete(dest) + return None + delete(src) + info['original path'] = newsrc + info['changed'] = 2 + log("[OK]\n", True) + return info + + else: # no Ogg file -> move directly + if move_file(src, dest): + g_freeze_error_count += 1 + return None # failed + else: + return info + + +def freeze_dir(cache, index, allocator: Allocator, playlists=[], base="", artwork=None): + global g_freeze_error_count + try: + flist = list(filter(None, [check_file(base, fn) for fn in os.listdir(base or ".")])) + except KeyboardInterrupt: + raise + except: + g_freeze_error_count += 1 + log(base + "/\n" + " runtime error, traceback follows ".center(79, '-') + "\n") + traceback.print_exc(file=Logger) + log(79*'-' + "\n") + return [] + + # generate directory list + directories = [i for i in flist if i[0] < 1] + directories.sort() + + # add playlist files + playlists.extend([x[2] for x in flist if (x[0] > 0) and (x[4] == ".m3u")]) + + # generate music file list + music = [i for i in flist if (i[0] > 0) and (i[4] in SUPPORTED_FILE_FORMATS)] + music.sort() + + # if there are no subdirs and no music files here, prune this directory + if not(directories) and not(music): + return [] + + # generate name -> artwork file associations + image_assoc = dict([(x[5], x[2]) for x in flist if (x[0] > 0) and (x[4] in (".jpg", ".png"))]) + + # find artwork files that are not associated to a file or directory + unassoc_images = image_assoc.copy() + for d0,d1,d2,d3,d4,key in directories: + if key in unassoc_images: + del unassoc_images[key] + for d0,d1,d2,d3,d4,key in music: + if key in unassoc_images: + del unassoc_images[key] + unassoc_images = list(unassoc_images.values()) + unassoc_images.sort() + + # use one of the unassociated artwork files as this directory's artwork, + # unless the inherited artwork file name is already a perfect match (i.e. + # the directory name and the artwork name are identical) + if unassoc_images: + if not(artwork) or not(artwork.lower().startswith(base[:-1].lower())): + artwork = find_good_artwork(unassoc_images, base) + + # now that the artwork problem is solved, we start processing: + # recurse into subdirectories first + res = [] + for isfile, dummy, fullname, s, ext, key in directories: + res.extend(freeze_dir(cache, index, allocator, playlists, fullname + '/', artwork)) + + # now process the local files + locals = [] + unique_artist = None + unique_album = None + for isfile, dummy, fullname, s, ext, key in music: + try: + # we don't need to move this file if it's already in the Music directory + already_there = fullname.startswith(MUSIC_DIR) + + # is this track cached? + log(fullname + ' ', True) + valid, info = find_in_cache(cache, index, fullname, s) + if valid: + info['changed'] = 0 + log("[cached] ", True) + else: + if info: + # cache entry present, but invalid => save iPod_Control location + path = info['path'] + changed = 1 + else: + path = fullname + changed = 2 + info = mp3info.GetAudioFileInfo(fullname) + iTunesDB.FillMissingTitleAndArtist(info) + info['changed'] = changed + if not already_there: + if isinstance(info['path'], str): + info['original path'] = info['path'] + else: + info['original path'] = str(info['path'], sys.getfilesystemencoding(), 'replace') + info['path'] = path + + # move the track to where it belongs + if not already_there: + path = info.get('path', None) + if not(path) or os.path.exists(path) or not(os.path.isdir(os.path.split(path)[0])): + # if anything is wrong with the path, generate a new one + path = allocator.allocate() + ext + else: + allocator.add(path) + info['path'] = path + info = move_music(fullname, path, info) + if not info: continue # something failed + else: + allocator.add(fullname) + log("[OK]\n", True) + + # associate artwork to the track + info['artwork'] = image_assoc.get(key, artwork) + + # check for unique artist and album + check = info.get('artist', None) + if not locals: + unique_artist = check + elif check != unique_artist: + unique_artist = False + check = info.get('album', None) + if not locals: + unique_album = check + elif check != unique_album: + unique_album = False + + # finally, append the track to the track list + locals.append(info) + + except KeyboardInterrupt: + log("\nInterrupted by user.\nContinue with next file or abort? [c/A] ") + try: + answer = input() + except (IOError, EOFError, KeyboardInterrupt): + answer = "" + if not answer.lower().startswith("c"): + raise + + except: + g_freeze_error_count += 1 + log("\n" + " runtime error, traceback follows ".center(79, '-') + "\n") + traceback.print_exc(file=Logger) + log(79*'-' + "\n") + + # if all files in this directory share the same album title, but differ + # in the artist name, we assume it's a compilation + if unique_album and not(unique_artist): + for info in locals: + info['compilation'] = 1 + + # combine the lists and return them + res.extend(locals) + return res + + +################################################################################ +## playlist sorting ## +################################################################################ + +def cmp_lst(a, b, order, empty_pos): + a = max(a.get('last played time', 0), a.get('last skipped time', 0)) + b = max(b.get('last played time', 0), b.get('last skipped time', 0)) + if not a: + if not b: return 0 + return empty_pos + else: + if not b: return -empty_pos + return order * cmp(a, b) + +def cmp_path(a, b, order, empty_pos): + return order * trackcmp(a, b) + +class cmp_key: + def __init__(self, key): + self.key = key + def __repr__(self): + return "%s(%s)" % (self.__class__.__name__, repr(self.key)) + def __call__(self, a, b, order, empty_pos): + if self.key in a: + if self.key in b: + a = a[self.key] + if type(a) in (types.StringType, types.UnicodeType): a = a.lower() + b = b[self.key] + if type(b) in (types.StringType, types.UnicodeType): b = b.lower() + return order * cmp(a, b) + else: + return -empty_pos + else: + if self.key in b: + return empty_pos + else: + return 0 + +sort_criteria = { + 'playcount': lambda a,b,o,e: o*cmp(a.get('play count', 0), b.get('play count', 0)), + 'skipcount': lambda a,b,o,e: o*cmp(a.get('skip count', 0), b.get('skip count', 0)), + 'startcount': lambda a,b,o,e: o*cmp(a.get('play count', 0) + a.get('skip count', 0), b.get('play count', 0) + b.get('skip count', 0)), + 'artworkcount': lambda a,b,o,e: o*cmp(a.get('artwork count', 0), b.get('artwork count', 0)), + 'laststartedtime': cmp_lst, + 'laststarttime': cmp_lst, + 'lastplaytime': 'last played time', + 'lastskiptime': 'last skipped time', + 'movie': 'movie flag', + 'filesize': 'size', + 'path': cmp_path, +} +for nc in ('title', 'artist', 'album', 'compilation', 'rating', 'path', \ +'length', 'size', 'track number', 'year', 'bitrate', 'sample rate', 'volume', \ +'last played time', 'last skipped time', 'mtime', 'disc number', 'total discs', \ +'BPM', 'movie flag'): + sort_criteria[nc.replace(' ', '').lower()] = nc + + +re_sortspec = re.compile(r'^([<>+-]*)(.*?)([<>+-]*)$') +class SSParseError(Exception): pass + +class SortSpec: + def __init__(self, pattern=None): + if pattern: + self.parse(pattern) + else: + self.criteria = [] + + def parse(self, pattern): + self.criteria = filter(None, map(self._parse_criterion, pattern.split(','))) + + def _parse_criterion(self, text): + text = text.strip() + if not text: return None + m = re_sortspec.match(text) + if not m: + raise SSParseError("invalid sort criterion `%s'" % text) + text = m.group(2).strip() + key = text.lower().replace('_', '').replace(' ', '') + try: + criterion = sort_criteria[key] + except KeyError: + raise SSParseError("unknown sort criterion `%s'" % text) + if isinstance(criterion, bytes): + criterion = cmp_key(criterion) + modifiers = m.group(1) + m.group(3) + order = 1 + if '-' in modifiers: order = -1 + empty_pos = -1 + if '<' in modifiers: empty_pos = 1 + return (criterion, order, empty_pos) + + def _cmp(self, a, b): + for cmp_func, order, empty_pos in self.criteria: + res = cmp_func(self.tracks[a], self.tracks[b], order, empty_pos) + if res: return res + return cmp(a, b) + + def sort(self, tracks): + self.tracks = tracks + index = list(range(len(self.tracks))) + index.sort(self._cmp) + del self.tracks + return [tracks[i] for i in index] + + def __add__(self, other): + self.criteria += other.criteria + return self + + def __len__(self): + return len(self.criteria) + + +################################################################################ +## playlist processing ## +################################################################################ + +def add_scripted_playlist(db, tracklist, list_name, include, exclude, shuffle=False, changemask=0, sort=None): + if not(list_name) or not(include or changemask) or not(tracklist): + return + tracks = [] + log("Processing playlist `%s': " % (list_name), True) + for track in tracklist: + if not 'original path' in track: + continue # we don't know the real name of this file, so skip it + name = track['original path'].encode(sys.getfilesystemencoding(), 'replace').lower() + ok = changemask & track.get('changed', 0) + for pattern in include: + if fnmatch.fnmatch(name, pattern): + ok = True + break + for pattern in exclude: + if fnmatch.fnmatch(name, pattern): + ok = False + break + if ok: + tracks.append(track) + log("%d tracks\n" % len(tracks)) + if not tracks: + return + if shuffle == 1: + shuffle = BalancedShuffle() + for info in tracks: + shuffle.add(info.get('original path', None) or info.get('path', "???"), info) + tracks = shuffle.shuffle() + if shuffle == 2: + random.shuffle(tracks) + if sort: + tracks = sort.sort(tracks) + db.add_playlist(tracks, list_name) + + +def process_m3u(db, tracklist, index, filename, skip_album_playlists): + if not(filename) or not(tracklist): + return + basedir, list_name = os.path.split(filename) + list_name = str(os.path.splitext(list_name)[0], sys.getfilesystemencoding(), 'replace') + log("Processing playlist `%s': " % (list_name), True) + try: + f = open(filename, "r") + except IOError as e: + log("ERROR: cannot open `%s': %s\n" % (filename, e.strerror)) + tracks = [] + + # collect all tracks + for line in f: + line = line.strip() + if line.startswith('#'): + continue # comment or EXTM3U line + line = os.path.normpath(os.path.join(basedir, line)).replace("\\", "/").lower() + try: + tracks.append(tracklist[index[line]]) + except KeyError: + continue # file not found -> sad, but not fatal + f.close() + + # check if it's an album playlist + if skip_album_playlists: + ref_album = None + ok = True # "we don't know enough about this playlist, so be optimistic" + for info in tracks: + if not 'album' in info: continue + if not ref_album: + ref_album = info['album'] + elif info['album'] != ref_album: + ok = True # "this playlist is mixed-album, so it's clean" + break + else: + ok = False # "all known tracks are from the same album, how sad" + if not ok: + # now check if this playlist really covers the _whole_ album + ok = len(tracks) + for info in tracklist: + try: + if info.get('album', None) == ref_album: + ok -= 1 + if ok < 0: break + except (TypeError, UnicodeDecodeError): + # old (<0.3.0) cache files contain non-unicode information + # for ID3v1 tags which can cause trouble here, so ... + continue + if not(ok) : + log("album playlist, discarding.\n") + return + + # finish everything + log("%d tracks\n" % len(tracks)) + if not tracks: + return + db.add_playlist(tracks, list_name) + + +def make_directory_playlists(db, tracklist): + log("Processing directory playlists ...\n") + dirs = {} + for track in tracklist: + path = track.get('original path', None) + if not path: continue + for dir in path.split('/')[:-1]: + if not dir: continue + if dir in dirs: + dirs[dir].append(track) + else: + dirs[dir] = [track] + dirlist = dirs.keys() + dirlist.sort(fncmp) + + for dir in dirlist: + log("Processing playlist `%s': " % dir, True) + tracks = dirs[dir] + tracks.sort(trackcmp) + log("%d tracks\n" % len(tracks)) + db.add_playlist(tracks, dir) + + +shuffle_options = { + "0": 0, "no": 0, "off": 0, "false": 0, "disabled": 0 , "none": 0, + "1": 1, "yes": 1, "on": 1, "true": 0, "enabled": 1, "balanced": 1, + "2": 2, "random": 2, "standard": 2, +} + +def parse_master_playlist_file(): + # helper function + def yesno(s): + if s.lower() in ('true', 'enable', 'enabled', 'yes', 'y'): + return 1 + try: + return (int(s) != 0) + except ValueError: + return 0 + # default values + skip_album_playlists = True + directory_playlists = False + lists = [] + # now we're parsing + try: + f = open(MASTER_PLAYLIST_FILE, "r") + except IOError: + return (skip_album_playlists, directory_playlists, lists) + include = [] + exclude = [] + list_name = None + shuffle = 0 + changemask = 0 + sort = SortSpec() + lineno = 0 + for line in f: + lineno += 1 + line = line.split(';', 1)[0].strip() + if not line: continue + if (line[0] == '[') and (line[-1] == ']'): + if list_name and (include or changemask): + lists.append((list_name, include, exclude, shuffle, changemask, sort)) + include = [] + exclude = [] + list_name = line[1:-1] + shuffle = False + changemask = 0 + sort = SortSpec() + continue + try: + key, value = [x.strip().replace("\\", "/") for x in line.split('=')] + except ValueError: + continue + key = key.lower().replace(' ', '_') + if not value: + log("WARNING: In %s:%d: key `%s' without a value\n" % (MASTER_PLAYLIST_FILE, lineno, key)) + continue + if key == "skip_album_playlists": + if list_name: log("WARNING: In %s:%d: global option `%s' inside a playlist\n" % (MASTER_PLAYLIST_FILE, lineno, key)) + skip_album_playlists = yesno(value) + elif key == "directory_playlists": + if list_name: log("WARNING: In %s:%d: global option `%s' inside a playlist\n" % (MASTER_PLAYLIST_FILE, lineno, key)) + directory_playlists = yesno(value) + elif key == "shuffle": + try: + shuffle = shuffle_options[value.lower()] + except KeyError: + log("WARNING: In %s:%d: invalid value `%s' for shuffle option\n" % (MASTER_PLAYLIST_FILE, lineno, value)) + elif key == "new": + changemask = (changemask & (~2)) | (yesno(value) << 1) + elif key == "changed": + changemask = (changemask & (~1)) | yesno(value) + elif key == "sort": + try: + sort = SortSpec(value) + sort + except SSParseError as e: + log("WARNING: In %s:%d: %s\n" % (MASTER_PLAYLIST_FILE, lineno, e)) + elif key in ("include", "exclude"): + if value[0] == "/": + value = value[1:] + if os.path.isdir(value): + if value[-1] != "/": + value += "/" + value += "*" + if key == "include": + include.append(value.lower()) + else: + exclude.append(value.lower()) + else: + log("WARNING: In %s:%d: unknown key `%s'\n" % (MASTER_PLAYLIST_FILE, lineno, key)) + f.close() + if list_name and (include or changemask): + lists.append((list_name, include, exclude, shuffle, changemask, sort)) + return (skip_album_playlists, directory_playlists, lists) + + +################################################################################ +## artwork ## +################################################################################ + +re_cover = re.compile(r'[^a-z]cover[^a-z]') +re_front = re.compile(r'[^a-z]front[^a-z]') +def find_good_artwork(files, base): + if not files: + return None # sorry, no files here + dirname, basename = os.path.split(base) + if not basename: + dirname, basename = os.path.split(base) + basename = basename.strip().lower() + candidates = [] + for name in files: + ref = os.path.splitext(name)[0].strip().lower() + # if the file has the same name as the directory, we'll use that directly + if ref == basename: + return name + ref = "|%s|" % ref + score = 0 + if re_cover.search(ref): + # if the name contains the word "cover", it's a good candidate + score = -1 + if re_front.search(ref): + # if the name contains the word "front", that's even better + score = -2 + candidates.append((score, name.lower(), name)) + candidates.sort() + return candidates[0][2] # return the candidate with the best score + + +def GenerateArtwork(model, tracklist): + # step 0: check PIL availability + if not iTunesDB.PILAvailable: + log("ERROR: Python Imaging Library (PIL) isn't installed, Artwork is disabled.\n") + log(" Visit http://www.pythonware.com/products/pil/ to get PIL.\n") + return + + # step 1: generate an artwork list + artwork_list = {} + for track in tracklist: + artwork = track.get('artwork', None) + if not artwork: + continue # no artwork file + dbid = track.get('dbid', None) + if not dbid: + continue # artwork doesn't make sense without a dbid + if artwork in artwork_list: + artwork_list[artwork].append(dbid) + else: + artwork_list[artwork] = [dbid] + + # step 2: generate the artwork directory (if it doesn't exist already) + try: + os.mkdir(ARTWORK_DIR[:-1]) + except OSError: + pass # not critical (yet) + + # step 3: try to load the artwork cache + try: + with open(ARTWORK_CACHE_FILE, "rb") as f: old_cache = pickle.load(f) + except (IOError, EOFError, pickle.PickleError): + old_cache = ({}, {}) + + # step 4: generate and save the ArtworkDB + artwork_db, new_cache, dbid2mhii = iTunesDB.ArtworkDB(model, artwork_list, cache_data=old_cache) + backup(ARTWORK_DB_FILE) + try: + with open(ARTWORK_DB_FILE, "w") as f: f.write(artwork_db) + except IOError as e: + log("FAILED: %s\n" % e.strerror + + "ERROR: The ArtworkDB file could not be written. This means that the iPod will\n" + + "not show any artwork items.\n") + + # step 5: save the artwork cache + try: + with open(ARTWORK_CACHE_FILE, "wb") as f: pickle.dump(new_cache, f) + except (IOError, EOFError, pickle.PickleError): + log("ERROR: can't save the artwork cache\n") + + # step 6: update the 'mhii link' field + for track in tracklist: + dbid = track.get('dbid', None) + mhii = dbid2mhii.get(dbid, None) + if mhii: + track['mhii link'] = mhii + elif 'mhii link' in track: + del track['mhii link'] + + +################################################################################ +## FREEZE and UPDATE action ## +################################################################################ + +def Freeze(CacheInfo=None, UpdateOnly=False): + global g_freeze_error_count + if not CacheInfo: CacheInfo = load_cache((None, [])) + state, cache = CacheInfo + + if UpdateOnly: + if (state != "frozen") and not(Options['force']): + confirm(""" +NOTE: The database is not frozen, the update will not work as expected! +""") + else: + if (state == "frozen") and not(Options['force']): + confirm(""" +NOTE: The database is already frozen. +""") + state = "frozen" + + # allocate the filename allocator + if not UpdateOnly: + log("Scanning for present files ...\n", True) + try: + allocator = Allocator(MUSIC_DIR[:-1]) + except (IOError, OSError): + log("FATAL: can't read or write the music directory!\n") + return + + # parse the master playlist setup file + skip_album_playlists, directory_playlists, master_playlists = parse_master_playlist_file() + + # index the track cache + log("Indexing track cache ...\n", True) + index = make_cache_index(cache) + + # now go for the real thing + playlists = [] + if not UpdateOnly: + log("Searching for playable files ...\n", True) + tracklist = freeze_dir(cache, index, allocator, playlists) + log("Scan complete: %d tracks found, %d error(s).\n" % (len(tracklist), g_freeze_error_count)) + + # cache save checkpoint + save_cache((state, tracklist)) + else: + # in update mode, use the cached track list directly + tracklist = cache + + # artwork processing + if not UpdateOnly: + model = Options['model'] + if not model: + try: + with open(MODEL_FILE, "r") as f: model = f.read().strip()[:10].lower() + log("\nLoaded model name `%s' from the cache.\n" % model) + except IOError: + pass + if model: + model = model.strip().lower() + if not(model in iTunesDB.ImageFormats): + log("\nWARNING: model `%s' unrecognized, skipping Artwork generation.\n" % model) + else: + try: + with open(MODEL_FILE, "w") as f: f.write(model) + except IOError: + pass + else: + log("\nNo model specified, skipping Artwork generation.\n") + else: + model = None + + # generate track IDs + if not UpdateOnly: + iTunesDB.GenerateIDs(tracklist) + + # generate the artwork list + if model and not(UpdateOnly): + log("\nProcessing Artwork ...\n", True) + GenerateArtwork(model, tracklist) + + # build the database + log("\nCreating iTunesDB ...\n", True) + db = iTunesDB.iTunesDB(tracklist, name="%s %s"%(__title__, __version__)) + + # save the tracklist as the cache for the next run + save_cache((state, tracklist)) + + # add playlists according to the master playlist file + for listspec in master_playlists: + add_scripted_playlist(db, tracklist, *listspec) + + # process all m3u playlists + if playlists: + log("Updating track index ...\n", True) + index = make_cache_index(tracklist) + for plist in playlists: + process_m3u(db, tracklist, index, plist, skip_album_playlists) + + # create directory playlists + if directory_playlists: + make_directory_playlists(db, tracklist) + + # finish iTunesDB and apply hash stuff + log("Finalizing iTunesDB ...\n") + db = db.finish() + fwids = hash58.GetFWIDs() + try: + f = open(FWID_FILE, "r") + fwid = f.read().strip().upper() + f.close() + if len(fwid) != 16: + fwid = None + except IOError: + fwid = None + store_fwid = False + if fwid: + # preferred FWID stored on iPod + if fwids and not(fwid in fwids): + log("WARNING: Stored serial number doesn't match any connected iPod!\n") + else: + # auto-detect FWID + if fwids: + fwid = fwids[0] + store_fwid = (len(fwids) == 1) + if not store_fwid: + log("WARNING: Multiple iPods are connected. If the iPod you are trying to freeze is\n" + + " a recent model, it might not play anything. Please try again with the\n" + + " other iPod unplugged.\n") + else: + log("WARNING: Could not determine your iPod's serial number. If it's a recent model,\n" + + " it will likely not play anything!\n") + if fwid: + db = hash58.UpdateHash(db, fwid) + if store_fwid: + try: + f = open(FWID_FILE, "w") + f.write(fwid) + f.close() + except IOError: + pass + + # write iTunesDB + write_ok = True + backup(DB_FILE) + try: + f = open(DB_FILE, "wb") + f.write(db) + f.close() + except IOError as e: + write_ok = False + log("FAILED: %s\n" % e.strerror + + "ERROR: The iTunesDB file could not be written. This means that the iPod will\n" + + "not play anything.\n") + + # write iPod shuffle stuff (if necessary) + if os.path.exists(CONTROL_DIR + "iTunesSD"): + backup(CONTROL_DIR + "iTunesSD") + log("Creating iTunesSD ... ", True) + db = iTunesDB.iTunesSD(tracklist) + try: + f = open(CONTROL_DIR + "iTunesSD", "wb") + f.write(db) + f.close() + log("\n") + except IOError as e: + write_ok = False + log("FAILED: %s\n" % e.strerror + + "ERROR: The iTunesSD file could not be written. This means that the iPod will\n" + + "not play anything.\n") + delete(CONTROL_DIR + "iTunesShuffle") + delete(CONTROL_DIR + "iTunesPState") + + # generate statistics + if write_ok: + log("\nYou can now unmount the iPod and listen to your music.\n") + sec = int(sum([track.get('length', 0.0) for track in tracklist]) + 0.5) + log("There are %d tracks (%d:%02d:%02d" % (len(tracklist), sec/3600, (sec/60)%60, sec%60)) + if sec > 86400: log(" = %.1f days" % (sec / 86400.0)) + log(") waiting for you to be heard.\n") + + # finally, save the tracklist as the cache for the next run + save_cache((state, tracklist)) + + +################################################################################ +## UNFREEZE action ## +################################################################################ + +def Unfreeze(CacheInfo=None): + if not CacheInfo: CacheInfo = load_cache((None, None)) + state, cache = CacheInfo + + try: + cache_len = len(cache) # type: ignore + except: + cache_len = None + + if not(state) or (cache_len is None): + fatal("can't unfreeze: rePear cache is missing or broken") + raise Exception + if state!="frozen" and not(Options['force']): + confirm(""" +NOTE: The database is already unfrozen. +""") + + log("Moving tracks back to their original locations ...\n") + success = 0 + failed = 0 + for info in cache: # type: ignore + src = printable(info.get('path', "")) + dest = printable(info.get('original path', "")) + if not src: + log("ERROR: track lacks path attribute\n") + continue + if not dest: + continue # no original path + log("%s " % dest) + if move_file(src, dest): + failed += 1 + else: + success += 1 + log("Operation complete: %d tracks total, %d moved back, %d failed.\n" % \ + (len(cache), success, failed)) + log("\nYou can now manage the music files on your iPod.\n") + save_cache(("unfrozen", cache)) + + +################################################################################ +## the configuration actions ## +################################################################################ + +def ConfigFWID(): + log("Determining serial number (FWID) of attached iPods ...\n") + fwids = hash58.GetFWIDs() + try: + f = open(FWID_FILE, "r") + fwid = f.read().strip().upper() + f.close() + if len(fwid) != 16: + fwid = None + except IOError: + fwid = None + if not fwids: + # no FWIDs detected + if fwid: + return log("No iPod detected, but FWID is already set up (%s).\n\n" % fwid) + else: + return log("No iPod detected, can't determine FWID.\n\n") + if len(fwids) > 1: + # multiple FWIDs detected + if fwid and (fwid in fwids): + return log("Multiple iPods detected, but FWID is already set up (%s).\n\n" % fwid) + else: + return log("Multiple iPods detected, can't determine FWID.\n" + \ + "Please unplug all iPods except the one you're configuring\n\n") + # exactly one FWID detected + log("Serial number detected: %s\n" % fwids[0]) + if fwid and (fwid != fwids[0]): + log("Warning: This serial number is different from the one that has been stored on\n" + \ + " the iPod (%s). Storing the new FWID anyway.\n" % fwid) + fwid = fwids[0] + if not fwid: + return log("\n") + try: + f = open(FWID_FILE, "w") + f.write(fwid) + f.close() + log("FWID saved.\n\n") + except IOError: + log("Error saving the FWID.\n\n") + + +models = ( + (None, "other/unspecified (no cover artwork)"), + ('photo', '4g', "iPod photo (4G)"), + ('video', '5g', "iPod video (5G)"), + ('classic', '6g', "iPod classic (6G)"), + ('nano', 'nano1g', 'nano2g', "iPod nano (1G/2G)"), + ('nano3g', "iPod nano (3G, \"fat nano\")"), + ('nano4g', "iPod nano (4G)"), +) +def is_model_ok(mod_id): + for m in models[1:]: + if mod_id in m[:-1]: + return True + return False + +def ConfigModel(): + try: + with open(MODEL_FILE, "r") as f: model = f.read().strip().lower() + if not is_model_ok(model): model = None + except IOError: model = None + print("Select iPod model:") + default = 0 + for i in range(len(models)): + if model in models[i][:-1]: + default = i + c = "*" + else: + c = " " + print(c, "%d." % i, models[i][-1]) + try: + answer = int(input("Which model is this iPod? [0-%d, default %d] => " % (len(models) - 1, default))) + except (IOError, EOFError, KeyboardInterrupt, ValueError): + answer = default + if (answer < 0) or (answer >= len(models)): + answer = default + if answer: + try: + f = open(MODEL_FILE, "w") + f.write(models[answer][0]) + f.close() + log("Model set to `%s'.\n\n" % models[answer][-1]) + except IOError: + log("Error: cannot set model.\n\n") + else: + delete(MODEL_FILE, True) + log("Model set to `other'.\n\n") + + +re_ini_key = re.compile(r'^[ \t]*(;)?[ \t]*(\w+)[ \t]*=[ \t]*(.*?)[ \t]*$', re.M) +class INIKey: + def __init__(self, key, value): + self.key = key + self.value = value + self.present = False + self.valid = False + def check(self, m): + if not m: return + if m.group(2).lower() != self.key: return + self.present = True + valid = not(not(m.group(1))) + if not(valid) and self.valid: return + self.start = m.start(3) + self.end = m.end(3) + self.comment = m.start(1) + def apply(self, s): + if not self.present: + if not s.endswith("\n"): s += "\n" + return s + "%s = %s\n" % (self.key, self.value) + s = s[:self.start] + self.value + s[self.end:] + if not(self.valid) and (self.comment >= 0): + s = s[:self.comment] + s[self.comment+1:] + return s + +def ConfigAll(): + ConfigFWID() + ConfigModel() + + +################################################################################ +## the two minor ("also-ran") actions ## +################################################################################ + +def Auto(): + state, cache = load_cache((None, [])) + if state == 'frozen': + Unfreeze((state, cache)) + else: + Freeze((state, cache)) + +def Reset(): + state, cache = load_cache((None, [])) + if (state == 'frozen') and not(Options['force']): + confirm(""" +WARNING: The database is currently frozen. If you reset the cache now, you will + lose all file name information. This cannot be undone! +""") + return + try: + os.remove(CACHE_FILE) + except OSError: + try: + save_cache((None, [])) + except IOError: + pass + delete(ARTWORK_CACHE_FILE, True) + log("\nCache reset.\n") + +################################################################################ +## the main function ## +################################################################################ + +class MyOptionParser(optparse.OptionParser): + def format_help(self, formatter=None): + models = list(iTunesDB.ImageFormats.keys()) + models.sort() + return optparse.OptionParser.format_help(self, formatter) + """ +Artwork is supported on the following models: + """ + ", ".join(models) + """ + +actions: + help show this help message and exit + freeze move all music files into the iPod's library + unfreeze move music files back to their original location + update update the frozen database without scanning for new files + dissect generate an Artist/Album/Title directory structure + reset clear rePear's metadata cache + cfg-fwid determine the iPod's serial number and save it + cfg-model interactively configure the iPod model + config run all of the configuration steps +If no action is specified, rePear automatically determines which of the +`freeze' or `unfreeze' actions should be taken. + +""" + +if __name__ == "__main__": + parser = MyOptionParser(version=__version__, + usage="%prog [options] []") + parser.add_option("-r", "--root", action="store", default=None, metavar="PATH", help="set the iPod's root directory path") + parser.add_option("-l", "--log", action="store", default="repear.log", metavar="FILE", help="set the output log file path") + parser.add_option("-m", "--model", action="store", default=None, metavar="MODEL", help="specify the iPod model (REQUIRED for artwork support)") + parser.add_option("-L", "--lameopts", action="store", default=DEFAULT_LAME_OPTS, metavar="CMDLINE", help="set the LAME encoder options (default: %s)" % DEFAULT_LAME_OPTS) + parser.add_option("-f", "--force", action="store_true", default=False, help="skip confirmation prompts for dangerous actions") + parser.add_option("-p", "--playlist", action="store", default=None, metavar="FILE", help="specify playlist config file") + (opts, args) = parser.parse_args() + Options = opts.__dict__ + + if len(args)>1: parser.error("too many arguments") + if args: + action = args[0].strip().lower() + else: + action = "auto" + if action == "help": + parser.print_help() + sys.exit(0) + if not action in ( + 'auto', 'freeze', 'unfreeze', 'update', 'dissect', 'reset', \ + 'config', 'cfg-fwid', 'cfg-model' + ): + parser.error("invalid action `%s'" % action) + + oldcwd = os.getcwd() + open_log() + log("%s\n%s\n\n" % (banner, len(banner) * '-')) + if not logfile: + log("WARNING: can't open log file `%s', logging disabled\n\n" % Options['log']) + goto_root_dir() + + if Options['playlist']: + first = Options['playlist'].replace("\\", "/").split('/', 1)[0] + if first in (".", ".."): MASTER_PLAYLIST_FILE = os.path.normpath(os.path.join(oldcwd, Options['playlist'])) + else: MASTER_PLAYLIST_FILE = Options['playlist'] + log("master playlist file is `%s'\n" % MASTER_PLAYLIST_FILE) + + log("\n") + try: + if action=="auto": Auto() + elif action=="freeze": Freeze() + elif action=="unfreeze": Unfreeze() + elif action=="update": Freeze(UpdateOnly=True) + elif action=="dissect": Dissect() + elif action=="reset": Reset() + elif action=="config": ConfigAll() + elif action=="cfg-fwid": ConfigFWID() + elif action=="cfg-model": ConfigModel() + else: log("Unknown action, don't know what to do.\n") + code = 0 + except SystemExit as e: + sys.exit(e.code) + except KeyboardInterrupt: + log("\n" + 79*'-' + "\n\nAction aborted by user.\n") + code = 2 + except: + log("\n" + 79*'-' + "\n\nOOPS -- rePear crashed!\n\n") + traceback.print_exc(file=Logger) + log("\nPlease inform the author of rePear about this crash by sending the\nrepear.log file.\n") + code = 1 + quit(code) diff --git a/repear_playlists.ini b/repear_playlists.ini new file mode 100644 index 0000000..f36aae4 --- /dev/null +++ b/repear_playlists.ini @@ -0,0 +1,32 @@ +; ***** rePear master playlist file ***** +; This file contains descriptions of playlists that rePear shall create +; automatically based on certain properties of the audio files. It consists of +; two parts: global options and playlist specifications. +; For more detailed information on how to use this file, read usage.html + +; ----- global options ----- + +skip album playlists = yes +; YES if .m3u files that are just a directory index shall be skipped [default] +; NO if every .m3u playlist found on the iPod shall be turned into a playlist + +directory playlists = no +; YES if rePear shall create a playlist for *every* folder found on the iPod +; NO if it shall not do that [default] + +; ----- playlist specifications ----- +; playlist specifications use the following syntax: +; [Playlist Name] ; the playlist name +; include = /path/to/directory ; include a directory into this playlist +; exclude = /another/diretory ; exclude a directory from the playlist +; new = 1 ; include all new tracks [optional] +; changed = 1 ; include all changed tracks [optional] +; shuffle = balanced ; enable balanced playlist shuffling [opt.] +; shuffle = random ; enable random playlist shuffling [opt.] +; sort = ; sort tracks [optional, see documentation] + +; an example playlist that contains all tracks that have been modified since +; the last freeze operation +[Hot New Stuff] +new = 1 +changed = 1