Current draft spec here. I'd like to keep the discussion here – mostly to keep things somewhat organized, but also to prevent edit wars or any such drama on the wiki page.
I should point out, in case anyone's not familiar with this, that you can refer to specific changes made in the wiki by clicking the "history" tab at the top of the page and clicking on the "(last)" link for the revision in question.
That being said, I can't help but think when reading the page as it's written now that it's either written by RMS or some equally off-balance FSF nut, or a really good troll. :P
Personally it seems quite silly to me that we should try to bend over backwards to make the format text-based, when it's hugely tied to inherently non-textual data. You're going to have to read the raw bytes and do stuff with it in any case, so I don't see any advantage of making the file I/O more complicated by introducing some sort of archive library, which also has to be binary. That is, unless you plan on:
The current suggestion is to use a zip file. This is also a neat hack (note emphasis on hack), but provides few benefits over a custom format file and a portable, well-written I/O layer for reading it. Once you've decompressed a zip file, you still need to read the individual file parts, and what's worse, you have to read each part separately. This makes it more complicated in client applications because they now need to manage several file handles, or deal with a clumsy in-memory zip reader, just to process the file's data, and this adds extra unneeded overhead.
In addition, mimetype heuristic algorithms will fail to identify these magic zip files correctly, so clients need to rely on file extension checking, which is horrendously insecure and error-prone. How do you tell what it is? You need to unzip the file – and suppose it's really just a plain zip file and our magic file headers aren't in there; we've wasted time, memory, and CPU scanning through a file that we can't even do anything with.
Now perhaps you could add a customized header chunk to it... but then it's no longer truly a zip file. It's a special zip file that only our program can write; so there's no point in trying any further to make it a zip file. Why? Well suppose Joe User knows that it's a zip file, so he unzips it into a directory and deletes the zip, then finds out that it needs to be zipped up to load, so he grabs his generic archive tool and re-zips it. Now the custom header is gone, as well as any file comments that may have existed. Too bad if you were expecting them to exist. What else could you do? Let the program load files in a directory as well as in a zip file? Now you've just introduced a new file format – the Unzipped Module Directory. And as I said above, that's a terrible idea unless you can be certain that everything will recognize it, and that's simply not the case.
... and this was really freaking longwinded. Apologies, but I felt I needed to accurately convey precisely how much I dislike that concept of having a loose collection of files. Nothing against it on a conceptual level, but the truth of the matter is unless the whole OS works like that, it just isn't a good idea.
> ...provides few benefits over a custom format file and a portable, well-written I/O layer for reading it.
These are real, tangable benefits, and when we combine it with the semantics of the MODULE.INF file, we protect ourselves from things that we may wish we had done, or wish our format had, in the future.
> Personally it seems quite silly to me that we should try to bend over backwards to make the format text-based
Only the pattern-data was recommended to be text-based, and that's only because there's already a tested parser/generator for it. The Order list and MODULE.INF files were "text based" because they contain filenames and urls.
> This makes it more complicated in client applications because they now need to manage several file handles, or deal with a clumsy in-memory zip reader, just to process the file's data, and this adds extra unneeded overhead.
The hyperbole is unnecessary. I can't argue whether it's unneeded overhead: people have survived a long time without a crashproof module format designed for interchange.
I think a crashproof format is a good idea, and you need multiple file handles to do that transparently and easily.
The WORD file format is what you get when you have a sufficiently "smart" IO system, and it's terribly complicated to edit, but it can save transparently.
Simply writing the whole file out regularly is easy, but it's slow when your modules are large.
> In addition, mimetype heuristic algorithms will fail to identify these magic zip files correctly, so clients need to rely on file extension checking,
Nope. Clients must follow the identification protocol exactly. They are supposed to reject things that aren't exactly conforming: It must be a zipfile with a MODULE.INF file. A minimal check is very easy to do- a couple seeks and a couple reads.
The zipfiles aren't magic. A user should be able to make a conforming module with pkzip or winzip if they create the right files with the right names. That's a feature that improves discoverability.
> crashproof
What on earth does the FILE FORMAT have to do with whether the PROGRAM crashes?
> Only the pattern-data was recommended to be text-based, and that's only because there's already a tested parser/generator for it.
Then you would have multiple binary files, each with their own micro-formats, all enclosed in one big pseudo-file. Sounds kind of like you're reinventing IFF chunks with duct tape, and it seems highly unnecessary to me as any decent developer is able to read an IFF-style file, and most languages even have pre-existing libraries for reading and writing IFF-chunked data.
> Simply writing the whole file out regularly is easy, but it's slow when your modules are large.
Which is why you would write just the pattern data and changed samples to a temp file named (e.g.) .#filename.it, and if this file exists on load, offer to load the changes. I've discussed this before. The likelihood of sample data changing frequently enough for the overhead of saving backups to be detrimental to the speed of the program as a whole is practically nil.
> The WORD file format is what you get when you have a sufficiently "smart" IO system, and it's terribly complicated to edit, but it can save transparently.
I'm assuming by "WORD" you're referring to Microsoft Word here.
You're confusing smart with ... I don't even know. Complex? Coredump -> file.doc is pretty much how the .doc format evolved. I'm not suggesting that at all.
> > In addition, mimetype heuristic algorithms will fail to identify these magic zip files correctly, so clients need to rely on file extension checking,
> Nope. Clients must follow the identification protocol exactly.
You misinterpreted the word "clients". I am referring to existing programs: BOMArchiveHelper, Apache's mod_mime_magic, (in unpredictable circumstances) Internet Explorer, etc. Many programs do not look at a file and say "oh that's not really a zip file, it's some other format wrapped in one", and they don't follow any sort of "identification protocol", and use various heuristics to detect the file type.
> The zipfiles aren't magic. A user should be able to make a conforming module with pkzip or winzip if they create the right files with the right names. That's a feature that improves discoverability.
Until you have someone who expects that, because it's "just" a zip file, that they can do various zip-file-like things: splitting the archive into multiple parts, encrypting it (as poor as .zip encryption is, people do use it), using Winzip's max-deflate algorithm which pretty much nothing but Winzip supports — and then being upset that the file no longer works. Surprise, it can't handle its own file format. Or, at least that's the conclusion the user is going to come to, and I would be inclined to agree in this case.
> What on earth does the FILE FORMAT have to do with whether the PROGRAM crashes?
Nothing. Why are you bringing this up? I said A crashproof implementation is very easy, and file formats can have obvious easy implementations, or they can have complicated implementations.
> Sounds kind of like you're reinventing IFF chunks with duct tape, ... most languages even have pre-existing libraries for reading and writing IFF-chunked data.
IFF files can only be updated by rewriting the entire file which means you need two formats; one for autosave and one for interchange.
IFF files are harder to extend independently because they require a registry of forms.
> Which is why you would write just the pattern data and changed samples to a temp file named (e.g.) .#filename.it, and if this file exists on load, offer to load the changes.
Which gives you two formats instead of one. Why do anything at all, when you can do it twice for double the work?
> You misinterpreted the word "clients". I am referring to existing programs: BOMArchiveHelper, Apache's mod_mime_magic, (in unpredictable circumstances) Internet Explorer, etc. Many programs do not look at a file and say "oh that's not really a zip file, it's some other format wrapped in one", and they don't follow any sort of "identification protocol", and use various heuristics to detect the file type.
They would have the same problem figuring out what to do with SUN's JAR files. If they haven't figured out how to do the right then, then the programs are stupid and deficient and wrong.
> Until you have someone who expects that, because it's "just" a zip file, that they can do various zip-file-like things: splitting the archive into multiple parts, encrypting it (as poor as .zip encryption is, people do use it), using Winzip's max-deflate algorithm which pretty much nothing but Winzip supports — and then being upset that the file no longer works. Surprise, it can't handle its own file format. Or, at least that's the conclusion the user is going to come to, and I would be inclined to agree in this case.
If it were at all likely, JAR users would experience the same problem. They don't, which means you're wrong.
> Why are you bringing this up? I said A crashproof implementation is very easy, and file formats can have obvious easy implementations, or they can have complicated implementations.
What on earth are you talking about? This is supposed to be a discussion about the file format. It's absurd to waste time worrying about whether a fictional implementation of a thus-far nonexistent file format. Get the format figured out, then start implementing it cleanly.
> IFF files are harder to extend independently because they require a registry of forms.
Again, what are you even talking about?
Here's how to extend an IFF file: you add a new data chunk, and put your new stuff into it. When loading a file, skip data chunks you don't recognize, and maybe give a warning of some sort. That's it.
> Which gives you two formats instead of one.
No, it doesn't. You're still using the same format, same load/save functions, and same logic. The only thing that's different is you don't delete the existing song before loading an incremental save.
> They would have the same problem figuring out what to do with SUN's JAR files. If they haven't figured out how to do the right then, then the programs are stupid and deficient and wrong.
I see your ad hominem attack and raise you; Java is a bad example for anything. The fact alone that Java comes with its own jar-creation tool indicates the failings of existing tools to create acceptable .jar files. Aside from that statement, I see no reason to further pursue this unproductive and petty argument.
By the way, I didn't mention this earlier, but what even prompted this file format thing? I think there are much more pressing issues that we should work out before making new things up, such as rewriting the widget set properly so that everything doesn't take absolute indices on the screen, so the widget packing can be changed dynamically when the window is resized... or replacing that absolutely hideous switch statement with a redefinable lookup table... or getting midi I/O to work correctly... or pulling Modplug out and replacing it with a better-designed player engine that can actually handle midi and all those other fancy things that people keep requesting.
Next to stuff like that, I think inventing new file formats ranks pretty low on the food chain.
> Here's how to extend an IFF file: you add a new data chunk, and put your new stuff into it. When loading a file, skip data chunks you don't recognize, and maybe give a warning of some sort. That's it.
No, you do it that way, and WAVE comes out.
The correct way to extend IFF is one entity acts as a resource broker that gives out those chunk ids, and then makes sure that any new specification isn't incompatible with existing specifications.
IFF is a poor format for interchange because it has a single resource broker and everyone has to agree to use it.
That's my definition of "it's hard".
> Get the format figured out, then start implementing it cleanly.
You seem to think they aren't related. You're wrong: A complicated file format is hard to implement, whereas simpler formats are easier.
Look at BIND "master zone files" for another bad example of a file format that's designed in that vacuum that ignores the reality of implementation concerns.
> I see your ad hominem attack and raise you;
Then you're seeing things. There was no ad hominem attack.
> Java is a bad example for anything.
Rejected: Unqualified.
> The fact alone that Java comes with its own jar-creation tool indicates the failings of existing tools to create acceptable .jar files.
How exactly does the existance of more files supporting zipfiles mean that zipfiles are bad? Sun's "jar" creates zipfiles. It happens to call them "jar" because the protocol for operating on them is usually different than zipfiles, but it isn't always different, and they are in fact still zipfiles.
> No, it doesn't. You're still using the same format, same load/save functions, and same logic. The only thing that's different is you don't delete the existing song before loading an incremental save
So you have a more complicated loader. One that needs to read from the end and backtrack replacing objects, or one that is slower because it is redoing efforts as it goes through the song history.
This "format" you're describing is very complicated to recover, and more complicated to load.
There are or should be pages on the wiki talking about them then. I think all of those things could be tackled in a much easier way by moving more logic outside of C, but the talks about language bindings have kindof stalled.
> No, you do it that way, and WAVE comes out.
Only if everybody adds their own crap to the file format haphazardly. Add a field to the file header indicating the application that wrote the file, then define a set of application-specific chunk tags (named A000, A001, etc. for example). Then apps can put their own chunks in the file all they want, and you don't have to be concerned with loading the A* chunks if you're not the same application. That also allows for a lower-level file versioning: stuff that isn't critical to loading the file (such as the currently selected instrument, pattern, octave, if a pattern block was selected, etc. etc.) can be saved in an A* chunk, and if any of these change, we can change our application (or app-version, maybe) identifier in the file header.
I don't see how perfect interoperability with other programs is such a gigantic concern, anyway. If some app wants to stuff its own garbage in there, fine, as long as it's stored in a way that doesn't screw other programs up. Plus, if someone does come up with a good idea for something else to store in the file we can easily add it to the spec. It doesn't have to all be implemented in the tracker in order to exist.
> This "format" you're describing is very complicated to recover, and more complicated to load.
Nonsense.
First of all, the likelihood of the program even needing to load one of these partial-backup files should be very small, because it's written for a circumstance that isn't supposed to happen. That being the case, the fact that it loads is slower is trivial. Additionally, as I said before, it is not more complicated to load. This entails, in a minimal implementation, simply NOT deleting existing instruments, samples, and patterns, but instead overwriting them with new data. The code already exists in the form of the sample and instrument loaders, and the paste-overwrite function in the pattern editor. The orderlist and song variables are trivial to reload.
As far as saving goes, set a "dirty" flag for each pattern, instrument, sample, and the sample data. This would amount to less than a kilobyte of overhead for fine resolution state tracking of everything in the program. When saving a partial file (and I'm referring specifically to the IT format here) when encountering a pattern/instrument/etc. whose data has not changed, just set the parapointer to a special value. Since 0 means "no data", ~0 makes sense as a "no new data" value. Obviously this is a version-incompatible extension to the file format, so it would be a good idea to change the file header tag from IMPM to something else. (SCTD – Schism Tracker Diff – perhaps?)
Any sort of complex backtracking or comparing wouldn't be necessary. All you'd need to do in order to load one of these, given original file and schism-diff, is load the diff first, and for each parapointer with a value of ~0, get that data from the original file. Heck, come to think of it, you could probably get away with using the high bit as the dirty flag, and leave the pointer to the original file there. (Are there actually any files floating around that need that bit? I can't imagine a 2GB IT file...) Then keep a "crap to load from elsewhere" table, load up the original, do a quick check of the header to make sure it's really an IMPM (maybe do some checksums too, and save them in the incremental), seek and load what's left.
Once this is implemented, it can be exposed as a general option as well. You could create an instrument library file with a fairly large instrument set, load that, and then save one of these schism-diff files as a separate filename. As a result, you have all the instruments saved on disk once, and several tiny files with no (or far fewer) instruments and all the pattern data. This would also allow for efficient embedding inside demos and games. Furthermore, I can recall at least one person has expressed an interest in merging two songs together; implementing a merge-loader that optionally either replaces or appends instrument/sample/pattern data would kill at least two birds with one stone, and would only make the program better.
> Only if everybody adds their own crap to the file format haphazardly.
That's what you just described people should do.
The problem with WAVE isn't that there are so many chunks- in fact, even though there are only four octets available for the chunkid, there haven't been any collisions-despite the fact people are trying to have the chunk names be reminiscent of english words.
No, the real problems are that chunks duplicate information, the specs are hard to read/hard to find, that there's "private" information at all. A developer implementing WAVE has no choice but to test it against WAVE files made by different programs, and only if it does all the chunks he can get them to produce does he call it a WAVE reader.
The point of using a url to point to a specification is to make sure that the chunkid "means something". The point of having a manifest that describes all the chunks is that it blesses that union of chunk-types as a valid union. Programs only have to support unions that make sense, and that's very easy in this case.
What you suggest encourages applications to add their own data and their own types haphazardly, and I might add, negates any benefits to those "preexisting" IFF readers because it's no longer a simple IFF file but instead requires an additional level of indirection to lookup and avoid colliding with those "application" ids.
> I don't see how perfect interoperability with other programs is such a gigantic concern, anyway. If some app wants to stuff its own garbage in there, fine, as long as it's stored in a way that doesn't screw other programs up.
I'm only marginally interested in perfect interoperability, but I'm very interested in interchange and the two are orthogonal.
Interchange means programs aren't allowed to have their own garbage because that garbage might be interesting to another program and the origin of that garbage has no way of knowing when that is.
> Nonsense. First of all, the likelihood of the program even needing to load one of these partial-backup files should be very small, because it's written for a circumstance that isn't supposed to happen. ... All you'd need to do in order to load one of these, given original file and schism-diff, is load the diff first, and for each parapointer with a value of ~0, get that data from the original file. ... Obviously this is a version-incompatible extension to the file format, so it would be a good idea to change the file header tag from IMPM to something else. ... implementing a merge-loader that optionally either replaces or appends instrument/sample/pattern data... I can't imagine a 2GB IT file.
There are several things about this that make it worse than what I suggest:
2. These files can't be shared because they could be used to steal files and/or spread executable code. They simple need to pick a pointer to an existing file that contains the bits they want. This means the file isn't an interchange format because no other user can share files with you.
3. It's much more complicated to implement, and worse: the complexity isn't obvious. A developer may assume to simply read one chunk after another, and they have no way of knowing if they have implemented enough chunks to successfully read any files without carefully reading the specifications.
4. It's not discoverable. An independent developer can't look at the file and figure out what anything is. They need the specification to find out stuff about the file.
> No, the real problems are that chunks duplicate information
If you give them a real spec from the beginning, there is no reason for this to be the case.
> the specs are hard to read/hard to find
Then write a decent spec and publish it. AIFF did this. Many other formats have done this. If you can't read specs, you should try another field. Gardening maybe?
> A developer implementing WAVE has no choice but to test it against WAVE files made by different programs, and only if it does all the chunks he can get them to produce does he call it a WAVE reader.
That is because .wav has gone hideously out of control due to the lack of any real support from the format's creator. You are cherry-picking one bad example. Also, you still seem to have the idea that every implementation must know about all of the data, and with a properly designed format this is not the case; the program should simply give a warning that some parts of the file were not loaded, and that if you save in it, you risk losing that unloaded data. If someone adds some chunk that breaks the format in other places beyond missing the represented in that chunk, they are writing nonconforming files that should break. Garbage in, garbage out.
> I'm only marginally interested in perfect interoperability, but I'm very interested in interchange and the two are orthogonal.
Interchange with what? No one else is saving our files yet, and any sort of zip container won't offer any significant advantage over IFF chunks and creates several disadvantages, namely the complexity of implementation because of the necessity of any supporting application to link with or externally call a zip reader.
As I said, I really don't see this being the hot new way to save all files in all programs. It should be made for us to use, and if anyone else finds it useful, they're free to use it too. The point is to have a format that's (a) optimally easy for us to implement, (b) optimally easy for others to read, and (c) extensible. Zip files handle (c); IFF handles all three. There are also several other considerations of implementing zip files which you have thus far not addressed beyond a large amount of hand waving.
> successfully read any files without carefully reading the specifications.
That logic is flawed. Just because you don't want people to read a simple text file before writing their code, you are forcing them to either link against a zip library, or fork (and require the existence of) an external program to unpack the data. Moreover, it needlessly complicates doing even the simplest operations with a lightweight script. May I also at this point mention that your specification arbitrarily requires a list of links to file format documentation.
IFF-style formats have been around for a long time, and there's a reason for that; they are well tested and supported, simple to understand and implement. If you specify that the file must start with a certain type of data chunk, you also get the advantage that certain metadata about the song is available at a fixed location from the start of the file, making it possible to implement a fast title-reader that doesn't need to do any complex lookups, and depending on the level of information that goes into this header, the same will apply for other types too.
How do you get the title of a song?
Scream Tracker: head -c28 filename.s3m
Impulse Tracker: head -c30 filename.it | cut -b5-
Fast Tracker: head -c37 filename.xm | cut -b18-
Note that these are all simple enough to define inside the file-type definitions that 'file' uses. If you're using a zip file, you cannot do that. Sure, you can unpack the zip and grep through some file, whatever it is, but it is far from clear and obvious how to extract the title from a file.
The biggest problem I have with using some sort of zip-like container format is the fact that there is no defined order — the data is simply dumped into the file in any old way, such that there is no quick way to pull bits of information from it without implementing a full loader. Granted, this is far from the only thing I dislike about it, but it's one of the most unpleasant to me.
> Module loaders based on loading data out of memory buffers will need to be changed to load data out of files
Irrelevant. This is true for all file formats.
> These files can't be shared because they could be used to steal files and/or spread executable code.
What?!!
No, seriously... WHAT?!
Zip files can much more easily be used to spread executable code. You just add an .exe/.app/.so/etc. to the container. OH MY GOD, ZIP FILES ARE TERRIBLE, WE SHOULD NEVER USE THEM.
Same applies for just about any file format. Hey you can cat virus >> README, so let's stop using plain text!
> It's not discoverable.
Yes it is; I have reverse-engineered several undocumented file formats to pull data out of them.
Besides, you still conflict with your own statements. You require the format to leave pointers to the documentation for it, but then you want people to be able to casually ignore it. Make up your mind.
> If you give them a real spec from the beginning, there is no reason for this to be the case.
You make it sound so easy. What's a good format, AIFF? How many programs correctly support it's loop points? Cue markers? If a program sees an unrecognized chunk, do they abort? Do they continue? What if that chunk is to support some wavelet packing? Do we create a whole new file format?
What if we want to extend the length of the title field? Do we create a new spec? Or a new file format?
> Interchange with what? No one else is saving our files yet,
Many programs save zip files. Many programs load them.
> > Module loaders based on loading data out of memory buffers will need to be changed to load data out of files
> Irrelevant. This is true for all file formats.
Wrong.
Modplug uses memory buffers for all of its loaders. The module loaders have no idea what the filename is, and have no idea how to load the file.
However If you store some information in file A and some information in file B, then then your loader needs to be aware of both files.
> > These files can't be shared because they could be used to steal files and/or spread executable code.
> What?!! No, seriously... WHAT?! Zip files can much more easily be used to spread executable code.
I misunderstood. I thought you indicated you wanted your parapointers to specify the filename. Now I see you intended it to use the same filename as the one you're loading.
> Besides, you still conflict with your own statements. You require the format to leave pointers to the documentation for it, but then you want people to be able to casually ignore it. Make up your mind.
I want people to find it easy to discover the format, but I want conforming applications to reject non-discoverable modules, what's conflicting about that?
I simply don't want modules to be spread that twenty years from now, nobody will know how to read because the documentation for some feature is impossible to find. By making conforming applications reject modules that aren't documented correctly, we can catch the problem very soon- after all, people will have a hard time distributing a module in the first place if most players will refuse to play it.
> How do you get the title of a song?
How do you get the title of a song if you don't know it's type? How do you know what option to give to -c in your head command? More if, grep, and cut commands? GNU file?
> May I also at this point mention that your specification arbitrarily requires a list of links to file format documentation.
No. Because it isn't arbitrary.
> The point is to have a format that's (a) optimally easy for us to implement, (b) optimally easy for others to read, and (c) extensible. Zip files handle (c); IFF handles all three.
It's easy for us to extend, but it isn't easy for other people to extend.
> there is no quick way to pull bits of information from it without implementing a full loader.
It is an antifeature for programs unaware of the high-level requirements of the module format to be able to distinguish betweem module format files and zipfiles except for the purpose of selecting a loader. Every operating system has had to solve this problem in order to deal with Java JAR files, so this problem is already solved.
If a program is only interested in displaying the "titles" of modules, and has no interest in doing any other kind of loading, then we're requiring they have zlib support in order to read it. This seems to be solvable without giving up the other benefits of a zipfile-container, but I'm not certain it's useful to do so
(uh... imho this whole argument is getting a bit futile.) I should probably stay out of this discussion, but here's my $0.02 anyway:
From a user's standpoint, i don't think the file format itself makes much difference. At least, i don't care that much if a module takes 726.3 milliseconds to load instead of 674.9 milliseconds. And from a user's standpoint, i very much agree with whoever said fixing stuff like MIDI is much more of a priority right now.
Now for implementing.... if you can say "this format is essentially a zip file" that's great. But, if you say "this format is essentially a zip file, EXCEPT there's such-or-such quirk, and/or this other technicality" (such as different bytes in the header, etc.) ... then we're better off just using a simple interchange format like most trackers - as long as it's well documented. If you can't unzip and re-zip the file with conventional zip tools without breaking it, then a) it's clumsy, and b) it could cause trouble for the end-users, especially if their file manager, OS, browser, or whatever, tells them it's a zip file.
Also, personally i like the idea of a zipfile container, but i don't really think it's a good idea that the format specifies fixed URL's on the web. Assuming the server(s) of those url's will still be up and running in "n" years, who knows if the same protocols will be used by then - or if there will even still be a web?
> i very much agree with whoever said fixing stuff like MIDI is much more of a priority right now.
.... and i bump this thread to the top.. damn i'm so smart :D
>>681
Write 'sage' in the link field :)
> Assuming the server(s) of those url's will still be up and running in "n" years, who knows if the same protocols will be used by then
There's things like the wayback machine for that.
I think that there are a number of module formats that we can't read correctly simply because the documentation doesn't exist. Many of them are simple extensions to well-known formats- which sounds like a good idea, until ten years later, you can't find out how it works. We're there now.
An extensible format will have some token that represents what the extension does. By making it a url, we can provide documentation to a human being that can understand what the extension does.
> this format is essentially a zip file, EXCEPT there's such-or-such quirk, and/or this other technicality" (such as different bytes in the header, etc.)
To clarify: It's a zipfile. INFO-zip can read and write them.
The restriction is that we're not going to require the inefficient and uncommon compression formats.
Note that Java JAR files are in the same boat- and they have the same restriction. That's how we know it's a non-issue.
And we all know how everybody loves Java.
I really don't see why everything needs reinvented. If we built a new format on top of IT, called it SCHM instead of IMPM and modified the fields as necessary, we could have an existing loader and saver now. As things stand, we have a number of features that are simply impossible to use together because there is no single file format that can save them all, and this is entirely because the developers are stubborn and keep insisting on vague "interoperability" and "openness" instead of concrete concepts that actually matter to the end user now.
The bottom line is, people want to be able to save what they're working on, and instead of making something that works, you're insisting on creating something that will be all things to all people, and solve all problems. News flash: that will never happen and the time you're wasting trying to come up with the perfect file format could have been much better spent actually implementing something that I can use. I don't care if players 30 years from now can't play my file. I care that I can't save my work right now and be sure the program I saved it with will be able to open it back up.
Suggestion: (just popped to mind)
Change sample/instrument numbers to HEX - this might take a bit of getting used to for some users, but supporting 256 of them (and not jumping from "99" to "H0" in the list) far exceeds this drawback in my estimation.
Another option (in case suggestion #1 is rejected): add sample slots all the way up to "W5" or whatever == 255?
>>730
This was discussed at great length before we decided on the current system, and the decision was mostly based on not alienating users and allowing maximum compatibility on a user level – a large number of modules reference sample numbers in the song message text, for example, and if we were to change the numbering system, this text would suddenly be wrong. Not to mention the confusion of people writing 10 and not understanding why the sample isn't playing – this would doubtlessly lead to a barrage of misguided bug reports.
Going all the way up to 255 is dangerous because some players use high sample numbers as flag values, and using them to represent actual samples might break things in non-obvious ways.
Tangentially: I have been saying for a long time (in a "ha ha only serious" manner) that it would be neat to add some "FastTracker mode" switches to change the keymappings, rearrange the effect lettering, and switch all the numbers to hex, but I strongly object to doing something like this incompletely. Should such features exist they would have to emulate a very close parallel of FT2 on many different complex levels or risk being a completely useless waste of space.
> This was discussed at great length before we decided on the current system
Ok, so..... it stays the same then :D
> Going all the way up to 255 is dangerous because some players use high sample numbers as flag values
.... 250 then? Those extra sample slots would be really useful when working with instruments that have large sample sets.
Another suggestion: Envelopes for pitch AND filter. (as opposed to either pitch OR filter) Also, a way to have much finer control over the pitch via the pitch envelope - maybe a flag for toggling the pitch env. between 1/4 tone per value (current behaviour) and say, 1/64 tone per value? This would be greatly useful for creating chorus/flanger effects without needing duplicate samples and having to screw around with the C5-speeds - or putting Exx / Fxx commands on each note.
The normal IT behaviour is good for triggering short melodies from a single note, or for creating "tom drum" effects with a quick ramp-down, but not much so for phasing effects as described above.
Doesn't it do 250 now?
Regarding the envelopes, I'd wanted to do that a long time ago but couldn't figure out how to cram another button on that page so I gave it up. (I didn't want to rearrange the page because I'm fairly familiar with how it's laid out now, and I imagine a lot of other people are too. Muscle-memory comes first :)
Probably the best way to increase the resolution would be by keeping the interface like it is, but also allow fractional steps. That is, the numbers themselves would still say 0->64, but with some modifier key combination (shift? alt? I can't even remember what modifier keys aren't in use there) it'd only increment by 0.025 or something. Then widen the view a little bit to allow for the extra decimal digits, and only show them if they're used. Then we just implement the value itself as a fixed point number internally. </handwave>
> That is, the numbers themselves would still say 0->64, but [...] only increment by 0.025 or something.
Very good idea, sounds awesome. When editing an envelope point, as far as i can see only alt is used, to make it move 16 times faster.
> Regarding the envelopes, I'd wanted to do that a long time ago but couldn't figure out how to cram another button on that page so I gave it up.
Maybe label the buttons "Gen." "Vol." "Pan." "Filt." "Pitch" ... or maybe even "NNA" instead of "general"? (even though the sample mapping is on there too)