DM10 file formats (warning: technical!)

[Corvidae]

Okay, so a few people have expressed interest in trying to figure out how the DM10 settings and soundrom SysEx files are laid out. I've done a fair bit of research and attempted some reverse-engineering on the files, and I think I've actually figured a few things out. Most of my work has been on the soundrom file, so that's what these notes will apply towards. The settings files are... interesting. A stock module's file is tiny, but ANY changes will inflate it to many times the original size. I have no idea what's going on with that.

Anyhow, here we go:

1: Both files are composed of the same "style" of SysEx message; they all start and end the same way:

F0 00 00 0E 2F <data> F7

2: The first byte of the <data> section in Note 1 above seems to indicate the "type" of message. So far I've identified the following:
  0x0A (in file "header")
  0x0B (in file "header")
  0x0C (in file "header")
  0x03 ("data" message, see below)

3: The length and content of messages are very similar between the stock Alesis soundrom and the BlueJay one:

  * Message 1: 0xF0 0x00 0x00 0x0E 0x2F 0x0A 0x40 0x01 0xF7
  * Message 2: 0xF0 0x00 0x00 0x0E 0x2F 0x0C <data> 0xF7 (no ideas on what this contains)
  * Message 3: 0xF0 0x00 0x00 0x0E 0x2F 0x0C 0x60 0x00 0x00 0x00 0x00 0x00 0x03 0x41 0xF7 (seems to match between Alesis and BJ soundsets)
  * Message 4: 0xF0 0x00 0x00 0x0E 0x2F 0x0B 0xF7 (end of header?)
  * Message 5: 0xF0 0x00 0x00 0x0E 0x2F 0x0A 0x68 0x04 0x49 0x3F 0xF7 (Alesis)
                     0xF0 0x00 0x00 0x0E 0x2F 0x0A 0x10 0x1C 0x38 0x1F 0xF7 (Bluejay)
  * Message 6 - <numMessages - 1>: "data message" (see below)
  * Last message: 0xF0 0x00 0x00 0x0E 0x2F 0x0B 0xF7

4: The "data messages" seem to hold the actual data that gets loaded into the module. Since MIDI requires that all "payload" bytes be between 0 - 127, the actual message is encoded to fit. This is pretty simple; check http://ruinwesen.com/blog?id=71 for some technical discussion of this.

5: I wrote a quick program to take all the data messages, decode them, and dump the data to a file. I was going to attach them to this post, but they're WAY too large (43MB for BlueJay and 90MB for stock, both compressed. Uncompressed is about 40-50% larger). There's also the problem with distributing the BJ soundset, even if the file isn't in a usable state. Looking at the resulting file, there are some interesting things (these are from the stock soundset):

  * The very start of the file seems to have a header with what look like the start and the size of different portions of the file. Here are the "section" names and data bytes for 1.03:

    AUX PGMS:     0CA4 / 034C
    COMP SYX:    0FDC / 049C
    EQ SYX  :      779C / 0EAE90
    INST TBL:  0F2618 / 073120
    PATT TBL:    143C / 4D40
    HID PATT:   165710 / 04A45B
    PGMS    :  1AFB54 / 0AAC
    REV SYX :   1B05EC / 07D73B60
    SAMP TBL:       ? / ? (this doesn't seem to match the structure for the other sections, so I don't know what's going on here)
    SONG TBL: 7F24174 / 0020
    VERSION :    6104 / 15F8 
    ICONKITS:    0001 / 0014


  * Most of the text strings start at file offset 0x7EC1. It looks like they're in alphabetical order, and each has some data around it (probably pointing to the location for its samples). I can't seem to find the link between its location in the file and the actual sample data, however, so I think there's something else going on that will be nearly impossible to figure out without knowing how it's laid out in memory on the module itself.

If anyone wants to delve into this and see if we can make some more progress, PM me and I'll send you the program I used to dump the files. Maybe you'll see something I missed...

[FD]

Hi, I want help, I have knowledge in microelectronics and programming.

I had begun to decode the boot files, app, trigger, I had not yet arrived in soundrom files.
I decoded and converted these files into ASM code (using IDA), but have not found anything useful yet

Its strategy is better, decoding the file soundrom.
I have a doubt, the file BlueJay is equal for everyone? or has any lock-based serial?

Please let us communicate by email, I want to help it.
fmgdias@gmail.com

[vaikl]

I have a doubt, the file BlueJay is equal for everyone? or has any lock-based serial?

The BlueJay extension file is locked with the serial number of the buyer's module. That's why we can't use RedBox soundpacks on the DM10 and vice versa.

[Corvidae]

They say that it's serial-based, but I'm not so sure. At the very least, I haven't found anything pointing to my serial in my copy of BJ.

[vaikl]

They say that it's serial-based, but I'm not so sure. At the very least, I haven't found anything pointing to my serial in my copy of BJ.

Hm, I'm not 100% sure but I recall that one of my installation attempts with an more or less unregistered copy of the BlueJay sysex (loaded at the times where all download processes were new and totally unclear) failed because of a wrong or invalid serial. If that was a general error or if only *my* serial was missing - I don't know.

[FD]

Corvidae,

at the end of each statement below SysEx, it is clear that the last bit before the F7 is some kind of checksum.
 "F0 00 00 0E 2F 03" <231 bytes of data> <one bit checksum> F7.
You know how this checksum is calculated?
I tried using XOR, but could not.

[Corvidae]

at the end of each statement below SysEx, it is clear that the last bit before the F7 is some kind of checksum.

I'm not sure if we can say that for sure. Did you find something pointing to that?

As far as the checksums go... maybe it's just the last byte of a simple addition? That and XOR are the only two "easy" checksum methods I can think of off the top of my head.

[FD]

at the end of each statement below SysEx, it is clear that the last bit before the F7 is some kind of checksum.

I'm not sure if we can say that for sure. Did you find something pointing to that?

I think the SysEx specification is as follows:
Total of 239 bytes per command.

"F0 00 00 0E 2F 03": write command
<220 bytes>: data that are written in ROM
<10 bytes>: data address in ROM
<1 byte>: Unknown (0A or 0B)
<1 byte>: Checksum
"F7": end

I came to these conclusions by analyzing the files boot, app, trigger, soundrom.

About Checksum:
https://code.google.com/p/mididuino/source/browse/trunk/hardware/libraries/Elektron/Elektron.cpp
https://code.google.com/p/libjdkmidi/source/browse/trunk/java/src/com/jdkoftinoff/midi/MIDISystemExclusive.java

[Corvidae]

Hmm. That would make sense (and could explain the "extra" bytes I was finding in seemingly-random places). I'll have to see what happens if I skip those bytes when I decode the SysEx messages.

[FD]

Corvidae,

you can already decrypt the file settings kit? For Example DeeDubsDm10_04-03-2010.syx.
I wish I could load it directly into DMEditor.

Code: [Select]

F0 00 00 0E 2F 04 56 31 2E 30 30 62 00 56 31 2E 30 33 00 F7
F0 00 00 0E 2F 05 00 00 00 73 00 00 00 01 08 00 00 3C 18 00 0C 01 67 41 0A 2D 6F 23 00 38 00 35 05 00 00 00 00 00 00 0A 01 01 00 01 01 01 01 00 00 00 01 00 00 01 00 02 01 01 00 0A 01 3C 00 0A 46 01 58 0C 06 00 00 09 10 10 55 2D 0F 07 00 01 00 02 0D 02 42 02 F7
F0 00 00 0E 2F 05 10 01 00 13 00 01 00 01 00 10 00 3C 37 0E 01 00 40 43 0A 00 48 00 00 00 00 50 60 42 00 0F 04 00 00 10 07 00 64 60 35 00 00 08 21 00 00 00 0A 78 00 60 55 07 0F 54 00 0A 02 06 00 03 0B 00 0C 60 00 66 0F 57 00 0D 01 0B 00 0F 75 0F 55 00 05 07 F7
...
F0 00 00 0E 2F 06 00 46 01 F7


[Corvidae]

you can already decrypt the file settings kit? For Example DeeDubsDm10_04-03-2010.syx.
I wish I could load it directly into DMEditor.

So do I... that was actually the primary reason I started looking into this approach in the first place. If you export the settings from a completely factory module, it's tiny. The second you change something, though, it balloons up to several times larger. Changing individual settings after that seems to make the file change in very strange ways that I couldn't figure out.

[pbear5]

This is very interesting and I don't understand a bit (no pun intended) of it!  So what's the goal here?  Are you working towards a way for people who legitimately own the Bluejay to maybe combine some of the stock instruments?  From what I have read of the Bluejay it seems to fall down on Hi-Hat being variable and some missing accent cymbals. If you could take your legitimate Bluejay and compile a new sound set with some pieces from the stock set I think that would be a sweet hack that would greatly increase Bluejay sales and maybe spur development of new sound sets. Unfortunately I only know the user side of MIDI but this is fascinating. Keep it going! 

[Sharkuel]

Same boat here pbear. All this "nerdtalk" about MIDI is making me feel quite dumb but i hope you crack something usefull out of it. Keep it going you guys!!!

[Trondster]

I guess the primary goal is to be able to read sysex files and communication from the module - to be able to back settings up to a sysex file and load those settings into the DMEditor.

The current problem with the DmEditor is that while it is able to write settings to the module, it can't read settings off the modules..

[Sharkuel]

Well... This is going to sound retarded from my behalf, but... xD here it goes.

The Alesis DM10 and the RED Box are almost the same, right?! As anyone in his right mind tried to analyse the updates from Pearl?! I mean, manipulate them in a way that they become "useful" on the Alesis DM10, and vice-versa?!

[Corvidae]

You're both correct; I believe that once we get one of the two SysEx files figured out, we should be able to get the other working (they're pretty similar, and since they're both loaded the same way, they likely use the same encoding / format. This could lead to a few nice things, including DMEditor being able to read full module backups and "mixing" the soundsets. We still have a looooooong way to go, but we may be making some small amount of progress.

Shark: I haven't looked into the REDbox, but as I don't actually have a module to test with, that could prove to be difficult (as I can't compare the file to the module). There's also some talk about an "unlock file" you'd need to load, so I think they may have changed the SysEx loading code in the firmware, making it largely useless for this purpose.

Plus, they have far more soundsets than we do.

[Sharkuel]

It was a longshot, but i had to make that observation. xD It would be neat if it was possible.

[pbear5]

Corvedia, you mentioned that the sysex file size increases in size disproportionately after you make a change to anything. FWIW from a non-programmer type; I wonder if this is because, unaltered, the file might just reference the "factory restore" ROM but, once you change something, everything has to be duplicated. So again, FWIW, the original sysex just says "see ROM"

Do you know if setting the module back to "read only" makes any difference?

In addition to not being a programmer I am also a Mac user so, you know, when you get this all worked out on PC get right to work on the OSX version!    I suppose I could reinstall Fusion but I don't have any other use for Windows.

[bobccock]

when do u guys play drums?

[Corvidae]

Corvedia, you mentioned that the sysex file size increases in size disproportionately after you make a change to anything. FWIW from a non-programmer type; I wonder if this is because, unaltered, the file might just reference the "factory restore" ROM but, once you change something, everything has to be duplicated. So again, FWIW, the original sysex just says "see ROM"

I'm pretty sure that's what's happening, yeah. It looks like it goes from "we're completely defaulted" to "we have a ton of changes everywhere," though, which makes it difficult to see what exactly HAS changed.

Do you know if setting the module back to "read only" makes any difference?

Doesn't seem to... as far as I can tell, that just keeps you from being able to save settings. It doesn't seem to affect the output files at all.

In addition to not being a programmer I am also a Mac user so, you know, when you get this all worked out on PC get right to work on the OSX version!    I suppose I could reinstall Fusion but I don't have any other use for Windows.

Buy me one and I'll work on it.

[pbear5]

http://www.hackintosh.com   

[Corvidae]

Like I said, buy me one and we'll talk.

[rosbia]

sorry for the English, then I possess the blue jay and two modules DM10, one I use with internal sounds and the other with blue jay, blue jay, unfortunately, the one with a client during a night (stupid) has spilled over the liquid, since there I want to buy another of blue jay (most likely I'll have to buy another module) in the following days I tried to do experiments on file. syx of the blue jay, I tried to install it but clearly without the other DM10 gave me an error, then I opened the file with word pad BJ and I accidentally discovered that the last 3 numbers of the serial module appear 2 times (I did search with word pad)
EXAMPLE: the last five digits of the module 00643, opening BJ with word pad I found 643 twice, at this point or change the two-digit found (643) with the last three of the other DM10 (623) after it gave me an error different, practically had recognized the serial but it gave me error for the file slightly larger than the first, I hope you understand what I wrote and can help someone, unfortunately I do not mean computer.

[Corvidae]

Hmm, okay. I'll have to try that with my files and see if I can duplicate that. Do you remember where in the file you found those (my guess is near the very beginning, on the first "page" or so)?

Sorry to hear about your module. Your English is just fine, though, and thanks for the report.

[FD]

after making any change to the file syex, in your case the serial module, you should find the byte checksum, and recalculates it.
But I do not know what position the checksum, just know it's one per line.
And also do not know the algorithm used to recalculate it, but I have strong suspicions that subtraction is something like: CHECKSUM = REST - BYTE, and if, Checksum > 128, so Checksum = 128 - Checksum.
PS: Use this program to edit (HxD - Freeware Hex Editor).