Patches don't necessarily remember their location correctly

[christofmuc]

Reported on the sequential forum:

If I import a single bank and then save it as a SysEx file, it defaults to bank 1, rather than preserving the bank number from which it was taken.

I need to check if this is actually a valid use case - not if the download was done via edit buffer dumps!?

[christofmuc]

It seems there is another bug in the GenericAdaptation that prevents the numberFromDump() function being used correctly, see #133 for more details.

[christofmuc]

This is a complex issue coming from the fact that not all synths store bank and program in the patch data itself. If they don't, then the import position is somewhat random and hard to guess.

Possible cases:

1. The synth has a program dump capability, and stores both bank and program number. All DSI synths have this nice capability, and even if the OB-6 displays patch 266 as 266, it really is bank 2 and patch 66. The Access Virus is another synth with full position info in the patch.
2. The synth has a program dump capability, but does not store the bank number in the patch. The Matrix 1000 is an example where the program number is known (0-99), but the bank is lost if not deducted from somewhere else.
3. The DW8000 for example doesn't store either, and the "number" of the patch can only be generated when reading it. That is easy you read exactly 64 patches, but what about loading a file with only 10, or 100? Then the program numbers generated during reading don't make any sense.
4. Some stupid synths like the Yamaha Reface DX don't store positions, and don't even have a mechnism to store the patch in the ram - you can only send to edit buffer, and the user has to press save...

In addition to that, we get confusions from 0-based and 1-based banks, from program names displayed containing the bank (like Matrix 1000) even when the patch does not contain the bank, and the complexity that a bank switch is not necessarily standard MIDI for all synths.

Add to that the case that you can import the same patch multiple times and it will get multiple "original" patch positions.

There are two different ways to encode bank and program:

1. Use two variables, bank and program number.
2. Use only one program number, and if in a higher bank just keep counting program. You can later calculate the bank from the whole program number, if the bank size is known.

Knobkraft so far used version 2 - and all adaptations are based on this as well. So while working nicely, I run into another problem - the bank might or might not be known during importing the patch. And the software might need to work with patches that have an unknown bank (like Matrix 1000) and differentiate them from a known bank. This gets more serious with the #9 bank mangement feature, where we want to know where in the synth a patch is stored and if it can be addressed.

[markusschloesser]

And to add further confusion, have you seen my tempest dumps?
The same patch ("sound" in tempest nomenclature) can have positional info or not, depending on HOW it was dumped.
So how would you run a duplicate check if the same patch is either 154 or 174 big without knowing the full sysex data structure

[Andy2No]

154 * 8 / 7 = 176

This makes me think one is in 8 bit form, and one is in the 7 bit form where each group of 7 bytes is preceded by another one, with all the top bits in it - which we've been calling escaped / unescaped, but I get confused about which term means which.

The two byte discrepancy could be to do with a checksum, perhaps.

E.g. 152 bytes * 8/7 is 173.7 ish, so you need 174 bytes to put it in. That would be a true patch length of 154 including a two byte checksum, that doesn't make it into the 7 bit encoded form.

Or some similar variation on that theme :)

[christofmuc]

And to add further confusion, have you seen my tempest dumps? The same patch ("sound" in tempest nomenclature) can have positional info or not, depending on HOW it was dumped. So how would you run a duplicate check if the same patch is either 154 or 174 big without knowing the full sysex data structure

Sorry I have not yet looked at it, currently diving deep into KK 2.0.0 for the bank mangement. But having two different patch types is pretty normal, e.g. all DSI synths have either edit buffer dumps or program dumps, although the size difference is only 2 bytes (bank and program number)

[christofmuc]

154 * 8 / 7 = 176

This makes me think one is in 8 bit form, and one is in the 7 bit form where each group of 7 bytes is preceded by another one, with all the top bits in it - which we've been calling escaped / unescaped, but I get confused about which term means which.

Regarding the naming - "escaping" means we need to encode 8 bit into the 7 bits that are allowed to be transferred via MIDI sysex bytes. Unescaping puts the high bit back in place to have the raw data where a byte as 8 bytes.

The two byte discrepancy could be to do with a checksum, perhaps.

These formats usually have a fixed header that is not subject to the escaping mechanism, which is why it is hard to tell where the escaping starts. I figured that out for the Tempest in order to retrieve the names already, even if I do not understand how the 8th bits are encoded - not in the same way as in all other DSI synths, seems the Tempest has the software written by somebody else or at least not based on the same chips and libraries...

[markusschloesser]

I figured that out for the Tempest in order to retrieve the names already, even if I do not understand how the 8th bits are encoded - not in the same way as in all other DSI synths, seems the Tempest has the software written by somebody else or at least not based on the same chips and libraries...

It is written completely differently, thats because Roger Linn was involved in the making. It was a joint product.

[christofmuc]

This technical question was solved for the KK 2.0.0 release (and it had to be solved). The solution was to implement actually both ways to represent the Bank/Program Number calculation.

The C++ code now has MidiBankNumber. with the bank requiring to know its size so it can calculate the running program number, and the MidiProgramNumber can be generated with or without bank - and that means we can represent both a fully specified program number, with as a pair (bank number, program number) or as a running number (bank number * bank size + program number within bank). The Python code still uses only the running program numbers, which e.g. for a Matrix 1000 go from 0 to 999 or for a Rev2 from 0 to 1023, knowing each bank is 100 resp 128 in size.

The running number also can be used to deduce the patch type, should not all banks contain the same type of data. This feeds into #98 and #58.

Closing this ticket, as it was a technical marker for myself.