patch names blank in Evolver desktop dump

[ergodicbreak]

Hello, first thanks very much for KnobKraft, it looks nice and works smoothly! I cloned and built on Manjaro Linux.

I have an issue I think related to the alpha status of the Evolver adaptation, where all the patch names are blank (and editing a patch doesn't add the name -- it stays blank).

From the Python adaptation file TODO list it seems this is an issue with the Evolver specifically. I am willing to work on the adaptation to improve it, but since this is my first time with KnobKraft (and I am complete beginner with synths and MIDI) I would appreciate any advice/direction on how to proceed.

Screenshot below,

[christofmuc]

Welcome! Glad you like it so far! Let me actually start to think here - the code indeed mentions that the name is not part of the regular sysex message, but stored in a separate message called "Program Name Data Dump"

That's on page 43 of the manual:

https://www.sequential.com/downloads/evolver_keyboard/doc/Evo_Key_Manual_1.3.pdf

It also says that the Evolver will send the message in addition to the program data dump, so that's good news. Interestingly it doesn't send it according to the documentation when we request an edit buffer dump - it seems edit buffers have no name, only the programs stored at program locations.

Renames would need to be done not by modifying the patch but rather by sending an updated program name data dump message.

Now, the problem is that the current Python interface assumes that each patch is a single message, but we would have two messages - the regular program data dump plus the name dump. That is not unusual, just that the KnobKraft Orm currently would need C++ code to support that (ready to help if you're willing to jump into that!), and it cannot be done in an adaptation. The Yamaha reface DX implementation is an example for such a system.

The good news is that there is already code I wrote early this year trying to bring this capability also into the Python adaptations, it just has not been finished yet and things got quite complex.

I think the Evolver is a great test for continuing that work soon!

[ergodicbreak]

Thanks for the response! I'll see if I can add the data plus name handling in core. I'll take a look at the DX example.

Could you clarify one thing you said,

It also says that the Evolver will send the message in addition to the program data dump,

Maybe you are referring to this sentence,

Evolver will respond by sending out the Program Data in the format described in the
Program Name Data Dump on page 48.

That is in my Evolver Desktop (note the difference) manual.

The Evolver Keyboard manual however says,

The Evolver will respond by sending out the Program Data in the format described in the
Program Data Dump on page 43. It also sends a Program Name Dump message (also on
page 43) after the Program dump. This makes it a complete Program dump while
maintaining compatibility with other Evolver models.

So I wonder if the Evolver Desktop does the same thing, seems like it would have to?

Honestly I'm a little confused by the Evolver implementation. It doesn't seem like the Program Name is included in the program parameters, so if you change it, how do you send it to the device?

This message from the Sequential forums is somewhat cryptic but seems to say the name just isn't supported:

https://forum.sequential.com/index.php?topic=2875.0

The program format is exactly the same between the various Evolver models. However, program names are separate sysex messages not supported by Evolver desktop. So if dumping Evolver desktop programs until one of the other Evolver models the sound will change but the program name will stay the same.

[ergodicbreak]

I posted a message on Sequential,

https://forum.sequential.com/index.php/topic,5757.0.html

I have a feeling the program name on the Desktop is only there for compatibility and you can't change it, which means there'll have to be some solution to associate names in KnobKraft with some kind of patch fingerprint.

[composerjk]

I was guessing that the forum post was related to this issue. I started to glance at that last night, but noticed that the Python adaptation code likely assumes a single message for a patch, as @christofmuc verifies above. I was going to hack in sending the matching Request Program Name Dump SysEx right after the Request Program Data Dump message and then noticed it only stored a single message. So, it was going to be more than a very quick hack.

Unfortunately, the Evolver Desktop does not send the Program Name Data Dump when program data is requested. One needs to send both requests separately.

You can totally change the name of a bank/program (since version 2.0) and noted in the 3.2a manual: https://www.sequential.com/downloads/evolver/doc/Evolver_manual_3.2a.pdf

Here's the relevant part of my reply on that thread:

For example, using sendmidi, you'd request the name for bank 0 program 1 via (EVOLVER is whatever name the MIDI device connected to your Evolver is called):

sendmidi dev EVOLVER syx hex 01 20 01 10 00 01
which should return something like (using receivemidi here):

system-exclusive hex 01 20 01 11 00 01 4E 69 63 65 20 50 61 74 74 65 72 6E 20 20 20 20 dec
That 4E to that final 20 (16 bytes) is the name; in this example, Nice Pattern .

Change those 16 bytes and send it back to the Evolver Desktop to rename that bank/program in the format of the Program Name Data Dump. Note that the syx option to sendmidi sends the f0 start and f7 end bytes.

[ergodicbreak]

Thank you @composerjk, your examples are working on my Evolver (and thanks for turning me on to those handy utilities).

I guess the question now is whether to modify core to get Evolver Desktop working with KnobKraft, or direct effort toward extending the capabilities of the Python adaptation?

If the adaptation will be a lot of work I suppose a temporary shim in core could be worthwhile? What do you think @christofmuc?

[christofmuc]

Haha, that looks like quite a mess in Evolver firmwares!

So I understand:

• To keep backwards compability of the newer Evolvers with the original device (which is ?), they decided not to add the program name to the existing sysex message, but to add a second message with just the name.
• Some Evolvers (not the Desktop) send the second message automatically when a program dump is requested. For the Desktop, you need to send the explicit Name Request in addition to the Program Dump request
• I am still not clear whether the Edit Buffer Dump also has a name?

KnobKraft functions as follows:

• The database stores a byte vector for each patch. This could be the original MidiMessage of the patch, or multiple messages concatenated, or some processed form. It additionally stores a name in a separate data field to allow naming of patches even for synths that do not have name storage
• The database will not allow duplicates to be imported, which is based on fingerprinting the byte vector. Normally, you want to blank out the name before comparison, because otherwise patches that are identical except the name will create a new database entry. This functionality is enabled for Python adaptations with the calculateFingerprint method, which you can reimplement.
• The timing of some devices is fidgety. As you are allowed to return multiple MIDI messages as a response to requestProgramDump() from Python, you could just append the Request Name Dump message - ideally, the Evolver will buffer and send out both replies. Even if the name dump is not processed right now, you should see it incoming in the MIDI log of the Orm.

Also, sending a patch to the synth can create multiple messages (patchToProgramDumpSysex returns a vector of MIDI messages), so we are already good to send out multiple messages to the synth.

This leaves us with the one problem that the current simple parsing logic only wants one message per patch.

Today, I have looked at the code from January this year, and it might not be too far off to bring it into production. That would solve this problem (and many many more for other synths). Have a look at https://github.com/christofmuc/KnobKraft-orm/blob/banksFromDataTypes/adaptions/GenericProgramDumpCapability.h at let me know if that does make any sense.

A more complex synth that uses 7 messages per patch (typical for Yamaha) is the refaceDX, for which I made a prototype implementation here in Python (so far, it had to be C++): https://github.com/christofmuc/KnobKraft-orm/blob/banksFromDataTypes/adaptions/YamahaRefaceDX.py

My feeling is that work should be completed before we decide to do any other shortcut for the Evolver - I currently can't think of something simpler, but that is maybe because I already was way into this general stuff ;-)
*

[composerjk]

• To keep backwards compability of the newer Evolvers with the original device (which is ?), they decided not to add the program name to the existing sysex message, but to add a second message with just the name.

The first Evolver was the Evolver Desktop (2002). Next was the Poly Evolver rackmount (2003 dev, 2004 release). Yes, that seems about right. In the other Evolver products, some parameters in the Main/Global Parameter Dump are skipped in future Evolver models to try to remain a close match. I don't know why they didn't have the Evolver Desktop return the Program Name Data Dump when it receives the Request Program Dump message, unless they ran out of space. I should ask Dave, sometime.

• Some Evolvers (not the Desktop) send the second message automatically when a program dump is requested. For the Desktop, you need to send the explicit Name Request in addition to the Program Dump request

Only the Evolver Desktop requires the explicit Name Request in addition to the Program Dump request.

• I am still not clear whether the Edit Buffer Dump also has a name?

The Edit Buffer Dump does not include a name. With most DSI/Sequential gear, the Edit Buffer and Program Data Dumps are the same, only the SysEx header is missing the bank/program numbers (well, and the different type number). The actual program data is the same. Most of their synths do include the name, as I recall. Since the Name was not part of the program data originally with the Evolver Desktop, it's stored in a separate table.

There might be a workaround. The Program Number and Bank Number are the first two bytes of the unpacked Main Parameters Data Dump. So, to get the name for the Edit Buffer:

1. Send the Request Edit Buffer Dump message.
2. Send the Request Main Parameters Dump message.
3. Grab the Program and Bank numbers from the Main Parameters Data Dump.
4. Send the Request Program Name Dump for that Bank and Program.

I haven't tested that, but I believe it should work.

KnobKraft functions as follows:

• The timing of some devices is fidgety. As you are allowed to return multiple MIDI messages as a response to requestProgramDump() from Python, you could just append the Request Name Dump message - ideally, the Evolver will buffer and send out both replies. Even if the name dump is not processed right now, you should see it incoming in the MIDI log of the Orm.

A quick test in which I sent the Request Program Dump followed by the Request Program Name Dump messages in a single request showed that both messages were returned, as expected. I need to rework my hack to use the Python adaptations as homebrew seems to have overwritten the links to get KnobKraft to find Python on macOS, so I'll assume KnobKraft saw both response messages.

[ergodicbreak]

My feeling is that work should be completed before we decide to do any other shortcut for the Evolver - I currently can't think of something simpler, but that is maybe because I already was way into this general stuff

I agree this makes the most sense.

This leaves us with the one problem that the current simple parsing logic only wants one message per patch.

Are you saying the Python adaptation of the Yamaha DX is using new functionality? I'm wondering based on this,

KnobKraft-orm/adaptions/YamahaRefaceDX.py

Line 47 in ccbd1dd

def nameFromDump(message):

Where the code comment is # Actually, those are multiple messages; does this mean we could send two sysex messages to the Evolver Desktop like @composerjk did in his last comment and parse the returning two messages like this nameFromDump does?

I've only read through the programming guide once so I might be confusing something basic though.

[christofmuc]

This leaves us with the one problem that the current simple parsing logic only wants one message per patch.

Are you saying the Python adaptation of the Yamaha DX is using new functionality? I'm wondering based on this,

KnobKraft-orm/adaptions/YamahaRefaceDX.py

Line 47 in ccbd1dd

def nameFromDump(message):

Where the code comment is # Actually, those are multiple messages; does this mean we could send two sysex messages to the Evolver Desktop like @composerjk did in his last comment and parse the returning two messages like this nameFromDump does?

The current version (master branch) is able to send a multi-message program dump request (just concat two MIDI messages to be one byte list in Python), so we could already trigger the Evolver's program dump. But the current version does not parse multiple messages, and only passes the first message to the Python code. The reason is that it wouldn't know when a patch is done and the next patch is started, if there are patches that consists of 2 messages (like the Evolver) or 7 (like the reface DX).

Maybe we could just specify a "number of messages per patch" and then treat them as one array input into the isProgramDump() / nameFromDump() messages? That could be simpler than the generic stream load capability I have in the branch mentioned above, which is really an m:n mapping (e..g the Roland MKS80 sends 4 messages with 64 patches in them, plus a handfull of handshake messages in between).

The Yamaha file above is on the branch and not in the master yet, as the released Orm would not use that Python code.

[ergodicbreak]

Maybe we could just specify a "number of messages per patch" and then treat them as one array input into the isProgramDump() / nameFromDump() messages? That could be simpler than the generic stream load capability I have in the branch mentioned above

Yes a simple declarative solution sounds good, unless there are indeed some synths with not well defined responses?

[christofmuc]

@ergodicbreak @composerjk I thought about our problem tonight, and made an experimental 1.14.0 version here in the branch "multiMessageEditBuffers" at sha 29417de. I can upload a Windows installer as well if desired.

The idea: make the isEditBufferDump() method take more than one MIDI message. In Python, this is a list of numbers anyway and the difference is only if is a single complete MIDI message or multiple concatenated.

The trick: You can additionally to the normal methods implement a new function isPartOfEditBufferDump() that takes only a single MIDI message (again a list of numbers). When this function is implemented, it is called prior to isEditBufferDump().

isEditBufferDump() should return True only if it got all MIDI messages required to have a complete Edit Buffer. I

The same has been implemented for isProgramDump() - there is a new function isPartOfSingleProgramDump() waiting to be implemented.

So the mechnism could be:

isPartOfSingleDump() returns True for both the Name Dump and the Program Dump of the Evolver.
isSingleDump() only returns True when it can parse the list of numbers and finds both the program dump (first) and after that the name dump.
createProgramDumpRequest() is also allowed to return a list of numbers with multiple MIDI messages concatenated, and will return both the program dump request and the name dump request messages.

Then nameFromDump() and renamePatch() can be implemented properly by modifying the stored second message. Remember the two messages will be stored as one list of numbers.

This is a bit brittle parsing in case there are extra messages or MIDI echos flying around, but for a simple case this could already work and create value. Let me know what you think!

[ergodicbreak]

Sounds promising! However I can't get this new branch to build. At first there was a build error missing CMakeLists.txt in third_party/json and third_party/json-schema-validator. I'm not familiar with the build system really so I just commented out the adding of those dirs (they're not added in master); the build proceeded but error'd out around here:

https://gist.github.com/ergodicbreak/67193f4c1663ade84ad16b0e3c6b15c9

[christofmuc]

@ergodicbreak Oh. Terribly sorry, the two submodules were missing from the branch. If you checkout now you should get content into the two directories, they are definitely needed :-)

[ergodicbreak]

edit:

I got the build working after some user errors, I'll leave the original comments below in case it helps some wayward soul. Sorry I obviously don't know how to work with submodules and branches! :)

These were the working steps:

# git clone --recurse-submodules https://github.com/christofmuc/KnobKraft-orm
# cd KnobKraft-orm
# git checkout multiMessageEditBuffers
# git submodule update --recursive
# LDFLAGS="--disable-lto" cmake -S . -B builds && cmake --build builds

original comments:

Second attempt in a clean directory:

# git clone --recursive https://github.com/christofmuc/KnobKraft-orm
# cd KnobKraft-org
# git checkout multiMessageEditBuffers
# git submodule update --recursive --remote
# LDFLAGS="--disable-lto" cmake -S . -B builds && cmake --build builds

Error was:

##vso[task.setvariable variable=ormVersionNumber]1.14.0
-- Configuring done
CMake Error at The-Orm/CMakeLists.txt:104 (add_executable):
Target "KnobKraftOrm" links to target "fmt::fmt" but the target was not
found. Perhaps a find_package() call is missing for an IMPORTED target, or
an ALIAS target is missing?

CMake Error at juce-widgets/CMakeLists.txt:90 (add_library):
Target "juce-widgets" links to target "fmt::fmt" but the target was not
found. Perhaps a find_package() call is missing for an IMPORTED target, or
an ALIAS target is missing?

-- Generating done
CMake Generate step failed. Build files cannot be regenerated correctly.

First attempt in a clean directory:

# git clone --recurse-submodules https://github.com/christofmuc/KnobKraft-orm
# cd KnobKraft-orm
# git checkout multiMessageEditBuffers
# LDFLAGS="--disable-lto" cmake -S . -B builds && cmake --build builds

The error was

-- Exporting juceaide
CMake Error at CMakeLists.txt:194 (add_subdirectory):
The source directory

/home/george/work/tools/KnobKraft-orm/third_party/json

does not contain a CMakeLists.txt file.

CMake Error at CMakeLists.txt:195 (add_subdirectory):
The source directory

/home/george/work/tools/KnobKraft-orm/third_party/json-schema-validator

does not contain a CMakeLists.txt file.

##vso[task.setvariable variable=ormVersionNumber]1.14.0
-- Configuring incomplete, errors occurred!

[christofmuc]

@ergodicbreak Thanks for the update! My mistake to not add the submodules in the first place, but yes, pulling the latest version and then running the proper git vodoo (probably git submodule update --init --recursive) should do the job. Let me know how you progress with the Evolver!