Adaptation Generator

[markusschloesser]

How about a simple form based adaptation .py generator?
A user would need to

1. name the synth / adaptation = generates file names and identity
2. storage (number of banks etc)
3. all the rest, divided into mandatory and optional. Sysex strings should be copy/pastable, tool should handle hex to python /0x00 conversion.
   The tool itself in knobkraft (or website) would spit out a .py file so that users don't need to actually mess around with code. I guess that would lower the barrier for users to actually do Adaptation.

[Andy2No]

Good idea. I expect there are some synths that won't work for, and some .py code would be needed to handle the special cases, but that could probably cover a lot of them.

How about a generic .py adaptation with just some commented constants and string constants at the top for people to change, to make a first draft? That's basically the same idea, but without a form to fill in - just some simple editing to do without needing to know anything about Python.

[christofmuc]

Interesting! I believe it would only work for the simple cases, but even of these are many. You could also make the structure confiom to the fields from the SoundDiver Adaption editor, so if you open an ADA file in SoundDiver, all you need to do is to copy over the form data from SoundDiver. Just an idea.

You wouldn't necesarily need a generator, you could just format the Python file in that there is a section at the top which defines the data fields, and code below uses that data. Later you could then load the data from a file generated by some tool etc.

Originally I though you would just start with a rather similar synth in the create new Adaptation button function, and then just edit the file which has enough comments to get you going. But even finding out which one is similar can be hard task (we had that over there with the idea of the table showing all the features of the adaptations).

I had a long discussion of GS about code-driven Adaptations vs. data-driven Adaptations, and while I think the code-driven approach of KnobKraft is far more future-proof than all the data-driven Librarians, it clearly requires a programmers mind. Recent example pro code-driven method: To implement the renamePatch() function for the Electra One adaptation, you have to turn sysex binary data into a JSON string, parse the JSON, set the attribute, convert new JSON back to binary. A few lines of Python, and importing the json library, and you're done without any change to the KnobKraft. In data-driven Librarians - impossible.

Another way to make it easier is to implement higher-level abstractions in Python, e.g. a Sequential implementation (they are all very much alike) which you import and just provide some data points for the generic Sequential module.

Then a Roland base module, a Yamaha base module etc. that at least takes away code copying between adaptations and would provide more tricky stuff like the escapeSysex/denibble functions.

[christofmuc]

Sorry when I repeated AndyNo's comment, I answered Markus question first before getting into his answer ;-)

[christofmuc]

So this was interesting. I took the SoundDiver Adaptation manual, and went through the Tutorial "3.1 Bank Manager Kawai K1". I distilled the data entered into the diverse dialog boxes of SoundDiver to the following data structure (short of Cartridge banks, the complication with the Edit Buffer and the missing Multi-Bank Dumps, for the sake of brevity):

    adaptation = {
        "Manufacturer Name": "Kawai",
        "Manufacturer MIDI ID": 0x40,
        "Model": "K1",
        "Device ID": (2, 1, 16),  # sysex offset, min value, max value
        "Default Timeout": 200,  # milliseconds to wait during Scan
        "Default Send Pause": 80,  # milliseconds to wait during sending multiple messages
        "Scan with Universal Device Inquiry": False,
        "Scan request": [0xf0, 0x40, 0x00, 0b01100000, 0xf7],
        "Scan reply": [0xf0, 0x40, 0x00, 0b01100001, 0x00, 0x03, 0xf7],
        "Data Types": [{"Name": "Single", "Size": 87, "Name Size": 10, "Name Offset": 0, "Name format": "Ascii"}],
        "Bank Driver": [{"Bank Name": "Int-Singles I/1", "Data Type": "Single", "# of Entries": 32,
                         "Transmission Format": "7bit", "Checksum Type": "Kawai K1/K4",
                         "Memory Bank": True,
                         "Offsets": (0, 0, 0),  # These are the program offsets for single, bank, program change
                         "Single Request": [0xf0, 0x40, 0x00, 0x00, 0x00, 0x03, 0x00, "EN#", 0xf7],
                         "Single Reply": [0xf0, 0x40, 0x00, 0x20, 0x00, 0x03, 0x00, "EN#", "SUM", "SIN", "CHK", 0xf7],
                         "Bank Request": [0xf0, 0x40, 0x00, 0x01, 0x00, 0x03, 0x00, 0x00, 0xf7],
                         "Bank Reply": [0xf0, 0x40, 0x00, 0x21, 0x00, 0x03, 0x00, 0x00, "[", "SUM", "SIN", "CHK", "]",
                                        0xf7]},
                        {"Bank Name": "Int-Singles i/2", "Data Type": "Single", "# of Entries": 32,
                         "Transmission Format": "7bit", "Checksum Type": "Kawai K1/K4",
                         "Memory Bank": True,
                         "Offsets": (32, 32, 32),  # These are the program offsets for single, bank, program change
                         "Single Request": [0xf0, 0x40, 0x00, 0x00, 0x00, 0x03, 0x00, "EN#", 0xf7],
                         "Single Reply": [0xf0, 0x40, 0x00, 0x20, 0x00, 0x03, 0x00, "EN#", "SUM", "SIN", "CHK", 0xf7],
                         "Bank Request": [0xf0, 0x40, 0x00, 0x01, 0x00, 0x03, 0x20, 0x00, 0xf7],
                         "Bank Reply": [0xf0, 0x40, 0x00, 0x21, 0x00, 0x03, 0x20, 0x00, "[", "SUM", "SIN", "CHK", "]",
                                        0xf7]}
                        ],
    }

So yes, this contains everything that is needed to work with the Kawai K1. We could make a "SoundDiver" adaptation type that reads this data and implements the functions required by the Orm, and I think it would work.

There is a bit of black magic going on here however, typical for data-driven approaches:

• The name format "Ascii" is easy to understand, but what happens with synths having a completely different character set like the Access Virus or the Roland D-50, or a non-unique character set like the Matrix 6 where two different numbers could create the same character?
• The "Transmission Format" parameter also is a fixed list of character encodings, like nibbled, 7bit, MSB byte, ... I could see that this list is more limited than the list of character sets, but still it is unlikely that you'll catch all formats.
• The checksum type is quite hilarious, as it is called only "Kawai K1/K4", so that is certainly something that should be code and not data, imho,

I do like the "pseudo bytes" they use like "EN#" or even the repeat-loop with the brackets "[" "]", this is pretty powerful.

You can also see the limitation - while they have abstracted the entry number EN# for a program place, they failed to abstract the bank number but rather want a copy of the whole "bank driver" data structure for each bank, and then you need to figure out again where the differences between two banks are, and you could have a bug that one bank works but the other doesn't.

Still, it could be a worthwhile exercise to implement this, as it could make porting SoundDiver ADA files much easier. You just have to figure out all the various special cases...

[christofmuc]

It's pretty straightforward, I implemented a SoundDiver driver for the code above in about an hour, and it seems to work. A bit more is missing, but you get the idea. The full file is referenced in the commit above.

You can also see in the file how I add tests to the python module to check if my implementation is correct, without even needing to fire up KnobKraft.

[markusschloesser]

I expect there are some synths that won't work for, and some .py code would be needed to handle the special cases, but that could probably cover a lot of them.

MS: that's always the case anyway, isn't it? But a 80/20 approach should work. :-)

[markusschloesser]

I had a long discussion of GS about code-driven Adaptations vs. data-driven Adaptations, and while I think the code-driven approach of KnobKraft is far more future-proof than all the data-driven Librarians, it clearly requires a programmers mind. Recent example pro code-driven method: To implement the renamePatch() function for the Electra One adaptation, you have to turn sysex binary data into a JSON string, parse the JSON, set the attribute, convert new JSON back to binary. A few lines of Python, and importing the json library, and you're done without any change to the KnobKraft. In data-driven Librarians - impossible.

MS: I tested the adaptation and managed to crash knobkraft succcessfully :-) Do you automatically get the crash dmp files?

Another way to make it easier is to implement higher-level abstractions in Python, e.g. a Sequential implementation (they are all very much alike) which you import and just provide some data points for the generic Sequential module.

Then a Roland base module, a Yamaha base module etc. that at least takes away code copying between adaptations and would provide more tricky stuff like the escapeSysex/denibble functions.

MS: That's why I had the idea, I thought manually typing the sysex manufacturer ID 5 times can be done more efficiently (I'm lazy) :-)
MS: A manufacturer based template would ease writing adaptations significantly!

[christofmuc]

MS: I tested the adaptation and managed to crash knobkraft succcessfully :-) Do you automatically get the crash dmp files?

ha ha. Not that I have seen, strange. Can you crash it again, please :-) ?

MS: That's why I had the idea, I thought manually typing the sysex manufacturer ID 5 times can be done more efficiently (I'm lazy) :-)
MS: A manufacturer based template would ease writing adaptations significantly!

I will check. The DSI/Sequential stuff naturallly lends to it, as they use the same software in everything since the Evolver, it's mostly copy and paste from there. For Roland same, but for that I need to support the handshake stuff properly.

How are you getting along with making your own adaptation? Let me know if I can be of help.

[markusschloesser]

It's pretty straightforward, I implemented a SoundDiver driver for the code above in about an hour, and it seems to work. A bit more is missing, but you get the idea. The full file is referenced in the commit above.

You can also see in the file how I add tests to the python module to check if my implementation is correct, without even needing to fire up KnobKraft.

re your K1 sounddiver adaptation: I'm not sure I fully understand.

1. How do I open an existing diver ada in sounddiver to be able to copy stuff out of it and into a knobkraft adaptation?
2. is your sounddiver driver a generic one which, presuming I got 1. done, I can use use to get all existing sounddiver supported synths into knobkraft? And how would I do that? Use you're new k1.py file as a template?

[christofmuc]

1. How do I open an existing diver ada in sounddiver to be able to copy stuff out of it and into a knobkraft adaptation?

Hm, I have to look it up, in the SoundDiver application there is an Adaptation editor window that let's you see all these dialog boxes with the MIDI messages. Let me try to dig it out, last time it didn't want to start on my box.

2. is your sounddiver driver a generic one which, presuming I got 1. done, I can use use to get all existing sounddiver supported synths into knobkraft? And how would I do that? Use you're new k1.py file as a template?

Ah, yes, just not today! I am still working on it, and the goal would be that there is a file sounddiver.py or something which then get's imported into a KawaiK1.py, and the KawaiK1.py basically just contains the data definition and maybe one line of instantiation. That's a bit of more work though, as it is kind of duplicating the dynamic profile implementation - you should only write a function createBankDump if the synth supports BankDumps etc.

[markusschloesser]

MS: I tested the adaptation and managed to crash knobkraft succcessfully :-) Do you automatically get the crash dmp files?

ha ha. Not that I have seen, strange. Can you crash it again, please :-) ?

MS: That's why I had the idea, I thought manually typing the sysex manufacturer ID 5 times can be done more efficiently (I'm lazy) :-)
MS: A manufacturer based template would ease writing adaptations significantly!

I will check. The DSI/Sequential stuff naturallly lends to it, as they use the same software in everything since the Evolver, it's mostly copy and paste from there. For Roland same, but for that I need to support the handshake stuff properly.

How are you getting along with making your own adaptation? Let me know if I can be of help.

re crash: how can I upload the dmp file? GitHub won't let me
re adaptations: have started 5 (!), none work properly 🙄😂.

1. Andromeda: already written about, Currently only gets detected. It is supposed to have an edit buffer, but doesn't really. Can't read the edit buffer for "programs" only for "Mixes". Don't know why. Some things not really clear in the (A6) doc. Will need to continue.
2. K5000s: no edit buffer at all
3. Jomox Alphabase: insufficient sysex documentation, will need to write to Jürgen Michaelis to find out if there's more sysex supported but not documented
4. Yamaha FS1R: when I tried yesterday, I had to wrap my head around the whole performance, parts, voices, fseq thing. Will try later with the new knowledge from today
5. Quasimidi Cyber-6: insufficient sysex support, no device detection, only "Request Data from device" and "Dump Data to device" plus an "Address Map"
   So if I can get 1,2,4 for free by using ada, I'll save a lot of time. Plus, some concepts are hard for me to understand. I am good at inspiring, not so much at "doing" when it comes to code.

btw if you ever find that there's enough "ideas" from my side, please let me know :-)

[markusschloesser]

as for the other synths:

1. Mks80: waiting for your adaptation ☺😇
2. Prophet 12: very happily using Andy's
3. Kijimi: I have supported the kickstarter for babufrik, which is a librarian and editor for the kijimi (and now open source, might be worth checking out, also Juice based iirc)
4. RD-8: waiting for your adaptation, but as there's not presets anyway, doesn't really matter
5. SE ATC-X: Have the official editor for it, which is ok-ish, will be maybe look at it later.

[markusschloesser]

1. How do I open an existing diver ada in sounddiver to be able to copy stuff out of it and into a knobkraft adaptation?

Hm, I have to look it up, in the SoundDiver application there is an Adaptation editor window that let's you see all these dialog boxes with the MIDI messages. Let me try to dig it out, last time it didn't want to start on my box.

2. is your sounddiver driver a generic one which, presuming I got 1. done, I can use use to get all existing sounddiver supported synths into knobkraft? And how would I do that? Use you're new k1.py file as a template?

Ah, yes, just not today! I am still working on it, and the goal would be that there is a file sounddiver.py or something which then get's imported into a KawaiK1.py, and the KawaiK1.py basically just contains the data definition and maybe one line of instantiation. That's a bit of more work though, as it is kind of duplicating the dynamic profile implementation - you should only write a function createBankDump if the synth supports BankDumps etc.

Adaptation editor: there's a menu entry in the "Install" windows, but greyed for me for all the synths (uni and specific)
ada/knobkraft: got it 😊 Thanks!

[christofmuc]

Ok, I found it in SoundDiver - you need go to the Install window, select a synth of type "UNI" (universal module, its listed in column 3 which by default is not shown) and add it to your setup. Then, in the Setup window you can do an "Open Device" double click on the device (first abort all the scan and autodetection messages). When you're in the Device Window, you suddenly get an Adaption menu in the main menu bar. First menu item is Edit Adaptation... which will open the Adaptation Editor.

I am just looking at the Novation Supernova, but it looks insanely complex with 19 different data types, I am not sure this is what I'd like to do...

[markusschloesser]

worked for the Andromeda!

[Andy2No]

as for the other synths:

2. Prophet 12: very happily using Andy's

That's a lot more credit than I deserve - I only helped a bit with bug finding. Christofmuc did the rest.

[christofmuc]

Marked the crash dump discussion as off-topic, as we now have #60.

Back on topic - I will look at the Andromeda ADA and see how much is missing over the K1 I did yesterday, I think the K5000 is epitome of Sysex, you will not find a synth with more different message types. For that, we definitely need the multi-data type function ready. I'd suggest to focus on one device and get that working first before trying others, the FS1R is also a beast, and the Yamaha format is also special, I don't think you can already do it with an adaptation. The Cyber-6 (never heard of that before, looks cool) should be doable, I'll give it a go.

[markusschloesser]

Ok, I found it in SoundDiver - you need go to the Install window, select a synth of type "UNI" (universal module, its listed in column 3 which by default is not shown) and add it to your setup. Then, in the Setup window you can do an "Open Device" double click on the device (first abort all the scan and autodetection messages). When you're in the Device Window, you suddenly get an Adaption menu in the main menu bar. First menu item is Edit Adaptation... which will open the Adaptation Editor.

I am just looking at the Novation Supernova, but it looks insanely complex with 19 different data types, I am not sure this is what I'd like to do...

How did you export the data in the "Edit Adaptation" window? Asking because Sounddiver does have an existing Waldorf Pulse UNI, which I could then put into the KawaiK1 adaptation?

[christofmuc]

@markusschloesser You can't export :-/ This is why I didn't continue that path, the file format is binary obscure, so you basically have to open SoundDiver and best case copy/paste the info. Note that I have not implemented all the different bits and pieces and details, so taking no a new synth probably needs to extend the implementation of the KnobDiver SoundDiver emulation. Not sure if this is really better than doing a clean implementation...

It certainly helps for synths where the documentation might be missing or unclear, becaues the SoundDiver hex strings will work!

[markusschloesser]

@christofmuc ah shit, what a pity! :-) my sysex is already valid, as in identical to the strings in SD, but I thought I could skip the implementation part. Will continue my questions in the correct issue, thanks! BTW U at Superbooth?

[christofmuc]

Can't make Superbooth (again :-/), though I'd love to. But from Munich even with the sprinter train you need to full weekend, and we have other plans. I will try to put it more strict into the calendar for next year!

[markusschloesser]

that's a pity! 😕