Have an example synth adaptation and other documentation suggestions for adaptations

[markusschloesser]

I seem to struggle a lot writing adaptations and thought about what would make it easier for me. Here are my ideas:

1. I think having an example adaptation which includes all possible functionality with docstrings and or comments in code, would be good. With that, I wouldn't have to switch between the guideline and the actual code. There are a couple of other adaptations I use, but never sure which one to choose.
2. Clearly divide the code into sections with explanation of what needs to be written for what feature and also what relies on what.
   a) For example the device detection code is not needed for somebody who will just manually select to correct ports.
   b) Do I need an edit buffer in the module? Documentation suggests that I can also use a Program Dump, but iirc other parts say one needs an edit buffer, which is counter intuitive when a synth doesn't have an edit buffer. So for example the Waldorf Pulse does not have sysex for "edit buffer request" but can only do "single program requests" or "multiple program requests", but if you send it a "single program" back, it will go to the edit buffer of the synth, whereas if you send "multiple program" back, it will directly overwrite everything.
   c) Do I need extractPatchesFromBank if I receive multiple patches in response to a "multiple program requests" or "bank dump request"?
   d) If synth only has numbers, I don't need any naming features, correct?
   e) maybe something like a decision tree?
3. I think focusing the adaptation guidelines more around the capabilities might make it easier for users. Also a table of content with direct links to the relevant sections and other formatting would be nice.
4. Some hints about using an IDE and which one, might be helpful.
5. under https://github.com/christofmuc/KnobKraft-orm/blob/master/adaptions/Adaptation%20Programming%20Guide.md#list-of-functions-to-implement it says "you need to implement the following list functions not marked optional. The optional functions can be implemented for additional functionality." But I don't see what's optional and what's not (anymore).
6. Ah, now I see "I list all functions in the order they appear in the example file of the Korg DW6000 adaptation". That should be on top and could be the example adaptation I am talking about in 1. But this adaptation does not contain a lot of features and no documentation. The Korg adaptation do contain a lot more information. Also Novation Ultranova,
7. Maybe explain why and for what there are wildly different adaptation formats (e.g. KawaiK1)
8. Best practices

What's your thought on this?
I can try to start with some things, but as I still lack knowledge, I probably cannot tackle everything (see questions above :-) )

[christofmuc]

@markusschloesser I understand you, and I think we circle back to your suggestion at #59.

One problem the adaptation author has is terminology - there is no clear consistent naming of things across the synth vendors, so the first thing that needs to be determined is how to map the synths capabilities to the Orm's capabilities.

Then yes, I though I had made a flow chart at some point to decide which functions to implement and which not, but I can't seem to recall it right now. My original advice would be something like 'first determine the type of synth and find an adaptation of a similar type'. Like, it is a mid-80s device with simple one message per patch, then go and look and DW8000 or Matrix 1000. If it is a simple bank dump, go look at Kawai K3. But then, already the DX-7 is totally funky with having to decode bitfields from bank dumps to arrive at individual patches...

We should add to the list the point about testing, as this has evolved over the time a lot and by now I have a fairly standard working model.

What is missing is definitely the sequence diagrams on how the internal Librarian component works and which functions are called in which order. I need to revisit the C++ code myself to remember how these functions work together at times, so it would be good to document.

And yes, PyCharm is the IDE I would recommend, there is a Community Version.

What we could do is to identify maybe the three or four major types of synths, and identify a recommeneded example synth for each. Plus, mentioning that just as in the SoundDiver days, some vendors are so consistent with their implementation that we have a generic Sequential and a generic Roland adaptation that should make implementing these much easier.

This could also go together with the other suggestions to simplify the Python interface, and to move to classes for the adaptations instead of the old module mechanics, because these make importing easy but all other tasks hard, e.g. testing and subclassing.

On the other hand, I don't want to start working on KnobKraft 3.0 just yet before we have even stabilized 2.0 with the bank management features.

[markusschloesser]

@markusschloesser I understand you, and I think we circle back to your suggestion at #59.

I had completely forgotten about that one! :-)

One problem the adaptation author has is terminology - there is no clear consistent naming of things across the synth vendors, so the first thing that needs to be determined is how to map the synths capabilities to the Orm's capabilities.

Understood and agreed. Maybe something like a Glossary?

Then yes, I though I had made a flow chart at some point to decide which functions to implement and which not, but I can't seem to recall it right now. My original advice would be something like 'first determine the type of synth and find an adaptation of a similar type'. Like, it is a mid-80s device with simple one message per patch, then go and look and DW8000 or Matrix 1000. If it is a simple bank dump, go look at Kawai K3. But then, already the DX-7 is totally funky with having to decode bitfields from bank dumps to arrive at individual patches...

I think that would be really helpful! Maybe you can find it again?
And exactly your advice could be a good hint for an improved adaptation doc.

We should add to the list the point about testing, as this has evolved over the time a lot and by now I have a fairly standard working model.

I am not up to speed on the testing part (yet)

What is missing is definitely the sequence diagrams on how the internal Librarian component works and which functions are called in which order. I need to revisit the C++ code myself to remember how these functions work together at times, so it would be good to document.

Fully agreed! For me that would be most beneficial!

And yes, PyCharm is the IDE I would recommend, there is a Community Version.

That's what I'm using. We could also put hints in the doc about certain plugins (e.g. Tabnine) and Python versions

What we could do is to identify maybe the three or four major types of synths, and identify a recommeneded example synth for each. Plus, mentioning that just as in the SoundDiver days, some vendors are so consistent with their implementation that we have a generic Sequential and a generic Roland adaptation that should make implementing these much easier.

This could also go together with the other suggestions to simplify the Python interface, and to move to classes for the adaptations instead of the old module mechanics, because these make importing easy but all other tasks hard, e.g. testing and subclassing.

On the other hand, I don't want to start working on KnobKraft 3.0 just yet before we have even stabilized 2.0 with the bank management features.

😁👍

[milnak]

1. I think having an example adaptation which includes all possible functionality with docstrings and or comments in code, would be good. With that, I wouldn't have to switch between the guideline and the actual code. There are a couple of other adaptations I use, but never sure which one to choose.

I've tried to do that with my latest adaptation for Yamaha YC -- take a look and see if this helps.

[markusschloesser]

@christofmuc I'll continue in here:

1. Adaptation Programming Guide def bankDescriptors(self) -> List[Dict] should be def bankDescriptors() -> List[Dict] (2 places)
2. "Program Dump Capability
   To enable the Program Dump Capability for your adaptation, which will be used instead of the Edit Buffer Capability in enumerating the patches in the synth for download, you need to implement the following three functions:
   def createProgramDumpRequest(channel, patchNo)
   def isSingleProgramDump(message)
   def convertToProgramDump(channel, message, program_number)
   The third function currently is not used, but is the basis for the upcoming feature of bank mangement - to send a patch into a specific position in the synths memory, not just the edit buffer or a hard-coded "pseudo edit buffer" like slot 100."
   Saying you need the 3, but the third is not used, doesn't make any sense :-), also you do now have bank management capabilities?
3. lots of adaptations use isOwnSysex, but this is not explained in the guide (especially the "why")
4. or do you want me to correct types etc directly in the md file?

[christofmuc]

Thank you! Do you know how to open a pull request for these changes? Then it is easy for me to review and accept/amend.

[markusschloesser]

Thank you! Do you know how to open a pull request for these changes? Then it is easy for me to review and accept/amend.

I do, but still need an answer to 3 (and 2 b and c from first post), also do I need convertToProgramDump or not?

[christofmuc]

You do not need edit buffer if you have either program dump or bank dump.

Since KnobKraft 2.0 we have bank management capabilities, and convertToProgramDump is required and used.

isOwnSysex is a pattern and not a requirement. I was thinking of making it an official API function for #284, to "sniff" the synth when trying to load any sysex file. So as of today, isOwnSysex is not required, it is just a helper function I like to call the same in every adaptation I do.

extractPatchesFromBank you only need if you want to load banks. If you only load single programs, there is no need to implement it.

[milnak]

Have a look at my adaptations. I've tried to add lots of comments in each which might assist here.

[markusschloesser]

Here's another question: currently I only have program dumps working.
With that that I can request banks A, D, E, F (file menu /midi / get patches from synths)

When I now implement bank dump requests as well, which function will kk use when doing get patches from synths?
Because from a user perspective use case 1 IS already a bank request.

Might makes sense to distinguish program and bank requests?

1. Get patches from synth (with a patch range selection, so that you do not have to request the whole bank (some synths are sloooooow))
2. Get banks from synths
   And then grey out 1 or 2 depending on the current adaptation

[christofmuc]

This is actually complex.

When getting a full bank is too slow, you could define smaller virtual banks.

The logic chooses the "best"way to do it, which is

	if (auto bankMethod = midikraft::Capability::hasCapability<BankDownloadMethodIndicationCapability>(synth)) {
			auto preferredMethod = bankMethod->bankDownloadMethod();
			if (preferredMethod != BankDownloadMethod::UNKNOWN) {
				return preferredMethod;
			}
		}

		// Default behavior - use the "best possible"
		if (midikraft::Capability::hasCapability<StreamLoadCapability>(synth)) {
			return BankDownloadMethod::STREAMING;
		}
		else if (midikraft::Capability::hasCapability<HandshakeLoadingCapability>(synth)) {
			return BankDownloadMethod::HANDSHAKES;
		}
		else if (midikraft::Capability::hasCapability<BankDumpRequestCapability>(synth)) {
			return BankDownloadMethod::BANKS;
		}
		else if (midikraft::Capability::hasCapability<ProgramDumpCabability>(synth)) {
			return BankDownloadMethod::PROGRAM_BUFFERS;
		}
		else if (midikraft::Capability::hasCapability<EditBufferCapability>(synth)) {
			return BankDownloadMethod::EDIT_BUFFERS;
		}
		else {
			return BankDownloadMethod::UNKNOWN;
		}

StreamLoad and HandshakeLoad are only available to C++ code.

You can see that the first thing it checks if the synth has a preferred way to circumvent this logic - and you can do that by implementing the python function

def bankDownloadMethodOverride()
  return either "EDITBUFFERS" or "PROGRAMS"

to prevent the bank dump mechanism to be the default. This is the place where we could add the bank number to retrieve different banks with different methods. (for the ROM banks)