Consider JSON RPC for Messaging

[emoon]

Right now a custom protocol is used for talking between backend and views. It would likely be nice to use a standard format for sending data as it would make it easier for external problems to supply data to ProDBG without actually implementing/using some API.

[SlNPacifist]

I think data format and protocol should be chosen wisely. JSON is very limited and is not well-suited for low-level tool like ProDBG. What are requirements for data format? Here is a list I can think of:

• binary data type
• low-level number data types (especially integer types of big sizes, like 64-bit unsigned integers)
• structures

And JSON does match these requirements. It has no type for binary data. Usually binary data is encoded in base64 for json. Numbers could also be a problem. JSON itself defines number data type, but every number is in floating-point format. Therefore, most implementation use internal floating-point variable to store JSON numbers, which could be a problem when dealing with 64-bit addresses.

What are requirements for protocol? I can only think of one requirement: errors as a response (when dealing with RPC one always have to handle errors). However, I guess pretty much every RPC protocol supports error as a response.

Other capabilities that need research:

• API changes. Current implementation supports many fields in a message. When API adds new data to response, it does not affect older modules that do not expect new fields. Change of response format (change of field type, removing/renaming field), however, is still a problem.
• Custom requests/responses. Current implementation allows any additional fields in messages. Is it needed in that "wild" form? Maybe it is better to allow plugins and backend define their own requests and responses, while all intermediate code should not parse message at all?
• Message/event/notification. A request that does not require a response. Used to notify other modules/plugins about some event (like user made next step in program). Not sure if this should be part of protocol since it could be implemented just by discarding response.
• Transport level. Currently ProDBG has remote debugging in plans. Should protocol be aware of this?

Anything else?

[emoon]

Good points:

Some alternatives may be:

http://msgpack.org/index.html
http://bsonspec.org/
https://capnproto.org/

API changes. Current implementation supports many fields in a message. When API adds new data to response, it does not affect older modules that do not expect new fields. Change of response format (change of field type, removing/renaming field), however, is still a problem.

Yeah this is an issue. One way to solve this is to either:

1. Plugins must ask what kind of data they can "request"
2. Versioning. When doing a request a version is added to the request and ProDBG has to "format" the response according to the version even if there are newer versions of the request available.

Custom requests/responses. Current implementation allows any additional fields in messages. Is it needed in that "wild" form? Maybe it is better to allow plugins and backend define their own requests and responses, while all intermediate code should not parse message at all?

I don't think this will work. As there may be several backends and the the views must be able to expect to get back data the same way from them. There may also be several views that request similar data from the backend in order to view the data in a different way. Say for example memory view shows memory as hex, numbers while there may be a bitmap view that shows memory as a bitmap or in a different way.

Message/event/notification. A request that does not require a response. Used to notify other modules/plugins about some event (like user made next step in program). Not sure if this should be part of protocol since it could be implemented just by discarding response.

I think this can be a separate things. I think we should separate events from actual requests as you say events are much simpler plugins can just ignore them if the wish.

Transport level. Currently ProDBG has remote debugging in plans. Should protocol be aware of this?

I don't think the protocol needs to care about it. It should be the same regardless of where the data comes from.

Another thing I thought about: Maybe it would make sense to stuff the data in in a sqlite db or something like that to allow easier way to request the data. I'm a bit hesitant because of perf/complexity issues doing that but that being said I haven't actually investigated it.

[SlNPacifist]

Transport level. Currently ProDBG has remote debugging in plans. Should protocol be aware of this?

I don't think the protocol needs to care about it. It should be the same regardless of where the data comes from.

This is mostly about protocol capabilities. Most protocols that want messages to be sent over network include network-related things like timeouts, disconnects, repeats, protection against repeating mutating requests (see comment at the end). Protocol messages that are not aware of this things are usually wrapped into higher-level messages.

Other thing that could be related is network performance. JSON-rpc allows batch requests. Capnproto introduces time travel that allows dependent requests to be sent in one network request.

Repeating mutating requests: client makes a mutating request (like cloning something on server) but does not get a response. At this moment client does not know if data was cloned or not. If client repeats request, server could end up with dangling first clone. Solved differently. I prefer keeping response result for request id during session. Repeating request will be responded with kept result.

[emoon]

Good points and yes I agree keeping the response request. The way ProDBG works today is that it's batching all requests that happen during one frame, sends them to the backend and backend fills in in the data and it comes back in one chunk of memory.

[SlNPacifist]

Custom requests/responses. Current implementation allows any additional fields in messages. Is it needed in that "wild" form? Maybe it is better to allow plugins and backend define their own requests and responses, while all intermediate code should not parse message at all?

I don't think this will work. As there may be several backends and the the views must be able to expect to get back data the same way from them. There may also be several views that request similar data from the backend in order to view the data in a different way. Say for example memory view shows memory as hex, numbers while there may be a bitmap view that shows memory as a bitmap or in a different way.

I was thinking more about interfaces there. For example, we can define "memory access" interface to get & set memory. Plugin could ask for this interface and if backend implements it, plugin will get an object to operate on it.
Now, if a group of people wants to use ProDBG to debug SNES games, they will need to:

• make custom backend that will interact with emulator
• implement interfaces they need (most likely, execution interface for stepping and memory interface)

If they want, they could

• define their own snes-specific interface
• write a plugin that will use this interface

[emoon]

Yeah I see what you mean. I have something similar for that

https://github.com/emoon/ProDBG/blob/master/src/addons/amiga_uae_view_plugin/src/lib.rs#L53
https://github.com/emoon/ProDBG/blob/master/src/addons/amiga_uae_plugin/src/lib.rs#L292

This registers a custom id that the Amiga UAE backend uses to send DMA timing data from backend to a Amiga UAE view to display the data.

Even if this would look a bit different in the RPC scenario I guess it's something similar you are thinking about?

[emoon]

So users should be allowed to implement their own specific thing for their use-case which is fine but in some cases it's just about reading some memory and displaying it in a different fashion. Take for example memory mapped hardware registers. On Amiga these will start at 0xdff000 but a view would likely just use the "regular" memory request to request 0xdff000 - 0xdffxxx but display them in a way that makes sense (0xdff180 is background color for example, that may be shown with a color or something)

So requesting memory should be a defined RPC call with clear rules on what responses that can be received/sent.

In some cases data must be a bit more flexible. Take the case of disassembly. ProDBG uses Capstone for disassembly and a backend can request to use it. Now Capstone supports a limited number of CPUs so if your backend has a CPU that isn't supported you can't use it.

Also the data sent back from the the disassembly can vary in richness. For example on per instruction there can be a list of registers being used. This can be used to high light registers in the disassembly view like this:

While in some cases the backend doesn't provide back that info thus the highlighting can't be done (as the disassembly view really has no clue about what the instructions does it needs the backend to provide enough information for it to do so)