QUESTION: Message with payload but no meta-data. . .

[jroper]

Migrated from eclipse/microprofile-reactive-streams-operators#40.

Comment by @olegz:

One of the value of the Message is to create a structure that can encapsulate data and the supplemental Information about such data. Some call it headers, some properties, some meta-data etc.. . . basically some key/value structure (Map).
Looking at the current Message I only see a payload. Is there a reason for omitting it?

public interface Message<T> {

  /**
   * The payload for this message.
   */
  T getPayload();
  
  /**
   * Supplemental Information about this message
   */
  Map<String, Object> getHeaders();
}

Comment by @jroper:
The only reason it's not there is because in the initial PR, we wanted the API to be very small so that individual features can be discussed over individual pull requests, not because we don't think it should be there.

One thing that I'm not sure about, we have the event API being proposed in #14 based on https://cloudevents.io, is it enough to simply have the metadata on Event (which is a subclass of Message), or does it need to be on Message? One nice thing about putting it on Event is that we can be more opinionated about the format, we can for example use Map<String, CloudEventObject> where CloudEventObject has three subclasses according to the cloud events spec for object - string, byte array and map. This is much nicer than just Object, because with just Object you could get anything. That's not to say that instances of Message couldn't provide metadata, different implementations can supply their own meta data maps using stronger typing if it makes sense, for example, implementations that only support string values can expose Map<String, String>, which would be nicer to use than Map<String, Object>.

Comment by @olegz:

@jroper Thanks for following up.
As the current javadoc states A message envelope and as such envelope usually contains meta-information as well as the payload and that is the true value of Message as a simple structure to communicate information in a portable way. Simply having this structure as a wrapper over a single T doesn't seem to add much value, Why not just sent T? In any event, I do believe we are on the same page with regard to intentions.
With regard to the type of value of these properties, sure it could be another H, although in my experience these property values can be of different types in a single message.

Slightly unrelated. . . I see that current state of the Message has few factory operations (i.e., Message.of(..)). Is it necessary or could they be moved to some utility class?

Personally I was always envisioning a simple structure that anyone can agree on. So having Message strategy which only has payload and headers and could also be easily described via simple json (making it essentially portable in the polyglot world) would be a far better approach.
I do understand that these factory operations would not affect any of what I am describing above, but it makes the class itself a bit "too busy" (for the lack of a better term).

Comment by @jroper:
The reason the message class currently exists is for correlation so that the right message can be acknowledged. If I am receiving messages, applying some transformation, and sending to a new destination, I need to ensure that the messages received are not acknowledged (committed, etc) until the transformed message has reached the destination, otherwise messages could get dropped. Now, if it's a simple 1:1 transformation, that's straight forward, the messages can be correlated easily. But what if the transformation is not 1:1? What if I filter the messages? This is what the Message envelope does currently, it allows the ack function to be carried from input messages to output messages so that the right message gets acknowledged at the right time.

Comment by @olegz:
Fair enough. But then would you consider separating the Envelope and the Message? Just doesn't seem right that the Message caries this responsibility. Think of a mail-man/post-office anolongy where they will attempt to deliver and they will take care of ack or nack. The Message itself just represents the data.

Comment by @jroper:
Are you suggesting the current Message gets renamed to Envelope?

Comment by @olegz:
All I am saying is that IMHO there is a data (Message) and the courier (???) of such data. The courier is responsible for all delivery controls (ack/nack), transfers (to possibly other curriers) etc, while the Message remains immutable once created.

Comment by @jroper:
I think that's worth considering. My question is, while they may be different concepts, what practical value will be gained by separating them? Are there use cases where the separation in the API would enable things that wouldn't be possible if they weren't separated? Can it lead to a more fluent API? etc...

By the way, what are your thoughts on handling acks, particularly when transforming an incoming stream to an outgoing stream? What does Spring Cloud Stream do here, in particular, when Reactor is used, and say a filter is done, so there is no 1:1 correlation of input to output events? Does it use a similar wrapper (envelope) to carry the correlation? Or something else?

Comment by @olegz:
I think portability and communication is the main case. A simple Tuple2-like structure for a Message existed for decades in many messaging frameworks/products with some slight variations. Standardizing around T, Map<String, Object> would ensure that we all have something in common. It also has implications outside of Java since the above structure can be easily represented in JSON thus supporting polyglot distributed messaging applications.

So, yes, Envelope or any other name that would encapsulate Message would suffice. And then you can support Envelope.ack/nack and may other operators. It would also follow the logic that "something" (not the Message itself) is responsible for ops-type/lifecycle operation pertaining to a Message.

With regard to Spring Cloud Stream, we are currently working with Project Reactor with Flux<Message<?>> being the center point.
The ops-type/lifecycle operation are currently tied to doOnComplete/Cancel(..), retryBackoff(..) etc., providing relevant information (i.e., delivery tag, offset etc) is contained in message headers.
I would certainly be open to revisiting that with regard to Envelope, as long as Message itself remains simply a data object.

FWIW, this is the Message I am speaking about.

Comment by @cescoffier:
After having played a little bit with the spec (and my implementation), I believe the Message should also have headers as a Map<String, Object> (might be also serializable as in the code shared by @olegz).

Adding headers also add a bit of complexity in term of propagation as the header might need to be propagated. This is particularly true for methods consuming the payload directly such as:

@Incoming(...)
@Outgoing(...)
public Item process(SomeData data) {
   // some clever code creating an Item out of some data
}

In this code, the headers associated with the Message conveying the SomeData may need to be propagated to the Message conveying the resulting Item.

Comment by @olegz:
Header propagation is indeed something we had to struggle with in the past, but got it down pretty good within the last decade, so we can certainly share some pros and cons. Perhaps a separate thread would be nice for it once all agreed on the Message structure with headers.

[Emily-Jiang]

dup of #10