Require server to handle Content-Length headers when receiveing HTTP requests

[provinzkraut]

Add a clause in the spec that:

• Requires servers to never read past an announced Content-Length
• Optionally respond with a 400 - Bad Request if the actual length exceeds the announced length

Motivation

Compatibility with the WSGI spec

As ASGI is a self-described "superset of the WSGI format", it should honour WSGIs stipulation around handling Content-Length. PEP 3333 says:

The server is not required to read past the client’s specified Content-Length, and should simulate an end-of-file condition if the application attempts to read past that point. The application should not attempt to read more data than is specified by the CONTENT_LENGTH variable.

The proposed addition would make ASGI compliant with the WSGI specification in this regard, and adding some additional guarantees by changing the server behaviour from should to must.

Confidence for downstream consumers

Currently, ASGI applications can not assume that the Content-Length header reflects the actual content length returned by successive calls to receive(). This means they can only use the Content-Length header as a hint if present, but would need to check its consistency themselves if they want to rely on it.

Compare this PR where such a feature is discussed: https://github.com/encode/starlette/pull/2328/files#r1397168621

[nggit]

Why apps don't just use more_body is False to know the Content-Length cutoff?

Optionally respond with a 400 - Bad Request if the actual length exceeds the announced length

Note that If a client sends more data than the Content-Length specified for a request, that excess data might belong to the next HTTP request in the pipeline.

[provinzkraut]

Why apps don't just use more_body is False to know the Content-Length cutoff?

Can you explain how that relates to the scenarios I'm describing? If a client say it's sending a request with a Content-Length: 100, but the actual length is 200, how does more_body=False help there? It's specifically a scenario where no_body=False does not work as intended, as the app would expected a length of 100, but currently the server happily reads, and sends to the app, more than that.

Note that If a client sends more data than the Content-Length specified for a request, that excess data might belong to the next HTTP request in the pipeline.

That sounds like something the server should deal with, as the application does not have enough context knowledge.

The behaviour described above is fairly standard webserver behaviour, in that they usually do not read beyond the announced content-length.

[nggit]

If a client say it's sending a request with a Content-Length: 100, but the actual length is 200, how does more_body=False help there?

If the app calls receive(), it will get 100 bytes of data with 'more_body': False. The server will not give more data to the app even if it calls receive() again. It should stay in the server buffer.

It's specifically a scenario where no_body=False does not work as intended, as the app would expected a length of 100, but currently the server happily reads, and sends to the app, more than that.

What server implementation does that? That should be considered a bug.

[provinzkraut]

If the app calls receive(), it will get 100 bytes of data with 'more_body': False. The server will not give more data to the app even if it calls receive() again. It should stay in the server buffer.

It's specifically a scenario where no_body=False does not work as intended, as the app would expected a length of 100, but currently the server happily reads, and sends to the app, more than that.

What server implementation does that? That should be considered a bug.

Let me rephrase: A server could read beyond the declared content length and serve it to the app, and would not be in violation of the spec, since the behaviour you're describing, while commonly understood to be standard server behaviour, is not codified.

As a result, an application cannot rely on this behaviour, or at least should not, as I'd argue all behaviour not defined in the spec should be assumed implementation specific :)

[nggit]

The server may respond with an 'HTTP 400 - Bad Request' error code in case the
request body exceeds the announced content length

This can break Quart, because it calls receive() until the scope ends.
Although it's not clearly stated in the spec, a common practice is that further receive() will be followed by an http.disconnect event; receive() will make the app wait until the scope ends or the connection is lost. This is already beautiful.

Then I think it just needs to be emphasized that message.get('more_body', False) is reliable.