Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Separate headers by CR LF #471

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

dallagi
Copy link

@dallagi dallagi commented Aug 28, 2023

While using HTTPretty together with the last version of aiohttp, aiohttp failed to parse the response generated by HTTPretty due to the way headers are separated.
Specifically, HTTPretty separates them via the LF (\n) control code while aiohttp expects CR LF (\r\n).

This behavior (on aiohttp's side) is also mentioned here aio-libs/aiohttp#7494 (comment) .

By searching a bit on the Internet (eg. here) it seems that http parsers are expected to be lenient wrt parsing newlines, so I'm not sure this is technically a fault on HTTPretty's side.

However I think it could be better to stay on the safe side and just use \r\n as separator for headers, which is the change I implemented with this PR.

@Dreamsorcerer
Copy link

By searching a bit on the Internet (eg. here) it seems that http parsers are expected to be lenient wrt parsing newlines, so I'm not sure this is technically a fault on HTTPretty's side.

It's literally not an HTTP message, which is clearly a mistake for a library with HTTP in its name. ;)

HTTP-message = start-line CRLF
*( field-line CRLF )
CRLF
[ message-body ]
https://www.rfc-editor.org/rfc/rfc9112.html#section-2.1-1

Allowing LF is completely optional, and can lead to security issues, particularly on the server side:

Although the line terminator for the start-line and fields is the sequence CRLF, a recipient MAY recognize a single LF as a line terminator and ignore any preceding CR.
https://www.rfc-editor.org/rfc/rfc9112.html#section-2.2-3

@@ -1130,7 +1130,7 @@ def fill_filekind(self, fk):
)

for item in string_list:
fk.write(utf8(item) + b'\n')
fk.write(utf8(item) + b'\r\n')

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, UTF-8 is not valid in headers:

Field values are usually constrained to the range of US-ASCII characters [USASCII]. Fields needing a greater range of characters can use an encoding, such as the one defined in [RFC8187].
https://www.rfc-editor.org/rfc/rfc9110.html#section-5.5-4

A recipient MUST parse an HTTP message as a sequence of octets in an encoding that is a superset of US-ASCII [USASCII]. Parsing an HTTP message as a stream of Unicode characters, without regard for the specific encoding, creates security vulnerabilities due to the varying ways that string processing libraries handle invalid multibyte character sequences that contain the octet LF (%x0A).
https://www.rfc-editor.org/rfc/rfc9112.html#section-2.2-2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants