Error decoding response from proxy

[SlyRock]

Bug description
When requesting some URI through the proxy from an activity pods application, there is a discrepency between original encoding format and proxy's format and the client can't decode the response.

Steps to reproduce

1. Call this URI : https://mastodon.social/users/nixCraft through the proxy

CURL example with a local pod provider :
curl --location 'http://localhost:3000/sylvain/proxy'
--header 'sec-ch-ua-platform: "Windows"'
--header 'authorization: Bearer
--header 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36'
--header 'sec-ch-ua: "Google Chrome";v="131", "Chromium";v="131", "Not_A Brand";v="24"'
--header 'Content-Type: multipart/form-data; boundary=----WebKitFormBoundary1HjF3ICd4MwIyf5m'
--header 'sec-ch-ua-mobile: ?0'
--header 'Accept: /'
--header 'Sec-Fetch-Site: same-site'
--header 'Sec-Fetch-Mode: cors'
--header 'Sec-Fetch-Dest: empty'
--header 'host: localhost'
--form 'headers="{"accept":"application/ld+json"}"'
--form 'id="https://mastodon.social/users/nixCraft"'
--form 'method="GET"'

Expected behaviour
Get a response equivalent to the one from the mastodon instance

Screenshots
From chrome :

From postman console :

[SlyRock]

After a longer digging phase and debugging of the situation I found that removing the content-encoding header from the original response solved my case. The value from mastodon.social was gzip.
It makes sense because the original response is decoded by the proxy and re-encoded before being sent to the application, so removing the original encoding header seems legit.

semapps/src/middleware/packages/crypto/signature/proxy.js :

async query(ctx) {
  [...]

    // Remove headers that we don't want to be transfered
    response.headers.delete('content-length');
    response.headers.delete('connection');
    
    // ADDED THIS LINE
    response.headers.delete('content-encoding');

    return {
      ok: true,
      body: responseBody,
      headers: Object.fromEntries(response.headers.entries()),
      status: response.status,
      statusText: response.statusText
    };
  [...]
}

Did I miss something ?

[Laurin-W]

Ahh, well spotted!
I'm not in that part of the code base so @srosset81 might have a better understadning.
But might it make sense to go with a header allow-list instead of removing specific headers (so no unnecessary information is leaked (e.g. something auth-related)?

[SlyRock]

@Laurin-W IMO the goal of the proxy is to return a response as close to the actual server's response as possible.
We don't know in advance what we may be using the proxy to access, on what kind of servers, providing any kind of HTTP response.
If I'm right, we need to remove only headers that are susceptible to mislead the client as to the content of the response (length, encoding...).

That's of course, only what I understand of the situation, I may really miss something here !