feat(http): Support for Retry-After header

[zharinov]

Changes

• Wrap http requests into retry logic which works only for 429 responses with Retry-After header
• Provide optional configuration with maxRetryAfter option via host rules

Context

• Closes Support retry-after headers (e.g. for 429) #25499

Documentation (please check one with an [x])

• I have updated the documentation, or
• No documentation update is required

How I've tested my work (please select one)

I have verified these changes via:

• Code inspection only, or
• Newly added/modified unit tests, or
• No unit tests but ran on a real repository, or
• Both unit tests + ran on a real repository

[viceice]

I think we need to check got retry, it's doing 2 retries by default which shouldn't happen when we get a retry after header.

[zharinov]

Live test

• lib/test.ts:

import httpServer from 'http';
import { Http } from './util/http';

let countdown = 2;

const server = httpServer
  .createServer(function (req, res) {
    if (countdown === 0) {
      // eslint-disable-next-line no-console
      console.log('Response 200');

      res.writeHead(200, { 'Content-Type': 'application/json' });
      res.end(JSON.stringify({ msg: 'Hello World' }));

      server.closeAllConnections();
      server.close();
    } else {
      // eslint-disable-next-line no-console
      console.log('Response 429');

      countdown -= 1;
      res.writeHead(429, {
        'Content-Type': 'application/json',
        'retry-after': '5',
      });
      res.end(JSON.stringify({ msg: 'Too Many Requests' }));
    }
  })
  .listen(1337, '127.0.0.1');

const client = new Http('foobar');
client
  .get('http://127.0.0.1:1337/')
  .then((res) => {
    // eslint-disable-next-line no-console
    console.log(res);
    return 0;
  })
  .catch((err) => {
    // eslint-disable-next-line no-console
    console.error(err);
  });

❯ time pnpm ts-node lib/test.ts
Response 429
Response 429
Response 200
{
  statusCode: 200,
  body: '{"msg":"Hello World"}',
  headers: {
    'content-type': 'application/json',
    date: 'Sun, 19 Nov 2023 20:28:53 GMT',
    connection: 'keep-alive',
    'keep-alive': 'timeout=5',
    'transfer-encoding': 'chunked'
  },
  authorization: false
}
pnpm ts-node lib/test.ts  0.84s user 0.13s system 9% cpu 10.762 total

[rarkins]

1. Should also work for GitHub secondary rate limit, which I think is a 401
2. Make sure logging is adequate whenever this happens

[rarkins]

Any way to test this in practice?

[zharinov]

Any way to test this in practice?

• The simplest way I've figured out: feat(http): Support for Retry-After header #25859 (comment)

[rarkins]

Does this work if there are concurrent requests to the same host, e.g. a datasource rate limit and we have multiple requests to the same datasource?

[viceice]

otherwise LGTM

[zharinov]

Does this work if there are concurrent requests to the same host, e.g. a datasource rate limit and we have multiple requests to the same datasource?

When the first response is received, any further request will be made after the delay. None of the queueing is done for now, i.e. they will fire simultaneously after the delay.

[viceice]

otherwise LGTM

[viceice]

@rarkins can we get this done please?

[rarkins]

@viceice please check the outstanding conversation to be resolved

[viceice]

we now retry for any 4xx error that's fine by me

[renovate-release]

🎉 This PR is included in version 37.93.0 🎉

The release is available on:

• GitHub release
• 37.93.0

Your semantic-release bot 📦🚀