Ingest Error: 504 API Timeout

[RoloEdits]

Twice in the last 3 months has ingest stopped due to an error on timeout and a failure to reconnect. The fix has been to just keep deployments going at smaller intervals, but for some of our longer uptime services, the timeouts stopping all ingest has been confusing.

This is the error:

Error: could not ingest "stdin" into dataset "NAME": API error 504: Gateway Timeout

The deployments are done inside of a Docker container with this initiating the CLI:

ENTRYPOINT ./APP | axiom ingest DATASET

I'm not sure what would be causing the API timeout multiple times like this, but perhaps there could be a --continue-on-error flag that would keep trying to reconnect through all errors and just keep logging them to the console? The error was only logged once so its unknown how many times it took to give up trying. At least with a continual log we could see the number of attempts.

[lukasmalkmus]

Thanks for the report.

There can be various reasons for a 504. Axiom Go, which powers Axiom CLI has a sensible retry-mechanism but apparently not every 504 can be resolved adequately. I'll be checking out the parameters we're using over there but nonetheless, --continue-on-error might be a nice addition for ingest and stream.

[RoloEdits]

Unsure if the same issue as the error message is different but I am seeing this:

Error: could not ingest "stdin" into dataset "DATASET": Post "https://api.axiom.co/v1/datasets/DATASET/ingest": read tcp IP:PORT -> IP:443: read: connection reset by peer

It took 8 days this time before it was unable to send to ingest anymore.

[lukasmalkmus]

@RoloEdits Is CLI invoked exactly as you described above?

ENTRYPOINT ./APP | axiom ingest DATASET

No extra command line arguments or any specific environment variables set?

[RoloEdits]

CLI invocation has no flags, and AXIOM_TOKEN env is set.

[RoloEdits]

Got a new message, this time a 503 Error.

Error: could not ingest "stdin" into dataset "DATASET": API error 503: Service Unavailable

[lukasmalkmus]

@RoloEdits Ups, forgot to ping you about it but can you try the latest release with --continue-on-error applied?

[RoloEdits]

We have moved over to using https://vector.dev/ and sending to the axiom sink as it has had no issues ingesting. I will have to look into getting some services back to the axiom cli, but its pretty busy currently. Not sure how soon I can get that going.