Source idea: HTTP API collector for Fastly Signal Science

[NeilJed]

I've been playing with the idea of using Vector to try and pull log data from Fastly Signal Science solution using their API. https://docs.fastly.com/en/ngwaf/extract-your-data

It seems that the HTTP Client source would be a good base for this but their API throws up some interesting issues to solve.

Firstly, the responses are paginated so it would need to be aware of that and look for the next page URL given in the response and automatically fetch that, eventually getting all the pages for the full data set. So I guess the the source would be making multiple HTTP requests to the API as needed.

Also the from/until timestamp - as these require UTC timestamp values they'd need to be dynamic and based on the current time that the source collection start. I was thinking about having it expressed as an negative offset from "now" in minutes. i.e from = now-10m, until = now-5m then run the source with a 5m interval. I was also thinking if now should be aligned to the nearest 5 minute to account for any delay from the desired start time and the actual time.

So yeah, I don't think this is possible out-of-the box and requires modifying the default HTTP Client source but I'd love some feed back on possible approaches or gotchas.

[jszwedko]

The sort of challenges that API has (pagination and keeping track of where to start each request) seem to fairly common across other APIs that users want to consume from Vector with. I could see enhancing the http_client source to be able to handle this. Given the scope of the change I think we'd likely want to see an RFC for it first.

As a workaround, you could write a standalone program that reads the data and passes it to Vector over one of its existing sources (like http_server).