URL monitoring

[Rafiot]

It would be nice to add a monitoring feature to Lookyloo, the current implementation idea is as follows:

• Add a interval and an expiration fields on the capture page (optional fields)
  • Add sanity checks in the config: minimal capture interval (1h?) and maximal expiration (24h?), especially for the public instance
  • Optional: expire monitoring early if the URL is returning an error before the expected expiring time
• Each "monitoring capture" needs to have an identifier (uuid) that links to a page with all the related captures, sorted by time of capture, maybe with the status (up/down) => new template
• Submit notification when a new capture is done (there is no notification feature in Lookyloo at all yet, this will require another discussion)

[kimmoal]

My use-case would somewhat fit this.

I want to monitor an URL for a legal takedown request in an external service that I am using for managing the URL in question. Any sort of queue I can subscribe to per URL would be great.

One potential problem with monitoring certain urls: there might be x amount of redirects, could we monitor each redirect separately?

Example:
https://urlshortener.com -> https://badsite.wordup.com -> http://googlesites.googlecontent.com
URL shortener might get taken down but we still have wordup.com and googlecontent.com still up and usable so we might want to monitor those also until the last site gets taken down.

---

A bit of an far out idea: Playwright also supports code generation so there could be a possibility to submit code (of course very problematic for public instances) to be run on the monitored site so we could also check for interaction, POST forms etc. that could be taken down. Not sure about this, need to play around with the codegen to see if it is practical enough.

[Rafiot]

Monitoring each redirect separately

That's a good idea, and that's probably something we can implement as part of the monitoring module. Making it usable and not a complete will need to design the UI properly, but it should be possible.

Playwright code generation and submitting instrumentation for specific captures

Also probably doable, but as you said, probably only on private instances. And let's move this conversation to a new thread, I think it will be better.

[Assiente]

I think that similar monitoring services can be taken as a basis. Or build something on the basis of their functionality.