Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve ability to notice systemic issues by reporting error (and success) statistics for various platforms #1214

Open
jamezpolley opened this issue May 6, 2019 · 0 comments

Comments

@jamezpolley
Copy link
Member

Inspired by #1213 and #1204 and #1201 and #1196

The email alerts we send out at present do an okay job of letting a scraper owner know that their individual scraper is failing (and #1213 has suggestions for making that even better)
'
But morph as a platform has had some systemic issues that broke most SSL connections (#1196, #1201) or all PHP scrapers (#1204). At present we don't have any overall statistics that let us find these problems.

If we could log, at the end of each scraper run, details such as its language and language version, as well as the first line of the error message (if the scraper errored), I think this would help us to be able to spot things like "90% of PHP scrapers are failing" or "80% of scraper failures seem to be SSL errors".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant