You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As reported by @danielcrabtree under #2777 , Crawlee expects CriticalError to bring the entire process down and doesn't clean up all parts of the crawler correctly.
Example:
importlogfrom'@apify/log';import{CheerioCrawler,CriticalError}from'crawlee';log.setLevel(log.LEVELS.DEBUG);// to see the culpritconstcrawler=newCheerioCrawler({requestHandler: async()=>{thrownewCriticalError('Critical error! The crawler won\'t recover from this!');},statusMessageLoggingInterval: 1,// to see the culprit faster});try{awaitcrawler.run(['https://crawlee.dev']);}catch(e){// We catch the critical error here, so the process doesn't crash.// Right after this catch clause, the process should exit with code 0, // But it will hang indefinitely instead because of the periodic logging `setInterval` hogging the event loop.}
Proposed solution:
Revise the BasicCrawler.run method and ensure all timeouts / intervals / ... are cleared correctly in all the possible execution branches.
The text was updated successfully, but these errors were encountered:
As reported by @danielcrabtree under #2777 , Crawlee expects
CriticalError
to bring the entire process down and doesn't clean up all parts of the crawler correctly.Example:
Proposed solution:
Revise the
BasicCrawler.run
method and ensure all timeouts / intervals / ... are cleared correctly in all the possible execution branches.The text was updated successfully, but these errors were encountered: