Enhance Bulk URL Analysis Error Handling

[tsunoyu]

Feature Request: Enhance Bulk URL Analysis Error Handling

Description:

Improve the bulk URL analysis tool's error handling to ensure it continues processing even when individual URLs encounter errors. Provide detailed error reporting, including the specific URL that triggered the error, to allow users to investigate further.

Motivation:

The current behavior of the tool, halting analysis and report generation entirely when an error occurs, hinders productivity. Users need to manually identify the problematic URLs and re-run the analysis, which is time-consuming. This enhancement will improve the tool's robustness and usability.

User Story:

When analyzing 1000+ URLs in bulk I want the tool to gracefully handle errors and continue processing the remaining URLs so that it can complete the full analysis and get a comprehensive report even if some URLs fail.
I also want the tool to explicitly identify the URLs that triggered errors so that I can investigate them further if needed.

Acceptance Criteria:

• The tool should not stop the analysis when an error occurs for a specific URL.
• The final report should be generated even if some URLs failed.
• The report or tool output should clearly indicate which URLs caused errors.
• The error message associated with each failed URL should be included in the report or output.

Additional Information:

• Example error stack trace (provided):

  TypeError: Failed to set the 'text' property on 'HTMLScriptElement': This document requires 'TrustedScript' assignment.
  pptr:evaluateHandle;i.addScriptTag%20(%2Fusr%2Flocal%2F.npm-global%2Flib%2Fnode_modules%2F%40google-psat%2Fcli%2Fdist%2Fmain.js%3A3%3A961138):1:123
  

　　Which above won't tell which URL and it won't generate the report even other URLs has been completed the analysis.

• Related issues or discussions (if applicable)

This enhancement will significantly improve the user experience by making the bulk URL analysis tool more resilient and informative.

[milindmore22]

Hi @tsunoyu,

Thanks for reporting the error. You're absolutely right that errors should be handled gracefully to ensure the process continues for the rest of the URLs in the sitemap.

Unfortunately, we haven't been able to replicate the issue at our end. Please share the specific sitemap or CSV file you used so we can try to reproduce it and find a solution for the same.

[amedina]

@tsunoyu, thanks for your request. This is part of our roadmap and such more robust behavior will be part of PSAT in an upcoming version. In a nutshell, the output of aggregated analyses (i.e. sitemap/csv) should be resilient to URL errors, and a summary of such errors should be part of the generated report. Will keep this issue open until we merge the corresponding changes.

[maitreyie-chavan]

A related issue - #818