feat(scraper): add random user agent

[vvatelot]

No description provided.

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
bases/ecoindex/cli
__init__.py	0	0	100%
app.py	99	52	47%	125–128, 130, 145–146, 181–182, 184, 192, 201, 203–204, 206–208, 224, 226–228, 230–231, 234, 236–239, 241, 243–244, 249, 253–254, 256, 258, 261, 263–265, 267–268, 271–273, 281, 285, 317, 319, 327, 331, 335
arguments_handler.py	61	10	83%	38–41, 43–44, 50–52, 54
console_output.py	10	7	30%	6, 8–12, 14
crawl.py	15	6	60%	20–24, 27
helper.py	10	6	40%	15, 22–25, 27
report.py	63	46	26%	29, 31–32, 34–35, 37–41, 44, 49, 54, 59–60, 63–66, 74–77, 80, 86, 92–94, 97, 99, 103, 105–107, 109, 120–121, 124–125, 130, 136, 180, 183–184, 186–187
components/ecoindex/compute
__init__.py	2	0	100%
ecoindex.py	28	1	96%	48
components/ecoindex/data
__init__.py	12	0	100%
colors.py	7	0	100%
grades.py	7	0	100%
medians.py	3	0	100%
quantiles.py	3	0	100%
targets.py	3	0	100%
components/ecoindex/exceptions
__init__.py	0	0	100%
scraper.py	8	3	62%	7–9
components/ecoindex/models
__init__.py	6	0	100%
compute.py	65	5	92%	111, 113, 116, 118, 134
enums.py	27	1	96%	9
response_examples.py	5	0	100%
scraper.py	10	0	100%
sort.py	5	0	100%
components/ecoindex/scraper
__init__.py	2	0	100%
scrap.py	71	37	47%	46, 49–50, 52, 61–65, 71–74, 80–83, 86–87, 89–90, 92, 94, 101–104, 111–112, 114–115, 124–125, 127, 130–131, 133
user_agent.py	7	4	42%	6–7, 9, 13
components/ecoindex/utils
__init__.py	2	0	100%
files.py	45	19	57%	23–25, 29, 34, 36–37, 39–41, 46–47, 60–61, 63–64, 80–82
screenshots.py	12	7	41%	8–10, 12, 16, 22–23
test/bases/ecoindex/cli
__init__.py	0	0	100%
test_app.py	52	0	100%
test_arguments_handler.py	34	0	100%
test/components/ecoindex/compute
__init__.py	0	0	100%
test_ecoindex.py	38	0	100%
test_models.py	75	9	88%	122–130
test/components/ecoindex/scraper
__init__.py	0	0	100%
test_scraper.py	32	0	100%
TOTAL	819	213	73%

[vvatelot]

@CodiumAI-Agent /review

[CodiumAI-Agent]

PR Analysis

(review updated until commit 445355b)

• 🎯 Main theme: Adding random user agent functionality to the scraper.
• 📝 PR summary: This PR introduces a random user agent for each scraping request. This is achieved by adding a new dependency random-user-agent and using it to generate a user agent string for each request. The PR also modifies the test URL and increases the number of test runs.
• 📌 Type of PR: Enhancement
• 🧪 Relevant tests added: No
• ⏱️ Estimated effort to review [1-5]: 2, because the PR is relatively small and straightforward, involving the addition of a new dependency and its usage in the existing code.
• 🔒 Security concerns: No security concerns found

PR Feedback

💡 General suggestions: The PR is generally well done. However, it would be beneficial to add tests to ensure the random user agent is working as expected. Additionally, the print statement used for debugging should be replaced with proper logging.

---

✨ Usage guide:

---

Overview:
The review tool scans the PR code changes, and generates a PR review. The tool can be triggered automatically every time a new PR is opened, or can be invoked manually by commenting on any PR.
When commenting, to edit configurations related to the review tool (pr_reviewer section), use the following template:

/review --pr_reviewer.some_config1=... --pr_reviewer.some_config2=...

With a configuration file, use the following template:

[pr_reviewer]
some_config1=...
some_config2=...

Utilizing extra instructions The review tool can be configured with extra instructions, which can be used to guide the model to a feedback tailored to the needs of your project. Be specific, clear, and concise in the instructions. With extra instructions, you are the prompter. Specify the relevant sub-tool, and the relevant aspects of the PR that you want to emphasize. Examples for extra instructions: [pr_reviewer] # /review # extra_instructions=""" In the code feedback section, emphasize the following: - Does the code logic cover relevant edge cases? - Is the code logic clear and easy to understand? - Is the code logic efficient? ... """ Use triple quotes to write multi-line instructions. Use bullet points to make the instructions more readable.

How to enable\disable automation When you first install PR-Agent app, the default mode for the review tool is: pr_commands = ["/review", ...] meaning the review tool will run automatically on every PR, with the default configuration. Edit this field to enable/disable the tool, or to change the used configurations

About the 'Code feedback' section The review tool provides several type of feedbacks, one of them is code suggestions. If you are interested only in the code suggestions, it is recommended to use the improve feature instead, since it dedicated only to code suggestions, and usually gives better results. Use the review tool if you want to get a more comprehensive feedback, which includes code suggestions as well.

Auto-labels The review tool can auto-generate two specific types of labels for a PR: a possible security issue label, that detects possible security issues (enable_review_labels_security flag) a Review effort [1-5]: x label, where x is the estimated effort to review the PR (enable_review_labels_effort flag)

Extra sub-tools The review tool provides a collection of possible feedbacks about a PR. It is recommended to review the possible options, and choose the ones relevant for your use case. Some of the feature that are disabled by default are quite useful, and should be considered for enabling. For example: require_score_review, require_soc2_review, enable_review_labels_effort, and more.

More PR-Agent commands To invoke the PR-Agent, add a comment using one of the following commands: /review: Request a review of your Pull Request. /describe: Update the PR title and description based on the contents of the PR. /improve [--extended]: Suggest code improvements. Extended mode provides a higher quality feedback. /ask <QUESTION>: Ask a question about the PR. /update_changelog: Update the changelog based on the PR's contents. /add_docs 💎: Generate docstring for new components introduced in the PR. /generate_labels 💎: Generate labels for the PR based on the PR's contents. /analyze 💎: Automatically analyzes the PR, and presents changes walkthrough for each component. See the tools guide for more details. To list the possible configuration parameters, add a /config comment.

See the review usage page for a comprehensive guide on using this tool.

[CodiumAI-Agent]

Persistent review updated to latest commit 445355b

[CodiumAI-Agent]

Replace the print statement with a logging statement for better debugging and production readiness. [important]

[CodiumAI-Agent]

Consider adding error handling in case the user agent generation fails. [medium]