Proxy Pool Manager

[Aduomas]

I was thinking of implementing a proxy pool manager to store proxies and log their status such as current speed, used rate, etc. This would allow you to rotate proxies, reload browsers to change the proxies, etc.

@browser decorator takes proxy as a parameter but would be much more usable if it took a specific implementation of a class or a list of proxies instead.

take a look at a POC ProxyManager:

class ProxyManager:
    def __init__(self, proxies, rotate_every=100):
        self.proxies = proxies
        self.rotate_every = rotate_every
        self.request_count = 0
        self.current_proxy = None

    def get_proxy(self):
        if self.current_proxy is None or self.request_count >= self.rotate_every:
            self.current_proxy = self.select_new_proxy()
            self.request_count = 0 
        self.request_count += 1
        return self.current_proxy

    def select_new_proxy(self):
        return random.choice(self.proxies)

and I would try to implement the scraping task accordingly:

from botasaurus import *

proxy_manager = ProxyManager(proxies, rotate_every=100)

@browser(proxy=lambda: proxy_manager.get_proxy())
def scrape_heading_task(driver: AntiDetectDriver, url):
    try:
        driver.get(url)
        heading = driver.text("h1")
        return {"heading": heading, "url": url}
    except Exception as e:
        print(f"Error scraping {url}: {e}")
        proxy_manager.report_error(proxy_manager.current_proxy)

if __name__ == "__main__":
    urls = ["https://www.example.com/page1", "https://www.example.com/page2", ...]
    for url in urls:
        scrape_heading_task(url=url)

but I am not sure if this is the best solution. Has anyone experimented with this and has some suggestions? Exactly for this case there were not any examples.