Proxy Forge is a free and open-source tool that helps you to set up OSI level 4 proxies with rotating IPs on the Digital Ocean platform. It is a powerful and innovative solution that addresses one of the most pressing challenges faced by developers and businesses in today's digital landscape: IP throttling and blacklisting. Proxy Forge empowers you to conquer these obstacles by providing a seamless, reliable, and easy-to-deploy solution.
In simpler terms, Proxy Forge helps you to create a pool of rotating IP addresses that you can use to access the internet. This can be useful for a variety of purposes, such as avoiding IP bans, scraping websites, and testing web applications.
Proxy Forge is a powerful tool that can be used by both developers and businesses. It is easy to use and deploy, and it provides a number of features that make it a valuable tool for anyone who needs to access the internet anonymously or avoid IP bans.
- Rotating IPs: Automatically rotates IP addresses from a set of deployed droplets to avoid detection and blacklisting.
- Terraform Scripts: Provides Terraform scripts for straightforward setup and management.
- Scalability: Easily scale your proxy fleet up or down to handle your specific needs.
- Customizable Configuration: Fine-tune proxy settings to match your requirements. (check below)
- IP Throttling Avoidance: Effectively bypass IP throttling mechanisms used by websites and services.
- Detailed Logging: Comprehensive logging for monitoring and troubleshooting.
- Open Source: Proxy Forge is open-source, allowing for community contributions and customization.
I had to find a solution for one of my clients where certain eCommerce websites were supposed to be scraped but only allowed IP addresses from the UK region. I purchased a VPC from a cloud provider in the UK, but I later discovered that the target website implements IP throttling, which temporarily disables it. As a result, I built this solution manually (no Terraform scripts) and without load balancers.
I've been learning Terraform since then, and automated its setup for you. Now it takes less than 5 minutes to complete.
There are many reasons why you might choose to use DigitalOcean over other cloud providers. Some of the key benefits include:
- Simplicity: DigitalOcean is known for its simple and user-friendly interface. It is easy to get started with DigitalOcean, even if you are new to cloud computing.
- Affordability: DigitalOcean offers competitive pricing for its cloud services. Additionally, it provides a substantial 2 TB of free bandwidth each month.
- Performance: DigitalOcean uses high-performance hardware for its Droplets, so you can be sure that your applications will run smoothly.
- Reliability: DigitalOcean has a strong track record of reliability and uptime.
- Community: DigitalOcean has a large and active community of users and developers. This means that there is a wealth of resources available to help you get the most out of your DigitalOcean account.
Before using Proxy Forge, ensure you have the following prerequisites in place:
-
Digital Ocean Account: If you do not have an account, I would recommend creating it through my referral to get 200 USD credits for 60 days. In this time you can test this solution.
-
Terraform installed on your local machine
-
Clone the repository
git clone --depth=1 --branch=main https://github.com/tbhaxor/ProxyForge.git proxyforge cd proxyforge
-
Create a file for terraform variables.
touch terraform.tfvars
Note Skip it, if you wish to provide the variables and values from environment config.
-
Setup at least
token
andregion
in the file. See variables section below. -
Initialize the provider, plan and apply the changes
terraform init terraform plan terraform apply
-
Wait for some time for squid proxy to setup and load balancer to initialize routing.
-
(Optional) Test the deployment
LOAD_BALANCER_IP=$(terraform output -raw lb-ip) SQUID_USERNAME=proxyforge # assuming you did not change squid-credentials.username in the tfvars SQUID_PASSWORD=proxyforge # assuming you did not change squid-credentials.password in the tfvars while true; do curl -x "http://$SQUID_USERNAME:$SQUID_PASSWORD@$LOAD_BALANCER_IP:80" https://ifconfig.me; echo; done
Variable Name | Default | Description |
---|---|---|
token |
N/A | Required API token with read/write permissions. see more |
region |
N/A | Required Datacenter region to deploy all the resources. see more |
ssh-fingerprint |
null |
SSH fingerprint id for droplets to use. If this is ommited, it will send you one-time-password on the email. Can be obtained from security tab of accounts page. see also |
prefix |
pf | A prefix to quickly identify proxy-forge resources. |
slave-count |
2 | Number of instances on which squid proxy will be installed. |
lb-count |
1 | Number of master nodes to setup for load balancer, min 1 is required. |
tag-name |
proxy-forge-slave | Tag name to group slave droplets. |
droplet-size |
{ slave = "s-1vcpu-1gb-amd" } | Droplet size to use. |
project |
Proxy Forge | Name of the project to associate all the resources. |
squid-credentials |
{ password = "proxyforge", username = "proxyforge" } | Squid proxy HTTP basic authentication.credentials |
Configuring ProxyForge to suit your specific needs is a straightforward process. All you need to do is update the settings in the terraform.tfvars
file to align with your requirements. This file serves as a central configuration hub, allowing you to tailor ProxyForge precisely to your preferences without diving deep into complex setup procedures.
Note After you choose to make any changes from below, make sure to apply it on digitalocean
terraform plan terraform apply
squid-credentials = {
username = "YOUR NEW USERNAME"
password = "Y0ur53cr37P@55W0rD"
}
slave-count = 5
- Add a user friendly dashboard for administration of squid accounts
- Make a master service to destroy and spin-up new droplet in same region
- Multiple regions support
Proxy Forge is licensed under the MIT License. See the LICENSE file for details.
Thank you for choosing Proxy Forge! I hope this tool helps you overcome IP throttling and blacklisting challenges, making your web-related tasks smoother and more efficient. If you have any questions or encounter issues, feel free to reach out to me. Happy proxying!
Website: https://tbhaxor.com
Email Address: [email protected]
LinkedIn: @tbhaxor
Twitter: @tbhaxor
Reddit: @tbhaxor
Discord: @tbhaxor.com