Use residential proxies in Scrapy: meta["proxy"] with a login, a small middleware for rotating or sticky IPs per session, retries, timeouts and saving data.
Scrapy has proxy support built in. Its HttpProxyMiddleware reads request.meta["proxy"], and if the URL contains a username and password, it adds the Proxy-Authorization header for you. So the whole setup is: get the right URL into every request. This guide does it with one rotating line, then with a list of sticky sessions.
Step 1: Generate your proxy lines
Sign in to the Proxare dashboard, open the product you bought and switch to Generate. For this tool, use:
- Country: the country your accounts or targets belong to. Add a city only if you really need it, because a narrow target has fewer IPs.
- IP rotation: Rotating for independent pages (listings, prices, search results). Sticky when a spider logs in or follows a multi-step flow; 10 or 30 minutes is usually enough.
- Protocol: HTTP(S). Scrapy has no built-in SOCKS5 support, and an HTTP proxy handles https sites through a CONNECT tunnel, so there is no reason to switch.
- Server region: Global (nearest) unless your tool runs far from the target country. This picks the gateway you connect to, not the exit country.
- Format:
protocol://user:pass@host:port, the URL format Scrapy expects. - How many: one line for rotating; for sticky, as many lines as you want parallel sessions.
Each line carries your login plus the targeting and session options. On a Proxare Premium plan with a sticky US session it looks like this (yours has your own login, and the host and port depend on the product):
premium.proxare.com:6011:px_alice:k7Qm2x-country-us_session-ab12cd34_lifetime-30m
The part after your password (-country-us_session-…) is how the gateway knows which country and which session you want. Always copy the whole generated line; if you type your plain dashboard password into a tool, you get a random country and a new IP on every connection.
The quickest test
Scrapy also picks up the standard proxy environment variables, so you can check a line without touching the project:
https_proxy='http://px_alice:k7Qm2x-country-us@premium.proxare.com:6011' scrapy fetch --nolog https://api.ipify.org
It prints the exit IP. Run it twice: on a rotating line the IP changes. Single quotes stop the shell from reading characters in the password.
Option 1: one rotating line in the spider
import scrapy
PROXY = "http://px_alice:k7Qm2x-country-us@premium.proxare.com:6011"
class PricesSpider(scrapy.Spider):
name = "prices"
start_urls = ["https://example.com/catalog?page=1"]
def start_requests(self):
for url in self.start_urls:
yield scrapy.Request(url, meta={"proxy": PROXY})
def parse(self, response):
...
for href in response.css("a.next::attr(href)").getall():
yield response.follow(href, meta={"proxy": PROXY})
Every request gets a new IP from the gateway. Keep the login out of the code in real projects: read it from an environment variable or settings.py.
Option 2: a middleware for every request, rotating or sticky
A tiny downloader middleware sets the proxy on every request, so spiders do not have to. Save the lines from the dashboard (URL format, Copy all) to proxies.txt, one per line. With one rotating line, every request gets a new IP. With several sticky lines, each Scrapy cookiejar keeps its own line, which means one session, one IP and one set of cookies belong together.
# middlewares.py
import itertools
from pathlib import Path
class ProxareProxyMiddleware:
def __init__(self, lines):
self.lines = lines
self.next_line = itertools.cycle(lines)
self.by_jar = {}
@classmethod
def from_crawler(cls, crawler):
path = Path(crawler.settings.get("PROXARE_LIST", "proxies.txt"))
lines = [l.strip() for l in path.read_text().splitlines() if l.strip()]
if not lines:
raise ValueError(f"{path} has no proxy lines")
return cls(lines)
def process_request(self, request, spider):
if "proxy" in request.meta:
return None # set by hand for this request
jar = request.meta.get("cookiejar")
if jar is None:
request.meta["proxy"] = next(self.next_line)
else:
if jar not in self.by_jar:
self.by_jar[jar] = next(self.next_line)
request.meta["proxy"] = self.by_jar[jar]
return None
# settings.py
DOWNLOADER_MIDDLEWARES = {
"myproject.middlewares.ProxareProxyMiddleware": 350, # before HttpProxyMiddleware (750)
}
PROXARE_LIST = "proxies.txt"
DOWNLOAD_TIMEOUT = 45 # default is 180 s, too long for a dead IP
RETRY_TIMES = 3 # a retry through a rotating line gets a new IP
CONCURRENT_REQUESTS = 32
CONCURRENT_REQUESTS_PER_DOMAIN = 8
AUTOTHROTTLE_ENABLED = True
To pin a flow to one IP, give its requests the same meta={"cookiejar": "account-7"} and pass it on in every follow-up request. A sticky line from the dashboard looks like this:
http://px_alice:k7Qm2x-country-us_session-ab12cd34_lifetime-30m@premium.proxare.com:6011
Retries and status codes
- 407: wrong login. Scrapy does not retry it by default, which is right: fix the line instead.
- 429, 502, 503: retried by default. On a rotating line each retry goes out through a new IP, which is what you want.
- 403 or a CAPTCHA page with status 200: the site is blocking the request, not the proxy failing. Lower
CONCURRENT_REQUESTS_PER_DOMAIN, send realistic headers, and consider mobile vs residential proxies for the hardest targets.
Save data while you develop
Every run of a spider you are still debugging costs traffic. Turn on Scrapy's HTTP cache during development so repeated runs read pages from disk:
HTTPCACHE_ENABLED = True
HTTPCACHE_EXPIRATION_SECS = 86400
Scrapy downloads HTML only, no images or scripts, so it is one of the cheapest ways to use residential proxies: typically 50 to 300 KB per page. See how many GB you need.
Troubleshooting
- 407 or "proxy authentication failed": the password field is missing the options part, a character was lost when copying, or the line has a space at the end. Paste the generated line again.
- Connection refused or timeout: wrong port, or the protocol in the tool does not match the line (a SOCKS5 line set as HTTP or the other way round). Generate the line again with the protocol the tool uses.
- Wrong country: geolocation databases disagree, especially at city level. Check the exit IP on ipinfo.io before you trust a checker built into a tool.
- "No IP available" / 502: the city or carrier is too narrow. Widen to country level.
- Works in curl, fails in the tool: it is almost always the format or protocol. See the guide on proxy formats.
- "Could not resolve proxy" or a strange host in the error: the password contains
@,:or#. Encode it withurllib.parse.quote(password, safe=""). - Sticky requests come from different IPs: a follow-up request lost its
cookiejarmeta, so the middleware gave it the next line.
Still stuck? Run the line through curl first (see how to test a proxy with curl): if curl works, the problem is the tool's settings, and support can tell you which. Message us on Telegram with the error text, never with your password.
Setup guides for other tools
Menu names can change between Scrapy versions; if a label differs, look for the proxy section of the same screen. Proxare is not affiliated with Scrapy. All product names are trademarks of their owners.