Home / Setup guides

Scrapy proxy setup: rotating and sticky IPs

Updated November 15, 2026 · 5 min read

Use residential proxies in Scrapy: meta["proxy"] with a login, a small middleware for rotating or sticky IPs per session, retries, timeouts and saving data.

Scrapy has proxy support built in. Its HttpProxyMiddleware reads request.meta["proxy"], and if the URL contains a username and password, it adds the Proxy-Authorization header for you. So the whole setup is: get the right URL into every request. This guide does it with one rotating line, then with a list of sticky sessions.

Step 1: Generate your proxy lines

Sign in to the Proxare dashboard, open the product you bought and switch to Generate. For this tool, use:

Each line carries your login plus the targeting and session options. On a Proxare Premium plan with a sticky US session it looks like this (yours has your own login, and the host and port depend on the product):

premium.proxare.com:6011:px_alice:k7Qm2x-country-us_session-ab12cd34_lifetime-30m

The part after your password (-country-us_session-…) is how the gateway knows which country and which session you want. Always copy the whole generated line; if you type your plain dashboard password into a tool, you get a random country and a new IP on every connection.

The quickest test

Scrapy also picks up the standard proxy environment variables, so you can check a line without touching the project:

https_proxy='http://px_alice:k7Qm2x-country-us@premium.proxare.com:6011' scrapy fetch --nolog https://api.ipify.org

It prints the exit IP. Run it twice: on a rotating line the IP changes. Single quotes stop the shell from reading characters in the password.

Option 1: one rotating line in the spider

import scrapy

PROXY = "http://px_alice:k7Qm2x-country-us@premium.proxare.com:6011"

class PricesSpider(scrapy.Spider):
    name = "prices"
    start_urls = ["https://example.com/catalog?page=1"]

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(url, meta={"proxy": PROXY})

    def parse(self, response):
        ...
        for href in response.css("a.next::attr(href)").getall():
            yield response.follow(href, meta={"proxy": PROXY})

Every request gets a new IP from the gateway. Keep the login out of the code in real projects: read it from an environment variable or settings.py.

Option 2: a middleware for every request, rotating or sticky

A tiny downloader middleware sets the proxy on every request, so spiders do not have to. Save the lines from the dashboard (URL format, Copy all) to proxies.txt, one per line. With one rotating line, every request gets a new IP. With several sticky lines, each Scrapy cookiejar keeps its own line, which means one session, one IP and one set of cookies belong together.

# middlewares.py
import itertools
from pathlib import Path

class ProxareProxyMiddleware:
    def __init__(self, lines):
        self.lines = lines
        self.next_line = itertools.cycle(lines)
        self.by_jar = {}

    @classmethod
    def from_crawler(cls, crawler):
        path = Path(crawler.settings.get("PROXARE_LIST", "proxies.txt"))
        lines = [l.strip() for l in path.read_text().splitlines() if l.strip()]
        if not lines:
            raise ValueError(f"{path} has no proxy lines")
        return cls(lines)

    def process_request(self, request, spider):
        if "proxy" in request.meta:
            return None  # set by hand for this request
        jar = request.meta.get("cookiejar")
        if jar is None:
            request.meta["proxy"] = next(self.next_line)
        else:
            if jar not in self.by_jar:
                self.by_jar[jar] = next(self.next_line)
            request.meta["proxy"] = self.by_jar[jar]
        return None
# settings.py
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.ProxareProxyMiddleware": 350,  # before HttpProxyMiddleware (750)
}
PROXARE_LIST = "proxies.txt"

DOWNLOAD_TIMEOUT = 45          # default is 180 s, too long for a dead IP
RETRY_TIMES = 3                # a retry through a rotating line gets a new IP
CONCURRENT_REQUESTS = 32
CONCURRENT_REQUESTS_PER_DOMAIN = 8
AUTOTHROTTLE_ENABLED = True

To pin a flow to one IP, give its requests the same meta={"cookiejar": "account-7"} and pass it on in every follow-up request. A sticky line from the dashboard looks like this:

http://px_alice:k7Qm2x-country-us_session-ab12cd34_lifetime-30m@premium.proxare.com:6011

Retries and status codes

Save data while you develop

Every run of a spider you are still debugging costs traffic. Turn on Scrapy's HTTP cache during development so repeated runs read pages from disk:

HTTPCACHE_ENABLED = True
HTTPCACHE_EXPIRATION_SECS = 86400

Scrapy downloads HTML only, no images or scripts, so it is one of the cheapest ways to use residential proxies: typically 50 to 300 KB per page. See how many GB you need.

Troubleshooting

Still stuck? Run the line through curl first (see how to test a proxy with curl): if curl works, the problem is the tool's settings, and support can tell you which. Message us on Telegram with the error text, never with your password.

Setup guides for other tools

Menu names can change between Scrapy versions; if a label differs, look for the proxy section of the same screen. Proxare is not affiliated with Scrapy. All product names are trademarks of their owners.