Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for forwarding proxy #67

Open
TolstochenkoDaniil opened this issue Sep 11, 2020 · 0 comments
Open

Add support for forwarding proxy #67

TolstochenkoDaniil opened this issue Sep 11, 2020 · 0 comments

Comments

@TolstochenkoDaniil
Copy link

Prefase 📝

I am using scrapyd-client utility to deploy spiders on prod server.
In case to avoid my spiders to be banned, IT support configured forward proxy to hide host IP.


Issue ⚠️

After forward proxy was enabled, scrapy-deploy target -p project command was not available any more.
Server responded with 503 - Forwarding failure (Privoxy@unknown) error and such message:

Privoxy was unable to socks5-forward your request http://xxx.xxx.xxx.xxx:6800/addversion.json through 127.0.0.1: SOCKS5 request failed


Work around 🛠️

I was able to find solution to deploy spiders to JSONAPI scrapyd server.

  1. Create .egg file

run scrapyd
execute setup.py python setup.py bdist_egg

  1. Make post request as follow

curl --noproxy '*' http://<your_host>:/addversion.json -F project=<project_name> -F version=<version_name> -F egg=@<path_to_egg>

After doing steps i've mentioned above, i finally managed to deploy project. ✅

Hope, this would help someone in future! ✌🏼

@jpmckinney jpmckinney changed the title Forwarding proxy error when using scrapyd-deploy command Add support for forwarding proxy Sep 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants