1

I'm trying to make scrapy go through proxy, but without success.

I'm passing to request meta proxy like:

http://zqal.xyz
http://pro6y.com
http://fermatfibonacci.info
http://newproxy.ninja
etc.

And when I type some random page through shell, displayed page is not the one that I pass, but proxy page. It's looks like I go to the proxy page, not THROUGH proxy...

Maybe wrong proxy link? Any idea's?

I'm setting USER_AGENT like my in web browser, and referer to google page.

Bartłomiej Zając
  • 71
  • 6
  • 1
    Those sites probably wont work, as they are not really proxy servers in the classic sense, rather a http page that routes your requests through their proxies. You can find some proxy servers here (they are pretty unreliable, but they are free): http://www.gatherproxy.com/; This is how you configure scrapy to use the proxies: http://stackoverflow.com/questions/4710483/scrapy-and-proxies (see second answer) Another alternative would be to use TOR – Ryan Jul 04 '16 at 10:43
  • @Ryan do you have some tutorial, or something about integrate TOR with Scrapy on Ubuntu? – Bartłomiej Zając Jul 04 '16 at 12:44
  • To set proxy you can either use environment variable or set Request.meta key. See here http://stackoverflow.com/questions/4710483/scrapy-and-proxies How are you setting proxy? What do you mean by "type some random page through shell"? – Pawel Miech Jul 04 '16 at 13:47
  • @PawelMiech i set request.meta['proxy'] to some proxy address Like: `scrapy shell http://onet.pl ` But now i trying to connect through TOR – Bartłomiej Zając Jul 04 '16 at 13:58

0 Answers0