I have a website like IMDB. All data are publicly available. I want to know how to protect the data from web scrapers ?
Asked
Active
Viewed 2,574 times
0
-
You can find some good answers in this thread: https://stackoverflow.com/questions/14377295/protection-from-web-scraping – Adrian Pacala Mar 14 '18 at 09:56
-
there are lot of things that will need to be done if you have to implement it yourself.. its better to take third party service instead.. – rock321987 Mar 14 '18 at 10:29
-
@rock321987 what are the third party service available ? – Sayanthan Mar 14 '18 at 10:59
-
@Sayanthan distil network is the best.. glassdoor, manta and many others uses their service – rock321987 Mar 14 '18 at 11:02
1 Answers
0
There is only one full-proof method against scrapers, that is captcha. But as it affects user experience most of the websites avoid it.
Another option is using AJAX for loading data. This will help avoiding the scrapers which are not built to render JavaScript, but one can make one using Selenium WebDriver. In addition AJAX is also bad for SEO, in case you are into google rankings and all.
A more effiecient and awesome way will be tracking the user behaviour and saving the information into cookies, if something seems suspicious serve a captcha to user. Just how the google captcha works on several sites.
Check this link : https://blog.hartleybrody.com/prevent-scrapers/
Vipin Joshi
- 305
- 5
- 18