I've gotten interested lately in doing some scraping of a couple of websites, to collect data that's interesting to me (not a moneymaking thing, unfortunately!).
I am aware that scraping isn't always something website owners are always happy about, but I think it's generally more about the impact that it has on their site, or when it's used for nefarious purposes (scraping prices to undercut them) than the fact that you are collecting the information.
So with that out of the way, my questions are as follows:
1) Do providers generally have any issues with this practice? Let's say the "load" is 5-10 pages every 600 seconds or so, with each page being 20Kb in size. The parsing is usually CPU intensive, but generally for just seconds at a time. Then delivery of that data to a central repository.
2) Do you guys have any suggestions or techniques that you are willing to share for scraping? I need to be able to maintain sessions, and at least present an appearance of it being a legitimate user at a web browser (user agent, etc). I ran across a couple of proxies that mention maintaining the cookies on the proxy server side, which would be ideal for me, but I haven't found one that's still maintained and will do it (although I'm not a well versed proxy guy).
LowEndTalk http://ift.tt/Lj386v Web Scraping - Couple of questions Web vps hosting server Linux - Windows webvps.blogspot.com
0 comments:
Post a Comment