Background: Our client wanted to create a data mining and web scraping software to gather large amount of data from internet.
Challenge: With so many diverse set of data patterns, it was a challenging task to exactly pull the right information in 100% automatic mode.
Solution:Being already developed data mining and data extraction software, we had a head-start. We used microsoft .net for building the crawler and written custom algorithms to identify and extract desired data.
Result: After initial bug fixes and performance testing, the data mining and web scraping software crawled and scraped at an average of 20,000 pages / hour with useful data for business and competitor analysis.