80legs

80legs, developed by Datafiniti, LLC, is a web crawling and data extraction tool designed to meet the needs of small to large companies. According to the vendor, 80legs enables business intelligence professionals, data analysts, market researchers, e-commerce businesses, and real estate companies to retrieve valuable data from the web. It offers customizable web crawls and a cloud platform for efficient data retrieval.

Key Features

Custom Web Crawling: According to the vendor, users can create fully-customized web crawls and leverage the cloud platform for quick data retrieval. They can customize scraping requirements, choose which data to scrape, and which links to follow. Default crawling apps allow the collection of common HTML data like links, keywords, and meta tags.

Giant Web Crawl: The vendor claims that users can access a crawl of the entire web and extract data based on custom filters. They can customize the Giant Web Crawl to scrape data specific to their needs. It allows crawling thousands of websites every month and scraping data based on customized logic. Users can retrieve matching results from the Giant Web Crawl for their 80legs account.

Datafiniti: According to the vendor, users gain instant access to clean and structured web data for specific data types. They can skip web scraping and download entire datasets in minutes. Datafiniti constantly crawls the web and applies data sanitization and merging processes. Users can access comprehensive, accurate, and representative datasets through an online search interface or API.

Rate Limiting: According to the vendor, 80legs offers automatic rate limiting of requests to web pages for optimized crawl speeds.

Multiple IP Addresses: The vendor states that 80legs allows the distribution of requests across servers with different IP addresses.

RESTful API: According to the vendor, users can automate web crawl creation and result retrieval using the 80legs API.

Scraper Customization: The vendor claims that users can fully customize web crawling behavior using the Javascript-based app framework.

Coverage: According to the vendor, 80legs can crawl over 15 million US and European domains.

Speed: The vendor states that approximately one million URLs can be crawled every day using 80legs.

Accuracy: According to the vendor, Datafiniti provides more accurate and representative data than traditional scraped data.