Top Ruby Frameworks & Libraries for web crawling 15

Fast high-level web crawling Ruby framework

Quick and dirty web crawling.

A Ruby DSL for structured web crawling, with a robust caching system.

A loose framework for crawling and scraping web sites.

ToSBack crawls, archives and tracks changes in terms of service and privacy policies. ToSBack3, inspired by EFF's ToSBack, is built in Ruby on Rails, features a we...

Implements a MongoDB back-end for Spidey (https://github.com/joeyAghion/spidey), a framework for crawling and scraping web sites....

A simple Ruby web spider that uses Anemone to crawl every page of a site looking for email addresses. Stores the results with SQLite3 using Data Mapper....

Wgit enables you to crawl and extract the data you want from the web

A simple web scraper built with Ruby and the Nokogiri gem. It crawls a certain website and gets the prices and other data of cryptocurrencies. Rspec was used for t...

An example job that converts Common Crawl archived web pages into text

web crawler to crawl nptel site and extract data

A Zillabyte Component for crawling web domains and upload screenshots of webpage contents

A ruby gem for crawling social media stats from various web entities

Crawl web sites and download images/pictures or everything you want

Ruby classes to make it easier to crawl a website, find the items you need (products usually) and save them to CSV...

This is a simple web scraper that basically crawls a website and retrieve specific data from the site. Built with Ruby and Nokogiri....

Web crawling with ruby.

ruby web crawling module

Web-crawling framework for Ruby