NAME Scrappy - All Powerful Web Spidering, Scrapering, Crawling Framework VERSION version 0.9111110 SYNOPSIS #!/usr/bin/perl use Scrappy; my $scraper = Scrappy->new; $scraper->crawl('search.cpan.org', '/recent' => { '#cpansearch li a' => sub { print $_[1]->{href}, "\n"; } } ); DESCRIPTION Scrappy is an easy (and hopefully fun) way of scraping, spidering, and/or harvesting information from web pages, web services, and more. Scrappy is a feature rich, flexible, intelligent web automation tool. Scrappy (pronounced Scrap+Pee) == 'Scraper Happy' or 'Happy Scraper'; If you like you may call it Scrapy (pronounced Scrape+Pee) although Python has a web scraping framework by that name and this module is not a port of that one. METHODS crawl The crawl method is very useful when it is desired to crawl an entire website or at-least partially, it automates the tasks of creating a queue, fetching and parsing html pages, and establishing simple flow-control. See the SYNOPSIS for a simplified example, ... the following is a more complex example. my $scrappy = Scrappy->new; $scrappy->crawl('http://search.cpan.org/recent', '/recent' => { '#cpansearch li a' => sub { my ($self, $item) = @_; # follow all recent modules from search.cpan.org $self->queue->add($item->{href}); } }, '/~:author/:name-:version/' => { 'body' => sub { my ($self, $item, $args) = @_; my $reviews = $self ->select('.box table tr')->focus(3)->select('td.cell small a') ->data->[0]->{text}; $reviews = $reviews =~ /\d+ Reviews/ ? $reviews : '0 reviews'; print "found $args->{name} version $args->{version} ". "[$reviews] by $args->{author}\n"; } } ); AUTHOR Al Newkirk COPYRIGHT AND LICENSE This software is copyright (c) 2010 by awncorp. This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.