Initial commit

9780596005771

 No newline at end of file

## Example files for the title:  

# Spidering Hacks, by Tara Calishain

[![Spidering Hacks, by Tara Calishain](http://akamaicovers.oreilly.com/images/9780596005771/cat.gif)](https://www.safaribooksonline.com/)

The following applies to example files from material published by O’Reilly Media, Inc. Content from other publishers may include different rules of usage. Please refer to any additional usage rights explained in the actual example files or refer to the publisher’s website.

O'Reilly books are here to help you get your job done. In general, you may use the code in O'Reilly books in your programs and documentation. You do not need to contact us for permission unless you're reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from our books does not require permission. Answering a question by citing our books and quoting example code does not require permission. On the other hand, selling or distributing a CD-ROM of examples from O'Reilly books does require permission. Incorporating a significant amount of example code from our books into your product's documentation does require permission.

We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN.

If you think your use of code examples falls outside fair use or the permission given here, feel free to contact us at <permissions@oreilly.com>.

Please note that the examples are not production code and have not been carefully testing. They are provided "as-is" and come with no warranty of any kind.

#!/usr/bin/perl -w

#

# Progress Bar: Dots - Simple example of an LWP progress bar.

# http://disobey.com/d/code/ or contact morbus@disobey.com.

#

# This code is free software; you can redistribute it and/or

# modify it under the same terms as Perl itself.

#

use strict; $|++;

my $VERSION = "1.0";

# make sure we have the modules we need, else die peacefully.

eval("use LWP 5.6.9;");  die "[err] LWP 5.6.9 or greater required.\n" if $@;

# now, check for passed URLs for downloading.

die "[err] No URLs were passed for processing.\n" unless @ARGV;

# our downloaded data.

my $final_data = undef;

# loop through each URL.

foreach my $url (@ARGV) {

   print "Downloading URL at ", substr($url, 0, 40), "... ";

   # create a new useragent and download the actual URL.

   # all the data gets thrown into $final_data, which

   # the callback subroutine appends to.

   my $ua = LWP::UserAgent->new(  );

   my $response = $ua->get($url, ':content_cb' => \&callback, );

   print "\n"; # after the final dot from downloading.

}

# per chunk.

sub callback {

   my ($data, $response, $protocol) = @_;

   $final_data .= $data;

   print ".";

}

 No newline at end of file