It's a bit of a tautological observation, but I'm sure that web sites that
people actually visit don't have the problem that I do, which is that
90% of my web traffic is bots.  When my keystrokes start lagging, I can
take a look at the web logs and there's a bunch of web crawlers,
hammering away.  I could use a daemon that watches the logs for crawlers
and locks them out temporarily when it sees them ignoring Crawl-delay or
requesting crazy stuff or something.  Another interesting project for the
project jar.  I've got more full jars than I have years left on this planet,
which is why I have a hard time caring about things like the lawn or the roof.

Anyway, here're a couple things I noticed when looking through the logs for
"Googlebot"...

                             . . .

66.249.71.137 - - [27/Aug/2011:21:22:52 -0700] "GET /music/tusasada.mp3 HTTP/1.1" 200 2778094 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.72.227 - - [30/Aug/2011:06:18:43 -0700] "GET /music/base_nine.mp3 HTTP/1.1" 200 3861943 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Googlebot downloads entire MP3s off the Internet.  Since Google's an incredibly
wealthy downloader of MP3s, maybe they should consider helping out with the legal
defenses of some of the single moms that have been sued by the RIAA fishing goons
for doing exactly what they're doing...?

                             . . .

66.249.71.178 - - [16/Oct/2011:08:17:35 -0700] "GET /zikndkpn.html HTTP/1.1" 404 1973 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.67.183 - - [07/Sep/2011:06:19:30 -0700] "GET /s.. HTTP/1.1" 404 1974 "-" "DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)"

Googlebot is just trying random crap sometimes...?

                             . . .

66.249.71.137 - - [28/Aug/2011:17:43:29 -0700] "GET /palm/screen01.png HTTP/1.1" 200 824 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.71.137 - - [28/Aug/2011:17:43:30 -0700] "GET /palm/screen04.png HTTP/1.1" 200 523 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.71.137 - - [28/Aug/2011:17:43:30 -0700] "GET /palm/screen05.png HTTP/1.1" 200 545 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.71.137 - - [28/Aug/2011:17:43:30 -0700] "GET /palm/screen06.png HTTP/1.1" 200 302 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.71.137 - - [28/Aug/2011:17:43:31 -0700] "GET /palm/pose3.jpg HTTP/1.1" 200 21289 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.71.137 - - [28/Aug/2011:17:43:31 -0700] "GET /palm/screen07.png HTTP/1.1" 200 436 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.71.137 - - [28/Aug/2011:17:44:32 -0700] "GET /insurance.html HTTP/1.1" 304 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

This is kinda annoying.  Crawl-delay: 300 in my robots.txt means a delay of 300
seconds between GETs, not 300 milliseconds.  No wonder my keystrokes are lagging.

                             . . .

66.249.67.183 - - [26/Aug/2011:07:24:15 -0700] "GET /hardware HTTP/1.1" 301 325 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.67.183 - - [26/Aug/2011:07:24:16 -0700] "GET /hardware/ HTTP/1.1" 200 3205 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.67.183 - - [26/Aug/2011:08:27:16 -0700] "GET /atguard/atfaq.htm HTTP/1.1" 200 224 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.67.183 - - [26/Aug/2011:08:27:17 -0700] "GET /atguard/ HTTP/1.1" 200 1903 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Again with the apparently nonqueued subsystems ignoring Crawl-delay.  In all fairness,
Googlebot is actually pretty good about Crawl-delay most of the time, they just
sometimes push it with what I think they think will be smaller page elements.

                             . . .

OMG! Googlebot is having a baby!  Here is Googlebot stopping by to chat with the Baby Namer 2000
bot periodically, mentioning the first name that they've already come up with and asking
the Baby Namer 2000 to suggest a complementary middle name.  Unfortunately, I only recorded
Googlebot's side of the conversation!  :(  But, at least we can see what direction Googlebot
is headed in!  :)  Oh honey, anything but Latrisha... Drusilla's kind of nice, very sciency,
Maximina sounds.. I don't know, a little too overlordish perhaps?  No?  Okay, sure, I can see
that...  it's you.

66.249.67.183 - - [27/Aug/2011:06:42:40 -0700] "GET /babynames.shtml?br=1&first=santina
66.249.71.137 - - [28/Aug/2011:01:55:56 -0700] "GET /babynames.shtml?br=1&first=sharilyn
66.249.71.137 - - [28/Aug/2011:08:05:50 -0700] "GET /babynames.shtml?br=1&first=clotilde
66.249.71.137 - - [28/Aug/2011:09:03:46 -0700] "GET /babynames.shtml?br=1&first=contessa
66.249.71.137 - - [28/Aug/2011:10:01:38 -0700] "GET /babynames.shtml?br=1&first=kasandra
66.249.72.227 - - [30/Aug/2011:10:21:00 -0700] "GET /babynames.shtml?br=1&first=armandina
66.249.67.183 - - [30/Aug/2011:15:24:02 -0700] "GET /babynames.shtml?br=1&first=carolann
66.249.67.183 - - [31/Aug/2011:12:05:40 -0700] "GET /babynames.shtml?br=1&first=kathlene
66.249.67.183 - - [31/Aug/2011:14:58:30 -0700] "GET /babynames.shtml?br=1&first=shirleen
66.249.67.183 - - [05/Sep/2011:04:59:55 -0700] "GET /babynames.shtml?br=1&first=nicholle
66.249.67.183 - - [05/Sep/2011:05:58:07 -0700] "GET /babynames.shtml?br=1&first=hipolito
66.249.67.183 - - [05/Sep/2011:06:58:07 -0700] "GET /babynames.shtml?br=1&first=antonina
66.249.67.183 - - [05/Sep/2011:08:58:35 -0700] "GET /babynames.shtml?br=1&first=mellissa
66.249.67.183 - - [05/Sep/2011:11:59:12 -0700] "GET /babynames.shtml?br=1&first=enriqueta
66.249.67.183 - - [05/Sep/2011:12:59:07 -0700] "GET /babynames.shtml?br=1&first=drusilla
66.249.67.183 - - [05/Sep/2011:13:59:21 -0700] "GET /babynames.shtml?br=1&first=faustina
66.249.67.183 - - [05/Sep/2011:14:59:36 -0700] "GET /babynames.shtml?br=1&first=lucrecia
66.249.67.183 - - [06/Sep/2011:15:39:40 -0700] "GET /babynames.shtml?br=1&first=harland
66.249.71.137 - - [08/Sep/2011:07:20:40 -0700] "GET /babynames.shtml?br=1&first=melaine
66.249.71.137 - - [08/Sep/2011:15:27:41 -0700] "GET /babynames.shtml?br=1&first=terrilyn
66.249.71.137 - - [08/Sep/2011:16:20:04 -0700] "GET /babynames.shtml?br=1&first=petronila
66.249.71.137 - - [08/Sep/2011:17:29:10 -0700] "GET /babynames.shtml?br=1&first=domenic
66.249.71.137 - - [08/Sep/2011:19:47:08 -0700] "GET /babynames.shtml?br=1&first=jaqueline
66.249.71.137 - - [08/Sep/2011:20:56:03 -0700] "GET /babynames.shtml?br=1&first=alphonso
66.249.71.137 - - [08/Sep/2011:22:04:48 -0700] "GET /babynames.shtml?br=1&first=christinia
66.249.71.137 - - [08/Sep/2011:23:13:48 -0700] "GET /babynames.shtml?br=1&first=rodolfo
66.249.71.137 - - [09/Sep/2011:01:31:42 -0700] "GET /babynames.shtml?br=1&first=christoper
66.249.71.137 - - [09/Sep/2011:02:40:43 -0700] "GET /babynames.shtml?br=1&first=toshiko
66.249.71.137 - - [09/Sep/2011:03:49:54 -0700] "GET /babynames.shtml?br=1&first=armando
66.249.67.183 - - [09/Sep/2011:04:59:06 -0700] "GET /babynames.shtml?br=1&first=reynalda
66.249.67.183 - - [09/Sep/2011:06:07:55 -0700] "GET /babynames.shtml?br=1&first=willette
66.249.67.183 - - [09/Sep/2011:07:16:59 -0700] "GET /babynames.shtml?br=1&first=genoveva
66.249.67.183 - - [09/Sep/2011:09:35:07 -0700] "GET /babynames.shtml?br=1&first=jeanetta
66.249.67.183 - - [09/Sep/2011:20:50:05 -0700] "GET /babynames.shtml?br=1&first=mathilda
66.249.67.183 - - [09/Sep/2011:23:14:47 -0700] "GET /babynames.shtml?br=1&first=shalanda
66.249.67.183 - - [13/Sep/2011:11:58:07 -0700] "GET /babynames.shtml?br=1&first=anneliese
66.249.67.183 - - [13/Sep/2011:14:39:40 -0700] "GET /babynames.shtml?br=1&first=hermine
66.249.67.183 - - [13/Sep/2011:15:33:38 -0700] "GET /babynames.shtml?br=1&first=annamarie
66.249.67.183 - - [13/Sep/2011:16:27:48 -0700] "GET /babynames.shtml?br=1&first=dorthea
66.249.67.183 - - [13/Sep/2011:17:21:30 -0700] "GET /babynames.shtml?br=1&first=gabriella
66.249.67.183 - - [14/Sep/2011:10:46:44 -0700] "GET /babynames.shtml?br=1&first=sanjuanita
66.249.67.183 - - [14/Sep/2011:13:18:27 -0700] "GET /babynames.shtml?br=1&first=corrinne
66.249.67.183 - - [14/Sep/2011:14:43:12 -0700] "GET /babynames.shtml?br=1&first=marquetta
66.249.67.183 - - [14/Sep/2011:17:16:42 -0700] "GET /babynames.shtml?br=1&first=carlene
66.249.67.183 - - [14/Sep/2011:18:34:08 -0700] "GET /babynames.shtml?br=1&first=kathline
66.249.67.183 - - [14/Sep/2011:19:52:44 -0700] "GET /babynames.shtml?br=1&first=leonarda
66.249.67.183 - - [14/Sep/2011:21:10:47 -0700] "GET /babynames.shtml?br=1&first=bernardina
66.249.67.183 - - [14/Sep/2011:22:29:08 -0700] "GET /babynames.shtml?br=1&first=philomena
66.249.67.183 - - [16/Sep/2011:00:10:20 -0700] "GET /babynames.shtml?br=1&first=shawanna
66.249.67.183 - - [16/Sep/2011:01:02:55 -0700] "GET /babynames.shtml?br=1&first=latrisha
66.249.67.183 - - [19/Sep/2011:05:34:22 -0700] "GET /babynames.shtml?br=1&first=mauricio
66.249.68.247 - - [19/Sep/2011:20:16:46 -0700] "GET /babynames.shtml?br=1&first=madelene
66.249.68.247 - - [19/Sep/2011:23:33:19 -0700] "GET /babynames.shtml?br=1&first=demarcus
66.249.71.251 - - [21/Sep/2011:17:02:28 -0700] "GET /babynames.shtml?br=1&first=brenton
66.249.71.251 - - [21/Sep/2011:17:39:48 -0700] "GET /babynames.shtml?br=1&first=maximina
66.249.71.251 - - [22/Sep/2011:23:08:27 -0700] "GET /babynames.shtml?br=1&first=ivelisse
66.249.71.251 - - [24/Sep/2011:03:55:42 -0700] "GET /babynames.shtml?br=1&first=lakenya
66.249.71.251 - - [24/Sep/2011:04:53:59 -0700] "GET /babynames.shtml?br=1&first=kathrine
66.249.71.251 - - [25/Sep/2011:10:24:39 -0700] "GET /babynames.shtml?br=1&first=caroyln
66.249.68.178 - - [04/Oct/2011:00:37:13 -0700] "GET /babynames.shtml?br=1&first=willette
66.249.68.200 - - [10/Oct/2011:09:48:55 -0700] "GET /babynames.shtml?br=1&first=maryellen