I noticed that google was going crazy indexing trac for doctrine. Today it downloaded over 90000 pages, transfering 3 gig of data! It was causing quite a bit of load on the server (not huge amounts, but enough to show in my graphs!)
Eventaully , I came up with a nice little trick for reducing the number of hits google will make against a trac install. Google have extended robots.txt to allow some slightly improved pattern matching. Here’s my snippet, if you don’t understand it, please don’t use it.
User-Agent: Googlebot Disallow: /*?rev*