hop of those help? To be honest you are looking at nofollow wrong. Chances are the search spiders are already especially Google, Yahoo, and MSN searching the nofollow pages, because they still have to hit those pages to see if they have a noindex. The real problem is nofollow doesn't actually mean don't follow, it just means don't pass on my reputation to this link. So unless you are aggressively blocking bots, which it doesn't sound like you are, changing the ROBOTS meta tag and robot commands on links will not effect performance because they are already hitting your site. To confirm this just look at your HTTP Server Log.
What is a good Web search and web crawling engine for Java?
I wish this help you Is there a way to collect web content in order to use it in a search engine without passing by the web crawling phase? Any alternative to web crawling? , No, to collect the content you have to...collect the content. :-)