Agent: Nusearch Spider (www.nusearch.com)
Host: 84.9.136.223
The Nusearch Spider dropped by for a visit. It only followed the first ten top-level html links. It was not interested in going to other directories, and even tried to load directory-names as files by dropping the trailing "/" then ignoring the resulting redirect.
The Spider obeyed some of the directives in the robots.txt, but not all. My guess its a configuration issue with the bot at this time. We will be watching them to see what's up.
I dropped by their site, nusearch.com and its yet another search engine promising to be better than the other guys. Ya whatever, but they need to get their bot under better control, and clean up it's blacklist status if they want us to allow them to crawl our sites.
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment