if *they* are listing the same page more than once, then what is the problem?
because I am also spidering the site and i don't like having a search engine with mass duplicates!
say you you enter in a search word. and get 150 results. 30 are good and the other 120 are various versions of 4 different files? looks cluttery, silly and rookie-ish.
fyi i ran the spider 48 hours on forum. TWO DAYS. got over 600,000 'files' indexed. cost me $50 in bandwidth as the thing pounded away for two days inside the forum. still mass dupes. total waste of time and money to try to spider this version.
good thing google has unlimited resources. wonder how much it COSTS ME when they try to spider it and start rattling around.
