i crawled JUST the message board only this am, turned OFF all panels except the main board before i indexed. that shut down the shout box, whos online, etc. should have helped eliminate duplicate content.
spider still ran forever.
set a limit of 350,000 files, and hit that limit. still had over a 100,000 files in the queue and that number was rising.
thought that might help the spider actually hit an end. it didn't.
