Well, you have to think about it... You're specifically telling the spider what URLs you want crawled, so I'm not sure if it would respect your robots.txt in this case, since you're ordering it to crawl specific links lol... The only people with a valid answer here would be the Google folks I think...


I am a Web Development Contractor, I do not work for UBBCentral. I have provided free User to User Support since the beginning of these support forums.
Do you need Forum Install or Upgrade Services?
Forums: A Gardeners Forum, Scouters World
UBB.threads: UBBWiki, UBB Styles, UBB.Sitemaps
Longtime Supporter & Resident Post-A-Holic
VNC Web Services: Code Modifications, Upgrades, Styling, Coding Services, Disaster Recovery, and more!