i seems age and senility are catching up with me ... i recall having this same spider-friendly problem on my on my old server, and rick has already provided some of the answers.
something relating to PATH_INFO environment variable, and putting the following into the php.ini file:
cgi.fix_pathinfo =1
i recall that fixed the problem last time. only thing is, now i forget where the php.ini file is located ...