So I have a search engine spider that I use for my internal site. Problem is when I use that spider on my vBulletin forum when it goes to scan PDFs I have attached to a thread, my spider is getting an error: (meta robots "noindex" tag found).
When I look at the source of the thread I do not see any rel= tags associated with the attachments. Also attachment.php has no such code. So when a spider gets to the PDFs where is the URL picking up the noidex tag from?