I think it would be very nice if a standard robots.txt file could be included with vbulletin.
In this file, the basic rules should be included to prohibit all non-essential links to search engines.
I think it's easier for developers to create this file, as for us users.
Of course, the file may contain only the standard rules. Special rules, the user must create itself.