Feature #6585
Get google searches working...
Start date:
07/01/2014
Due date:
% Done:
0%
Estimated time:
Description
So it turns out Redmine makes a dynamically generated robots.txt (see https://cdcvs.fnal.gov/redmine/robots.txt)
to prevent bots from scanning parts of Redmine that would bog things down, etc. but this doesn't work
in our currenc configuration, because we map it to a subtree (/redmine) and
- robots.txt is only looked for at the top level
- it doesn't have the /redmine in the paths
So we need to put in a script to convert the Redmine one to a correct one and put it
at http{s,}://cdcvs.fnal.gov/robots.txt so it will take effect.
History
#1 Updated by Marc Mengel almost 7 years ago
- File make_robots_txt make_robots_txt added
- Description updated (diff)
Put a draft together of a script to do this, need to put it in crontab and
give it teeth (it has to rename robots.txt.new robots.txt at the end).
We'll need to do a change request for this.