.htaccess

擴張子無しでアクセス出來るようにする

Options +MultiViews

擴張子無しに轉送する

# file extension
RewriteCond %{THE_REQUEST} \.(html|htm)\ HTTP
RewriteRule ^(.*)\.(html|htm)$ /$1 [R=301,L]

indexを/に轉送する

# index.html to /
RewriteCond %{THE_REQUEST} ^(.*)/index\.[^/\.]+
RewriteRule ^(.*)/index\.[^/\.]+$ /$1 [R=301,L]
RewriteRule ^index\.[^/\.]+$ / [R=301,L]
RewriteRule ^index$ / [R=301,L]

MIMEの設定

AddType "application/xhtml+xml; charset=utf-8" xhtml xht
AddType "text/html; charset=utf-8" html htm
AddType "text/css; charset=utf-8" css
AddType "text/plain; charset=utf-8" txt
AddType application/pdf pdf
AddType image/png png
AddType image/jpeg jpeg jpg
AddType image/vnd.adobe.photoshop psd

アクセス拒否

ユーザーエージェントでの指定

SetEnvIf User-Agent "Googlebot" denybot
# 大文字・小文字を區別しない
SetEnvIfNoCase User-Agent "bingbot" denybot
Order Allow,Deny
Allow from all
Deny from env=denybot

IPアドレスでの指定

# crawl-XXX.googlebot.com
Deny from 66.249.64.0/19