I have a multi tenant application which changes styles/content based on the host name. This also means that there is a different sitemap for each site. In order to have a different sitemap for each site I'm going to need to have a dynamically served robots.txt file which updates the sitemap url depending on the current site it's in.
I've tried this:
Route::get('robots.txt', function(){
echo "Here!";
});
But the route always throws a 404 (https://www.dropbox.com/s/97kdwnxz1383i3a/Screenshot%202016-02-03%2008.08.07.png?dl=0)
Files not ending in .php won't get sent to PHP-FPM for processing - This is a configuration in the web server, however what and how depends on the web server and how you've configured it.
It's possible you may also need to edit the php-fpm conf to allow .txt as a file extension it will allow to be parsed as PHP, however that protection is off by default.
My company made a package that solves this problem.
https://github.com/spatie/laravel-robots-middleware
Instead of you having to create a robots.txt this package will add a x-robots-tag to every request. You can extend the middleware to add custom logic that determines if the tag should be added and what the contents of the tag should be.
I solved it! I added this rule to .htaccess file which allowed me to have a route for robots.txt:
RewriteCond %robots.txt -f [NC]
@okawei I think you are using apache. Please provide the answer for nginx.
if RewriteCond is not working try using RewriteRule ^robots.txt -f [NC]
that worked for me, apache server
Files not ending in .php won't get sent to PHP-FPM for processing - This is a configuration in the web server, however what and how depends on the web server and how you've configured it.
It seems this is not the case. I have been using a package for generating sitemaps "watson/sitemap" and out of the box it's allowed me to create routes for things such as "sitemap.xml" and "sitemap.index.xml"
Also just for testing I created a route for "test.txt" and that also works
The reason that "robots.txt" doesn't work is because both Laravel Homestead and Laravel Forge ship with a line in their nginx vhost configuration that says:
location = /robots.txt { access_log off; log_not_found off; }
And my interpretation through cause and effect is that this line causes "robots.txt" to be treated as a file even if it is missing
Hmm yeah, I think if a file doens't exist on the server, it may be getting sent to index.php with the usual nginx setup, so that could def make sense!
I'm using Laravel Valet and solved a 404 error simply securing a site with valet secure project
I test it with a route pointing to this simple function:
public function robots()
{
return response(view('robots'))->header('Content-Type', 'text/plain');
}
Before enabling TLS route it worked, content was correctly displayed but with a 404 status even forcing it to 200
I hope in production works the same way...
Please or to participate in this conversation.