This bundle provides robots.txt generator for Symfony2
$ composer require evheniy/robots-txt-bundle "1.*"
Or add to composer.json
"evheniy/robots-txt-bundle": "1.*"
AppKernel:
public function registerBundles()
{
$bundles = array(
...
new Evheniy\RobotsTxtBundle\RobotsTxtBundle(),
...
);
...
...
config.yml:
robots_txt:
- 'User-agent: *'
Or with sitemap:
robots_txt:
- 'User-agent: *'
- 'Sitemap: http://test.com/sitemap.xml'
The last step
app/console robots.txt:dump
The main idea is to use robots.txt for different environments, for example: for dev hiding project from indexing robots like Google
config_prod.yml:
robots_txt:
- 'User-agent: *'
- 'Sitemap: http://test.com/sitemap.xml'
config_dev.yml:
robots_txt:
- 'User-agent: *'
- 'Disallow: /'
And run it for prod:
app/console robots.txt:dump --env=prod
And run it for dev:
app/console robots.txt:dump --env=dev
Or just
app/console robots.txt:dump
This bundle is under the MIT license.