Adding robots.txt to a Django site

Not all bots are going to use this file, but most of the good ones will.

We can start by creating a new file called robots.txt and add it in your template folder.

Add the following content to it:

User-agent: * 
Disallow: /admin/
Disallow: /api/

Here I say that all robots "*" should not go into /admin/ and /api/. You can add more pages here if you want to.

Then, the only thing we need to do is to add this to the url patterns:

# Import function from Django
from django.views.generic.base import TemplateView

# Append to url patterns
path('robots.txt', TemplateView.as_view(template_name="robots.txt", content_type="text/plain")),

And that's it, you can now go to http://127.0.0.1:8000/robots.txt