Not all bots are going to use this file, but most of the good ones will.
We can start by creating a new file called robots.txt and add it in your template folder.
Add the following content to it:
User-agent: * Disallow: /admin/ Disallow: /api/
Here I say that all robots "*" should not go into /admin/ and /api/. You can add more pages here if you want to.
Then, the only thing we need to do is to add this to the url patterns:
# Import function from Django from django.views.generic.base import TemplateView # Append to url patterns path('robots.txt', TemplateView.as_view(template_name="robots.txt", content_type="text/plain")),
And that's it, you can now go to http://127.0.0.1:8000/robots.txt