What is a robots.txt file and how to create it?

Image Credit- Google Images

A robots.txt file is a text file that webmasters use to tell Googlebot, Other web crawlers which pages on their website should not be crawled and indexed.

[{"selector":"#anim-793a7c48-9ba6-46e7-82b8-9ccf0e4909ef","keyframes":{"transform":["translate3d(-115.97444%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-06efc596-cf9c-4790-bdf4-5ea148d8535d","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-72ea548b-7c33-4005-a2a1-f5738913d02a","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}] [{"selector":"#anim-66b0c09a-9236-4ae6-807b-7cfd865647f5 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate(0%, 0%) scale(1.5)","translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-f12691e8-bada-4e3d-abbb-e9976c608403","keyframes":[{"transform":"rotate(-540deg) scale(0.1)","opacity":0},{"transform":"none","opacity":1}],"delay":0,"duration":1000,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both","iterations":1}] [{"selector":"#anim-ecac2bbc-0640-4c8f-a456-d2d95868fb41","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-fcda35a4-8899-463e-a77c-3293ecd23bad","keyframes":{"transform":["translate3d(0px, 332.88870%, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] Image Credit- Google Images

The robots.txt file is placed in the root directory of your website.

[{"selector":"#anim-d3cec5de-a66f-47cb-9fff-411f2e289120 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate(0%, 0%) scale(1.5)","translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-57979d74-6ebd-4d2a-bc62-92b20b8cb1d7","keyframes":{"transform":["translate3d(-115.97444%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-ddf7f16a-53a1-4675-ba30-3ff2a4ceb225","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-6d7fd04f-09a0-476d-81f4-291cdd6b1ce8","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}] [{"selector":"#anim-42515523-a8da-4d48-873c-2caa2bfd927c","keyframes":[{"transform":"rotate(-540deg) scale(0.1)","opacity":0},{"transform":"none","opacity":1}],"delay":0,"duration":1000,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both","iterations":1}] [{"selector":"#anim-dfa29135-8257-4d82-bcdd-2d2aea630f9e","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-97c6683a-6e96-4c47-acb7-86043c7c0f9c","keyframes":{"transform":["translate3d(0px, 332.88870%, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] Image Credit- Google Images

And it can contain directives for multiple user-agents (such as Googlebot, Bingbot, etc.).

[{"selector":"#anim-519be4eb-1215-46c6-bd02-8d5ba318fc49 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate(0%, 0%) scale(1.5)","translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-b056fe8e-58ba-4c48-9584-86a86596b5ff","keyframes":{"transform":["translate3d(-115.97444%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-45b4ea1d-c394-4729-9c04-b3819d0d2555","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-5613c2f9-2f32-4ed9-845e-6a7987ff8c34","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}] [{"selector":"#anim-f19dbc50-29ae-4099-ad88-9786bb254973","keyframes":[{"transform":"rotate(-540deg) scale(0.1)","opacity":0},{"transform":"none","opacity":1}],"delay":0,"duration":1000,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both","iterations":1}] [{"selector":"#anim-1468031b-1bff-4db7-93b6-efa221b490bd","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-f13b8faa-26b5-4020-9133-ed4112360613","keyframes":{"transform":["translate3d(0px, 332.88870%, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] Image Credit- Google Images

The most common use for a robots.txt file is to tell web crawlers which pages on your website are off-limits.

[{"selector":"#anim-136404c1-9cf2-457e-94b9-433f45e35ebc [data-leaf-element=\"true\"]","keyframes":{"transform":["translate(0%, 0%) scale(1.5)","translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-27c477aa-fd6c-4d74-82db-8fa60410618f","keyframes":{"transform":["translate3d(-115.33547%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-cc703dbc-cea3-4b0a-ba74-aa6c17f46c55","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-faafca94-27b4-4082-94d7-d21a0a53422e","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}] [{"selector":"#anim-14408663-e889-402a-8aaa-4224810a9d3a","keyframes":[{"transform":"rotate(-540deg) scale(0.1)","opacity":0},{"transform":"none","opacity":1}],"delay":0,"duration":1000,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both","iterations":1}] [{"selector":"#anim-ec562955-0f6c-4402-90c1-2283c1a425e7","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-010f3ff7-5dea-4186-9876-0ff35461911b","keyframes":{"transform":["translate3d(0px, 332.88870%, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] Image Credit- Google Images

For example, you might use a robots.txt file to tell Googlebot not to crawl and index your site's admin area or private user profiles.

[{"selector":"#anim-650b986a-37df-4719-9e0f-6819cd18cc6c [data-leaf-element=\"true\"]","keyframes":{"transform":["translate(0%, 0%) scale(1.5)","translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-08eb1756-1471-434f-a56f-bb076759d4a1","keyframes":{"transform":["translate3d(-115.97444%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-cfa5726b-4433-486d-a1be-7910840013c3","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-2b0de999-3030-4984-a382-f5080d619291","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}] [{"selector":"#anim-42e83e24-8a6f-48d3-b943-f0b51af4988a","keyframes":[{"transform":"rotate(-540deg) scale(0.1)","opacity":0},{"transform":"none","opacity":1}],"delay":0,"duration":1000,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both","iterations":1}] [{"selector":"#anim-7066208c-e050-4e90-85e8-2f81f5dd8267","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-bb008997-f3b3-4de7-9abc-70976c951b8a","keyframes":{"transform":["translate3d(0px, 332.88870%, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] Image Credit- Google Images

How do I create a robots.txt file?

[{"selector":"#anim-3d21ffdc-e24c-4fd3-9724-6d09c2a21079 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate(0%, 0%) scale(1.5)","translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-81283c0d-5143-406d-ab0b-ebe54055f260","keyframes":{"transform":["translate3d(-115.97444%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-415026dc-867c-431d-8fe3-4e2dd5fa277c","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-4564625e-e4cc-410f-99e0-449a8e9bc1a0","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}] [{"selector":"#anim-1188fbf4-4073-4483-8ab3-d05336525456","keyframes":[{"transform":"rotate(-540deg) scale(0.1)","opacity":0},{"transform":"none","opacity":1}],"delay":0,"duration":1000,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both","iterations":1}] [{"selector":"#anim-6d36ae58-de42-48b4-8f17-f7106832e532","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-29ec63eb-803b-4acc-a13c-bbec94be4eca","keyframes":{"transform":["translate3d(0px, 332.88870%, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] Image Credit- Google Images

Creating a robots.txt file is a simple process that only requires a text editor and a basic understanding of how to use the robots.txt format.

[{"selector":"#anim-8fd0eec0-7b46-4acd-8e46-f05bb299038b [data-leaf-element=\"true\"]","keyframes":{"transform":["translate(0%, 0%) scale(1.5)","translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-be522638-c2b2-411d-bbb6-090a12550f3d","keyframes":{"transform":["translate3d(-115.97444%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-3c87390b-1499-4523-b9a0-5a5cac5c450e","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-e06a480c-c547-4b70-a75b-e55372d65cfb","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}] [{"selector":"#anim-9a5cf6c2-7392-4d5d-ad51-e79160acfd1f","keyframes":[{"transform":"rotate(-540deg) scale(0.1)","opacity":0},{"transform":"none","opacity":1}],"delay":0,"duration":1000,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both","iterations":1}] [{"selector":"#anim-b3a92ddb-2118-4c61-b24a-a3f605433639","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-845874c6-40f9-42b3-80c9-f58dd47975b1","keyframes":{"transform":["translate3d(0px, 332.88870%, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] Image Credit- Google Images

What is Googlebot and XML sitemap

[{"selector":"#anim-71cfc760-42c9-4375-930b-9fcbc88fe797","keyframes":[{"transform":"rotate(-540deg) scale(0.1)","opacity":0},{"transform":"none","opacity":1}],"delay":0,"duration":1000,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both","iterations":1}] [{"selector":"#anim-5dedff40-e958-4e4e-ab4c-6c1e9a13c9b8","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-0de3ae6c-5c82-4ee5-a2e4-89a236af1cf3","keyframes":{"transform":["translate3d(0px, 2660.88730%, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.2, 0.6, 0.0, 1)","fill":"both"}] [{"selector":"#anim-4e8c9338-42d4-4ab7-ab23-78728e4204f9","keyframes":{"transform":["translate3d(-115.97444%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-60c52ae3-608e-43cc-87e3-4deb5a552a95","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-65330d73-edeb-49e1-87e8-29ea7f6a6dd2","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}] Image Credit- Google Images Read More