# See https://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file User-agent: * Disallow: /admin/ Disallow: /job_search/toggle_job_tag Disallow: /job_search/recommended_jobs Disallow: /job_search/autocomplete_job_titles Disallow: /job_search/autocomplete_location