Welcome to the Treehouse Community

Want to collaborate on code errors? Have bugs you need feedback on? Looking for an extra set of eyes on your latest project? Get support with fellow developers, designers, and programmers of all backgrounds and skill levels here with the Treehouse Community! While you're at it, check out some resources Treehouse students have shared here.

Looking to learn something new?

Treehouse offers a seven day free trial for new students. Get access to thousands of hours of content and join thousands of Treehouse students and alumni in the community today.

Start your free trial

WordPress

Hunter G
Hunter G
6,612 Points
Posted by Hunter G
Hunter G
6,612 Points

Cannot Change robots.txt file in WordPress!!!

Hi everybody,

So I recently started helping out my uncle doing SEO on his Auto Detailing website. It was previously built a couple years ago by somebody he knew and now he wants to me to do the web mastering. I am fluent in HTML / CSS and learning .php as I go but I have never worked on WordPress before. . The problem I am having is that the site is not being crawled by any search engines because of the Robots.txt file. I have tried every way possible how to change it. Under Settings > Reading I have the "Discourage search engines from indexing this site" unchecked (i've tried checking it, saving, then unchecking again to see if it helps), i've downloaded the plug-in "Virtual Robots.txt" and typed out the URLs manually to be allowed in the content area of the plug-in and it STILL will not change the robots.txt file! There is a disclaimer at the top of the plug-in that says "If your robots.txt file doesn't match what is shown below, you may have a physical file that is being displayed instead." which may be the problem, but I have no clue where to find the physical file. I am currently going through WordPress classes on Treehouse to see if any of the teachers shed some light on this problem but it seems like the most common way people change this file is through Settings > Readings which isn't helping in my case : / Does anybody have a clue?

Please help! Thanks

  • Hunter

1 Answer

Michael Curtis
Michael Curtis
6,684 Points
Michael Curtis
Michael Curtis
6,684 Points

Hi, The robots.txt file is usually in the main directory on your server. To manually change it you can log into your webspace via ftp download it, edit it and then re-upload. if you have a robots.txt you can see what it's contents are by directing your browser there i.e www.example.com/robots.txt

I'm assuming google is the main search engine you want to crawl your site. You can create an account with them and provide details about you site that may help.