One of the most frustrating issues that can happen on the web is when your website does not get an index in Google search. Recently a client came to us with this exact problem. When looking up the domain on Google it was indexed but it did not show any information about the business, just the raw URL.
The meta information(or description about the website) was missing so all there was was the domain in Google search. The client was wondering how to fix this and was quite perplexed. On further inspection, I discovered that the robots.txt file (what tells the Google bots what to crawl and what not to crawl) was actually blocking indexing of all content.
This is rather easy to address, however you need to know how to set it up, so here is an example of what the file looked like before:
User-Agent: *
Disallow: (by not indicating any pages to allow and leaving only the Disallow option, it was causing no pages to get indexed)
and here’s an example of how it was edited to allow indexing:
User-Agent: *
Disallow:/wp-admin/
Allow: /
Here I will break down each part of the robots.txt so you can understand what they do:
User-Agent: tells which search engines can index the site (a * symbol lets all search engines index)
Disallow: tells which pages should not be indexed. In this case I am telling the bots to not index the admin of WordPress
Allow: tells which pages to index. The / tells the bots to index all pages in the root directory or the entire website, basically.
Having the correct robots.txt file is really important for getting found on the web because if the Google search bots aren’t allowed to crawl the site, there’s no way you’ll ever get indexed properly on Google search and get a lot less organic traffic.
An additional issue that was on this site is a missing title tag. We added a title tag to the site with the exact name of the business by going into the header.php file and adding a description of the site wrapped in < title> tags in the head area.
With not being able to index a site due to the robots.txt file and missing a title tag this site had no chance at all of ever competing in Google search. These are some things that you really want to think about and check on if you’re having issues when building a new website.
It’s extremely crucial to make sure that you have the proper ability to have Google and other search engines index your site and that you’ve provided all the necessary information for them to find out who you are what your website is about.
Have questions about how to get your robots.txt file or your title tag figured out correctly? Hit us up here and we’ll be glad to help you out!
Read next: How to Make Google Love Your Business
Leave a Reply
Want to join the discussion?Feel free to contribute!
Leave a Reply
Customer Support
____________________________________________ Looking for SEO in Billings, MT? Ask us why we are the secret ingredient to your business ranking #1
SEARCH OPTIMIZATION
____________________________________________ Looking for SEO in Billings, MT? Ask us why we are the secret ingredient to your business ranking #1
SEARCH OPTIMIZATION
What causes the robots.txt file to block indexing on Google?
It’s a command line that has been added to the robots.txt file that instructs the robots to not crawl or index the website. This should be removed in all production ready websites.
HI Jophiel, I think I need your help on my indexing issue, we used to show on Google and now… nothing. I am too scared to mess with the files myself.
Certainly, Lori! How regularly have you been updating the content on the website? Is it the same content on the site since you launched? That is probably the main reason for the decline. Drop the link and I’ll take a look though!