4 Common Server Header Response Codes and 4 Problems to Fix

In this post, I’m going to help you identify if you have server header response code problems, and in subsequent posts, teach you how to fix them with my best attempt to use non-technical lingo. When optimizing an existing website, there are a couple of problems where a site’s server sends the wrong response codes (also referred to as http header status) to the crawler (e.g. Googlebot) when it tries to get the url.  Without getting into the nitty-gritty technical stuff, I’m going to walk you through 4 easy checks you can do, and how you can fix them.  Before we start, you need a basic understanding of the 3 server response codes that are most often referred to in SEO.

4 Most Common Server Response Codes

1. 200 – Content found, page is ok
2. 404 – Tells the crawler that the page is not found
3. 301 – Tells the crawler that the page has been permanently moved to a new location
4. 302 – Tells the crawler that the page has been temporarily moved to a different location

4 Checks You Can Perform to Make Sure You’re Sending the Crawlers the Right Message

1. Does the Non-WWW Domain Address 301 Redirect to the WWW URL?

This is important because you want the search engines to know which url you want to use and more importantly, which url to consolidate for your link power.  Also, without doing this, it’s possible for the engines to include both www and non-www urls in the index, which is not ideal.To check this, use a handy server header checker tool.  There’s a bunch of them available online for free.  I like the Server Header Checker at SEOBook, because it allows you to choose the user-agent (e.g. who is trying to see your page – googlebot, firefox, etc.)Enter your domain into the tool without the www (for example: clairefy.com).  Choose your agent (or leave it the default).  Run the checker.  You want the response code to come back like this:

SERVER RESPONSE: HTTP/1.1 301 Moved Permanently
Redirecting to: http://www.yourdomain.com/

If it does, you’re in the clear. If it doesn’t, try the checker again, but this time, do it with the www (www.clairefy.com). What you want to happen this time, is that the checker tells you your response is Redirecting permanently (301) to: http://yourdomain.com.  If that checks out, you’re good to go to #2.

If your response code is 200 or 302: Follow instructions in this post: Redirecting non-www to www using htaccess.

2. Are There ANY 302 Redirects on Your Site?

The problem with 302 redirects (temporary) is that the crawler holds onto both the old url and the new url.  The biggest concern with this for SEO is that you will be losing the link power from any links pointing to the old url.  If you put a 301 redirect instead of a 302, you will be transferring all the link power pointing to that old url to the new url, helping your organic rankings!How to Identify 302 Redirects:
The easiest way to find 302 redirects is to use a tool that emulates a search engine crawler.  The best free one for a PC is Xenu Link Sleuth, and for Mac, I use Integrity. First, run the crawler from your root domain.  For Xenu, you’ll need to tell it to treat redirects as errors.  When the crawler is done, export the results and look for any 302s.

If it doesn’t find any, you’re halfway to being cleared for this problem.  If it does find temporary redirects, I’ll tell you how to fix it in a later post.

Next, you need to check and see if the engines have any urls in their index that are redirecting with a 302.  Just because your site is clean, doesn’t mean that there’s an old file out there not taking advantage of the power of a 301 redirect.

To check this, I like to use another tool from Aaron Wall at SEOBook, the Firefox plugin – SEO for Firefox.  Once you’ve installed the plugin, go to Google and set SEO for Firefox to show 100 results.  Then, execute a search using site:www.yourdomain.com – which will return all of the results Google has for your site in its index.  Now, you can export your results to .csv (you may have to go through multiple pages of results if you have a large site).

Next, open the .csv in excel and copy the urls into a text file (.txt). Finally, using your crawler tool, crawl the list of urls.  In Xenu, you can uncheck the “crawl external urls” option, which will make it crawl through your file more quickly.

If you don’t find any 302s, you’re cleared to go on to #3.

3. Do you have any 301 Redirects When You Crawl Your Site from the Domain?

This isn’t a HUGE problem, because at least you’re link juice is getting transferred to the right place; however, when a crawler encounters a 301, it is highly possible they will not follow it and will stop crawling.  Typically, when they do this, they come back later, but why would you want to delay crawl?So, when you crawled your domain in the first part of issue #2 above, you should be able to easily see if you have any 301 permanent redirects on your site.

If you do, the fix is easy… go to the place on your site and change the url to the new location.

4. Are your 404 pages sending a 404 response code?

It’s a good idea to create a pretty 404 page (or funny 404 page, if you rather), but you better be sure it’s actually telling the engines that it’s a 404!  If not, it’s possible for your pretty 404 page to be indexed under any number of garbley-gook urls.To check this, go back to your server header checker.  In the tool, enter your url with garbley-gook in the string.  (e.g. http://www.yourdomain.com/garbley-gook/).

Does it return a 404? If so, you’re in good shape.  If not, you need to talk to your host to get it fixed.

Other common issues with 404s: It’s also wise to check other url rules to make sure those are sending the right response.  For instance, I’ve often encountered sites using http://www.yourdomain.com/category/post-name/ and when you take out /category/ or mis-type it, the url still works (response code 200) and does not throw a 404.