You could be HIDING your site from the Search Engines!
HIDING? Yes, you heard me right, I said hiding from search engines! Let's take a look at a few of the ways you might
do that without meaning to do so. Hide and seek! Peekaboo!
Secure Server
Search engines do have the ability to spider secure server hosted pages, but often these pages require either that a
visitor fill out a form or log-in with a password and user name before being allowed past a certain point. If any
page requires filling out of forms or passwords to reach, search engine robots will simply leave. They can't log in
because they can't fill out forms, leave email addresses or enter passwords.
I was contacted by a webmaster for a 4500 page ecommerce web site. He wondered why search engines were ignoring such
a large site. I asked the URL of the site and visited the home page. I noted that upon loading, there was an immediate
passing of the URL http://anybusiness.com site to a secure httpS://anybusiness.com page. This has two
immediate faults that may be a problem - the forwarding method and different server. If the instant forward is by
javascript, bad news.
First, search engines often either penalize or downgrade sites that use immediate URL forwarding, especially from a
home page. URL forwarding suggests doorway pages (a search engine no-no) or affiliate URL's forwarding to an affiliate
program site, or the worst of all scenarios, cloaking software on your server. You may not be doing any of these
things, but the robots don't know, don't care, and don't index your site, plain and simple.
Secondly, secure servers are very often a separate web site, meaning that the secure server is actually a different
machine and is an entirely different site from the non-secure server site unless your site is hosted on a dedicated
server on it's own IP address, security certificate at the same domain. This can happen when secure shopping carts
are hosted by a third party host so that a small ecommerce site needn't purchase a security certificate or set up
complex shopping carts.
For example, if your shopping cart is hosted by Yahoo stores or other application service providers (ASP's), pages
hosted in the shopping cart don't reside on your domain and can't be recognized as pages on YOUR site unless you also
host your domain with the same company. Unfortunately, many shopping cart ASP's use dynamic IP addresses (IP address
is different each time you visit) and use database generated dynamic pages.
The process of serving dynamic pages is not the problem. The problem is simply that the URL of those pages contains
several characters that either stop or severely curtail search engine spiders. Question marks (?) are the
biggest culprit, followed by ampersands (&), equal signs (=) percent symbols (%) and plus
signs (+) in the URL's of dynamic pages.
These symbols serve as alarm bells to the spiders and either turn them away entirely or dramatically slow the
indexing of your pages. This is stated simply in the Google
Information for Webmasters page.
"Reasons your site may not be included.
"Your pages are dynamically generated. We are able to index
dynamically generated pages. However, because our web crawler
can easily overwhelm and crash sites serving dynamic content,
we limit the amount of dynamic pages we index."
Just because your site is dynamically generated, creating long URL's full of question marks, equal signs and
ampersands like
www.domain.com/category.asp?ct=this+%28that+other%29&l=thing
doesn't mean you are in search engine limbo. There are simple solutions available for your webmaster. Here are a
couple of articles explaining an elegant solution called "mod_rewrite".
You can read about that technique if technically inclined:
http://alistapart.com/articles/urls/
http://alistapart.com/articles/succeed/
This technique is simply creating a set of instructions for your web server to present URL's in a different form that
replaces those "bad" question marks and ampersands with slash marks (/) instead. The method will require that
your webmaster is a bit more technically savvy than most home business CEO's who created their own web site. Some
hosts will help here by simply turning on the "mod_rewrite" for shared hosting clients.
Don't play hide and seek with the search engines! Tell them EXACTLY where to find every page on your site and
if there's any question that they will find every page on your site, give them a map.
A Site Map
Hard code those dynamic URL's for most categories within the categories of different sections of your web site into
your comprehensive site map. As long as those dynamic links (even those that include ?=+%& symbols) are hard coded
into a site map, the spiders will follow them. Clearly those 4500 pages mentioned earlier would be too much for a site
map listing. But the main category pages could be provided for the engines.
I visited the site map page of the webmaster mentioned above and saw fourteen pages listed on the site map. That
explains why they have fourteen pages, not 4500, indexed by Google.
How to find out how many pages of your site are indexed? Go to Google search and type "allinurl:www.domain.com"
without the quotes. Replace "domain" in the above example with your own domain name. This query operator will
return a list of every page of your site. Look in the blue bar across the top of the Google results page and you'll
see the number of pages indexed at your site!
That should do it. Get indexed and stop playing hide-and-seek!
About The Author
Mike Banks Valentine is a Search Engine Optimization Specialist practicing ethical SEO for Online businesses
SEOptimism.com. Learn SEO
http://SearchEngineOptimism.com/SEO_Tutorial/. Then Take a
Search Engine Quiz to test your Skill Level.
