Initially, the World Wide Web was comprised predominantly of static web sites. Each URL within a web site pointed to an actual physical file located on a web server’s file system. Therefore, a search engine spider had very little to worry about. The spider would crawl throughout the web site and index every URL in a relatively straightforward manner. Problems such as duplicate content and spider traps did not typically exist.
Today, dynamic web sites dominate the World Wide Web landscape. Unfortunately, they frequently present problems when one looks at their URLs from a search engine’s perspective — especially with regard to spidering. For example, many dynamic web sites employ query string parameters that generate different URLs that host very similar or identical content. This is interpreted as duplicate content by search engines, and this can get the pages penalized. The use of many URL parameters may also result in spider traps (http://en.wikipedia.org/wiki/Spider_trap), or in linking structures that are hard to follow by a search engine. Needless to say, both scenarios reduce the web site’s ranking performance with the search engines.
