Chapter One
Surveying the Search Engine Landscape
In This Chapter
* Discovering where people search
* Understanding the difference between search sites and search systems
* Distilling thousands of search sites down to four search systems
* Understanding how search engines work
* Gathering tools and basic knowledge
You''ve got a problem. You want people to visit your Web site; that''s the purpose, after all - to bring people to your site to buy your product, or find out about your service, or hear about the cause you support, or for whatever other purpose you''ve built the site. So you''ve decided you need to get traffic from the search engines - not an unreasonable conclusion, as you find out in this chapter. But there are so many search engines! You have the obvious ones - Google, AOL, Yahoo!, and MSN - but you''ve probably also heard of others: HotBot, Dogpile, Ask.com, Netscape, EarthLink - even Amazon provides its A9 Web search box on many pages. There''s also Lycos, InfoSpace, Mamma.com, and WebCrawler.
To top it all off, you''ve seen advertising asserting that for only $49.95 (or $19.95, or $99.95, or whatever sum seems to make sense to the advertiser), you, too, can have your Web site listed in hundreds, nay, thousands of search engines. You may have even used some of these services, only to discover that the flood of traffic you were promised turns up missing.
Well, I''ve got some good news. You can forget almost all the names I just listed - well, at least you can after you read this chapter. The point of this chapter is to take a complicated landscape of thousands of search sites and whittle it down into the small group of search systems that really matter. (Search sites? Search systems? Don''t worry; I explain the distinction in a moment.)
If you really want to, you can jump to the "Where Do People Search" section (near the end of the chapter) to see the list of search systems you need to worry about and ignore the details. But I''ve found that when I give this list to someone, he or she looks at me like I''m crazy because they know that some popular search sites aren''t on the list. This chapter explains why.
Investigating Search Engines and Directories
The term search engine has become the predominant term for search system or search site, but before reading any farther, you need to understand the different types of search, um, thingies that you''re going to run across. Basically, you need to know about four thingies.
Search indexes or search engines
REMEMBER
Search indexes or engines are the predominant type of search tools you''ll run across. Originally, the term search engine referred to some kind of search index, a huge database containing information from individual Web sites.
TECHNICAL STUFF
Large search-index companies own thousands of computers that use software known as spiders or robots (or just plain bots) to grab Web pages and read the information stored in them. These systems don''t always grab all the information on each page or all the pages in a Web site, but they grab a significant amount of information and use complex algorithms - calculations based on complicated formulae - to index that information. Google, shown in Figure 1-1, is the world''s most popular search engine, closely followed by Yahoo! and MSN Live Search.
Search directories
A search directory is a categorized collection of information about Web sites instead of containing information from Web pages.
The most significant search directories are owned by Yahoo! (dir.yahoo.com) and the Open Directory Project (www.dmoz.org). (You can see an example of Open Directory Project information, displayed in Google - dir.google. com - in Figure 1-2.) Directory companies don''t use spiders or bots to download and index pages on the Web sites in the directory; rather, for each Web site, the directory contains information, such as a title and description, submitted by the site owner. The two most important directories, Yahoo! and Open Directory, have staff members who examine all the sites in the directory to make sure they''re placed into the correct categories and meet certain quality criteria. Smaller directories often accept sites based on the owners'' submission, with little verification.
TIP
Here''s how to see the difference between Yahoo!''s search results and the Yahoo! directory:
1. Go to www.yahoo.com.
2. Type a word into the Search box.
3. Click the Search button.
The list of Web sites that appears is called the Yahoo! Search results.
4. Look for Directory above the Search box. Above the Search box you''ll see either a Directory link or a More link that opens a drop-down menu with the Directory option inside the menu; either way, click Directory and you''ll end up in Yahoo! directory. (You can also go directly to the directory by using dir.yahoo.com.)
Non-spidered indexes
I wasn''t sure what to call these things, so I made up a name: non-spidered indexes. A number of small indexes, less important than major indexes (such as Google), don''t use spiders to examine the full contents of each page in the index. Rather, the index contains background information about each page, such as titles, descriptions, and keywords. In some cases, this information comes from the meta tags pulled off the pages in the index. (I tell you about meta tags in Chapter 2.) In other cases, the person who enters the site into the index provides this information. A number of the smaller systems discussed in Chapter 12 are of this type.
Pay-per-click systems
Some systems provide pay-per-click listings. Advertisers place small ads into the systems, and when users perform their searches, the results contain some of these sponsored listings, typically above and to the right of the free listings. Pay-per-click systems are discussed in more detail in Chapter 18.
Keeping the terms straight
TIP
Here are a few additional terms that you''ll see scattered throughout the book:
Search site: This Web site lets you search through some kind of index or directory of Web sites, or perhaps both an index and directory. (In some cases, search sites known as meta indexes allow you to search through multiple indices.) Google.com, AOL.com, and EarthLink.com are all search sites. Dogpile.com and Mamma.com are meta-index search sites.
Search system: This organization possesses a combination of software, hardware, and people that index or categorize Web sites - they build the index or directory you search through at a search site. The distinction is important because a search site might not actually own a search index or directory. For instance, Google is a search system - it displays results from the index that it creates for itself - but AOL.com and EarthLink.com aren''t. In fact, if you search at AOL.com or EarthLink.com you actually get Google search results. Google and the Open Directory Project provide search results to hundreds of search sites. In fact, most of the world''s search sites get their search results from elsewhere; see Figure 1-3.
Search term: This is the word, or words, that someone types into a search engine when looking for information.
Search results: Results are the information (the results of your search term) returned to you when you go to a search site and search for something. As just explained, in many cases the search results you see don''t come from the search site you''re using, but from some other search system.
Natural search results: A link to a Web page can appear on a search results page two ways: The search engine may place it on the page because the site owner paid to be there (pay-per-click ads), or it may pull the page from its index because it thinks the page matches the search term well. These free placements are often known as natural search results; you''ll also hear the term organic search results and sometimes even algorithmic search results.
Search engine optimization (SEO): Search engine optimization (also known as SEO) refers to "optimizing" Web sites and Web pages to rank well in the search engines - the subject of this book, of course.
Why bother with search engines?
Why bother using search engines for your marketing? Because search engines represent the single most important source of new Web site visitors.
You may have heard that most Web site visits begin at a search engine. Well, this isn''t true, though many people continue to use these outdated statistics because they sound good - "80 percent of all Web site visitors reach the site through a search engine," for instance. However, in 2003, that claim was finally put to rest. The number of search-originated site visits dropped below the 50 percent mark. Most Web site visitors reach their destinations by either typing a URL - a Web address - into their browsers and going there directly or by clicking a link on another site that takes them there. Most visitors don''t reach their destinations by starting at the search engines.
However, search engines are still extremely important for a number of reasons:
The proportion of visits originating at search engines is still significant. Sure, it''s not 80 percent, but with billions of searches each month, it''s still a lot of traffic.
According to a report by comScore, published in March of 2008, Internet users in the United States carry out almost 10 billion searches a month at the major search engines.
comScore also reported that in August of 2007, 750 million people around the world carried out a total of 61 billion searches on the world''s top search sites; that''s almost three searches every day by 10 percent of the world''s population.
Of the visits that don''t originate at a search engine, a large proportion are revisits - people who know exactly where they want to go. This isn''t new business; it''s repeat business. Most new visits come through the search engines - that is, search engines are the single most important source of new visitors to Web sites.
It''s also been well established for a number of years that most people researching a purchase begin their research at the search engines.
Search engines represent an inexpensive way to reach people. Generally, you get more bang for your buck going after free search engine traffic than almost any other form of advertising or marketing.
Here''s an example. One client of mine, selling construction equipment to the tune of $10,000 a month, rebuilt his site and began a combined natural-search and paid-search campaign, boosting sales to $700,000 a month in less than two years. It''s hard to imagine how he could have grown his company, with relatively little investment, so quickly without the search engines!
Where Do People Search?
You can search for Web sites at many places. Literally thousands of sites, in fact, provide the ability to search the Web. (What you may not realize, however, is that many sites search only a small subset of the World Wide Web.)
However, most searches are carried out at a small number of search sites. How do the world''s most popular search sites rank? That depends on how you measure popularity:
Percentage of site visitors (audience reach)
Total number of visitors
Total number of searches carried out at a site
Total number of hours visitors spend searching at the site
Each measurement provides a slightly different ranking. Although all provide a similar picture with the same sites generally appearing on the list, some are in slightly different positions.
The following list runs down the world''s most popular search sites, based on one month of searches during August of 2007 - almost 8 billion searches - according to a Nielsen/NetRatings study. These statistics are for U.S. Internet users:
Google.com 53.6% Yahoo.com 19.9% MSN.com 12.9% AOL.com 5.6% Ask.com 1.7% My Web Search 0.9% BellSouth Search 0.5% Comcast Search 0.4% My Way 0.4% SBC Yellow Pages Search 0.4%
Remember: This is a list of search sites, not search systems. In some cases, the sites own their own systems. Google provides its own search results, but AOL doesn''t. (AOL gets its results from Google.) My Web Search is a meta-search engine getting results from Google, Yahoo!, and Ask.com.
The fact that some sites get results from other search systems means two things:
The numbers in the preceding list are somewhat misleading. They suggest that Google has around 54 percent of all searches. But Google also feeds AOL its results - add AOL''s searches to Google''s, and you have 59.6 percent of all searches. Additionally, Google feeds Comcast and BellSouth (another 0.9 percent combined, according to NetRatings). Furthermore, My Web Search is a meta-search engine; therefore, if you search at My Web Search, you see results from Google, Yahoo!, MSN Live Search, and Ask.
You can ignore some of these systems. At present, and for the foreseeable future, you don''t need to worry about AOL.com. Even though it''s one of the world''s top search sites, you can forget about it. Sure, keep it in the back of your mind, but as long as you remember that Google feeds AOL, you need to worry about Google only.
Now reexamine the preceding list of the world''s most important search sites and see what you can remove to get closer to a list of sites you care about. Check out Table 1-1 for the details.
Based on the information in Table 1-1, you can whittle down your list of systems to four: Google, Yahoo!, MSN Live Search, and Ask. The top three search systems are all very important, with a small follower, Ask, which provides results to many smaller search sites. There''s one more system I want to add to these four systems, though. Very few people search at the Open Directory Project (www.dmoz.org). However, this directory system feeds data to hundreds of search sites, including Google and AOL.
REMEMBER
To summarize, five important systems are left:
Google
Yahoo!
MSN Live Search
Ask
Open Directory Project
That''s not so bad, is it? You''ve just gone from thousands of sites to five. Note, by the way, that the top three positions may shift around a little. Google had already lost a large proportion of its share (when I wrote the first edition of this book, Google had around three-quarters of the market- now it''s probably a little over one-half), and then regained about 8 percent of the market between the second and third editions of this book.
Of course we may soon see some real changes in this profile. Microsoft is trying to buy Yahoo! If it succeeds, it seems likely that the two search indexes will be merged at some point. (After all, the main point of the proposed merger is to share - and thus cut - costs.)
Then there''s Ask.com, which recently announced that it was no longer going head to head with Google - years of trying to gain market share got them nowhere - and was planning to focus on the search needs of married women. Perhaps more significantly, there are rumors that Ask may stop managing its own search index - and use Google''s! Ask already uses Google''s Pay Per Click ads (see Chapter 18), so extending its partnership with Google wouldn''t be such a surprise. So, who knows, we may soon see just two major search indexes: the Yahoo!/MSN index and the Google/Ask/and-everything-else index.
(Continues...)
Excerpted from Search Engine Optimization For Dummiesby Peter Kent Copyright © 2008 by Peter Kent. Excerpted by permission.
All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.
