Search Engines – How They Work
The term search engine is often loosely-applied to describe both
spider- or crawler-based Search Engines and Internet Directories that are created
with human intervention. Directories are not true "search engines," but
they accomplish the same thing. However, the user really doesn't care how they
achieve the results as long as they are the desired ones sought. Although both
Search Engines and Directories set out to accomplish the same goal, they work
differently.
Spiders, Crawlers and Other Creepy Things
True Search Engines, such as HotBot, use spidering or crawling technology
to automatically create their listings. They crawl or spider the web, looking
for what users search upon.
As you change your web pages and add new content, Search Engines will eventually
find those changes. This can affect your ranking. Page titles and actual content
become increasingly important.
The spider visits and reads the web page, then follows links to other pages
within the site. Then the spider returns to the site according to Search Engines
settings and looks for changes. The spidered information must then go to what
is known as the "index" or "catalog."
Search Engine software (and their algorithms) is the final, crucial part.
This is how the program(s) searches through content in the Index to find, match
and rank users' search.
The following chart shows how different engines rank web pages and who uses
what to include and rank those pages.
| Spiders Web Pages Submitted or Not |
| Does Not Follow Frame Links |
| Does Not Follow Image Maps |
| Does Not Support Robots Meta Tags |
| Link Popularity Helps Ranking |
| Learns Page Change Frequency |
| Allows Paid Inclusion or Pay-per-Click |
| Does Not Support Meta Tag Description |
| Does Not Support Meta Tag Keywords |
| Does Not Support Alt Text |
| Direct Hits Assist Rankings |
| Penalizes Site with Meta Refresh Tags as Spamming |
| Penalizes Site with Invisible Text as Spamming |
| Penalizes Site with Tiny Text as Spamming |
| NOTE |
• Inktomi results form portions of AOL Search, HotBot & MSN
• Excite covers portions of Excite-owned Magellan and WebCrawler
• FAST Search covers crawler data used by Lycos |
|
Internet Directories
Internet Directories, such as Yahoo, depend on human intervention for its
listings. You are usually asked to select the proper category for your site (can
be difficult and time-consuming) and then submit a short description of your
site. Sometimes it is left up to the editor of that particular category to describe
your site.
These Directories often require payment in order to review your site within
1-2 weeks. Some require payment in order to be reviewed at all. Still others
offer free reviews but with no time guarantee. It can take as long as 3 months
to get listed in these Directories – if you get listed at all! Keep in
mind that even if you pay to have your site reviewed, that does not guarantee
that they will actually list your site.
Things that help improve your ranking on a true Search Engine are not necessarily
the things that will improve your ranking on a Directory. Here is where good
content, fast-loading, user- and browser-friendly pages, properly optimized come
in. You must remember that a human being is reviewing your site – and thousands
of others.
Major Players |
(in alphabetical order)
|
| AltaVista |
Definitely one of the major concerns when getting your site listed. It has
comprehensive, wide-ranging coverage and search options that make it popular
among users. In addition to spider-based matches, it allows you to search specifically
for news, shopping, and multimedia. It also gets results from LookSmart. |
| AOL
Search |
The external version gets its listings from the Open Directory, Inktomi and
spider results as backup to the directory information. Members are allowed to
search AOL content as well. |
| Ask
Jeeves |
A human-powered search service that tries to answer exact questions from
the user and direct them to the proper page(s). |
| Excite |
Uses spidering technology for it's web page index and gives access to directory
results from LookSmart. |
| Fast
Search/All The Web |
One of the largest indexes on the Internet. Also known as All The Web, this
is a showcase for FAST Search's technologies. |
| Google |
Link popularity is the key to Google. Google uses its own massive index and
also provides results from Yahoo and Netscape. |
| GoTo |
An increasingly popular pay-per-click directory. Web page owners can bid
on their ranking for particular search terms. |
| HotBot |
With many advanced searching features, HotBot is popular with researchers.
First page of results usually come from Direct Hit and Inktomi. It's directory
information comes from the Open Directory project. |
| IWon |
Gives away daily, weekly and monthly prizes in an innovative marketing scheme.
It's directory comes directly from Inktomi. |
| LookSmart |
A directory of web pages reviewed by humans. It also provides Directory results
to MSN Search, Excite and many others. |
| Lycos |
Main listings come from the Open Directory project, and then secondary results
come from the FAST Search engine and Direct Hit. |
| MSN
Search |
This service is powered by LookSmart's directory of websites and gets its
secondary results from Inktomi. Some sites are also pulled from RealNames and
Direct Hit. |
| Netscape
Search |
Netscape Search's web page matches come mainly from it's own index, the Open
Directory and also gets some results from Google. |
| Northern
Light |
Another researcher's favorite because of its large index and ability to gather
documents by topic. It also has special collection documents from thousands of
sources, including news wires, magazines and databases. While these documents
can be searched, there is a charge of up to $4 to view them. |
| Open
Directory |
Uses volunteers to review and edit its catalog. Netscape, Lycos and AOL Search
all utilize the Open Directory's index. |
| Yahoo |
The oldest major website Directory on the Internet and it is the second most
popular search, next to Google. It has the largest human-compiled index of the
web with over 1 million sites listed. If a search fails to find a match within
it's own listings, it supplements its results with those from Google. Google
matches are also shown after all Yahoo matches. |
Using Both Technologies
With the vast number of websites being added to the Internet daily, some Directories
have combined spidering technology. They usually organize them in a way that
presents the Directory listings first and then presents the spidered results.
We Don't Know it All but...
| We'll do our homework and give you our best effort. CONTACT
US to ascertain if we're right for you! |
