Search Engine Ranking

Preface

Find out how search engines think and use the knowledge to start increasing your rankings in search results...

1 How to get High Rankings
2 What is a Search Engine?
3 How is the index created?
4 Getting listed
5 How to get a high ranking
6 Relevancy
7 Off the page factors
8 What needs to be done?

How to get High Rankings

An important aspect of Search Engine Optimisation (SEO) and Search Engine Marketing (SEM), though by no means the only one, is the business of getting ranked highly in search engine results. Amongst the other factors are "clickthrough" - getting the searcher to actually visit the page that is listed, keeping the visitor's interest once they have clicked through and "conversion" - having the visitor take the action you desire, usually a sale.

What is a Search Engine?

Internet search engines are special sites on the web that are designed to help people find information stored on other websites. There are differences in the ways various search engines work, but they all perform three basic tasks:

They search the internet – or selected parts of the internet – based on important words - (keywords).
They keep an index (a kind of catalogue) of the words they find, and where they find them.
They allow users to look for words or combinations of words found in that index and produce "ranked" lists of the sites or pages where they can be found.

Early search engines held an index of a few hundred thousand pages and documents, and received maybe one or two thousand inquiries each day. Today, a major search engine will index hundreds of millions of pages, and respond to tens of millions of queries per day.

How is the index created?

A search engine index is created in one of two ways, by machines or by humans. The indices that are machine created are built up by "spiders" or "crawlers". These are special programs, sometimes knows as "bots" - short for robots, which search the web for pages and index them according to an "algorithm" or set of rules, which vary from one search engine to another.

Crawlers work their way through the web using hyperlinks, which link the pages on a website together and link different websites together. They will start from the existing index and follow all the links, finding new and updated pages as they go.

The indices created by humans are called Directories. Directories are controlled by human editors who receive applications for inclusion and vet them before deciding whether or not they are truly relevant.

Whichever method is used to create the index, whenever you search for something on the web one or other, in some cases both, of these types of indices will be searched to give you a list of results in order of relevance. In "SEO speak" they will be "ranked".

For the purpose of getting your website onto these lists as near to the top as possible what matters are the processes by which the web pages are listed and ranked in the index.

Getting listed

Getting listed on a Directory is simply a matter of submitting your site to relevant directories, choosing a category to be listed under and waiting for a decision. That decision will based on the editors opinion of the relevancy of your site to the category you have chosen. This process can take several months but there is very little that can be done to speed this up. You just have to be patient. Being accepted is all about relevancy, which will be dealt with later. Once you are listed, however, it's unlikely that your listing will be removed, which is good. But you can usually only list in one category and it can be very difficult to get that changed. It is therefore very important to get it right first time...

Each Directory has its own set of rules for submission and they need to be followed if you want to succeed in getting listed in the category of your choice and with your chosen description displayed. Working through this maze and jumping through all the hoops can be complex and time consuming. It's well worth getting professional help if you are unsure. If you get it wrong it can takes months to sort out, if at all.

There are many directories on the web, some cater for special niches and others are more general. Some, commercial ones, will only list your site for a fee, some are free and others have free and paid submissions. The paid submissions will either speed up the process; give you a higher ranking, or both.

Getting listed by a crawler based index is, in one sense, much simpler. All you have to do is publish your new site and the major search engine web crawlers will find and index it - eventually. This can take up to two months. There are submission methods for the three main crawler based search engines, Google, Yahoo and MSN. But submission is a complete waste of time. It won't get you listed any quicker, so don't bother.

If you do submit your URL to a search engine, its crawler will visit the site but, unless there is at least one link to it from another page already on the index you won't be indexed at all or will be ranked very low. If there is an indexed page linking to a page on your site it will be crawled anyway, whether you submit or not. This is a major reason why it is important to get listed on the Directories.

Directories are one of the best sources of links and can have a significant effect on your ranking in the crawler based search engines. See the section on Off the page factors, below.

On the subject of submission to search engines, there a lot of services offered on the web, which claim to "Submit your site to over a 1000 search engines for as little as £x per month!" etc. These are scams!

Firstly, there are only about 7 engines with indices of any size and of these the Big 3 account for over 90% of all searches. What's more Google has been estimated to have somewhere between 70% and over 80% of all searches. So you can get listed where over 90% of searchers will find you for free. Further to this, many of these so called "1000 search engines" are in fact "free for all" pages on sites, which are feeding grounds for email harvesting bots. Even some of the submission services are actually fronts for email marketeers. Paying for these services is just paying for more spam! They should be avoided at all costs.

How to get a high ranking

For searches on a directory the ranking of results will be predetermined by the editor. The only things that will influence him or her is how well written and relevant your site is. For searches using a crawler based search engine, however, every search is potentially different. The same search words entered to Google and to MSN will give different results because they use different algorithms to rank entries in their indices.

It is even quite likely that a search done on the same engine at different times could give different results because the indices are being constantly updated. In the majority of cases, such searches are remarkably accurate. They aren't perfect. Some irrelevant listings will be included but most of the time you will get a list of, mostly useful, links in order of relevance to your search terms.

How do they do it? Understanding this is the key to getting your website at or near the top of the list.

Relevancy

Websites or, more accurately, web pages are ranked on a results list according to how relevant they are to the search term - keyword or keywords - entered. So how do they decide what is and what isn't relevant?

The key is location and frequency. The most important rule in a ranking algorithm involves the location and frequency of keywords on a web page. The first thing to note here is the Meta Keyword Tag myth. It is not now, nor has it ever been true that the Keyword Tag is the most important factor in getting ranked with internet search engines. These tags are a leftover from when the internet was young, much smaller than it is now and largely limited to academic institutions and scientific interest groups. The people who were creating new pages and adding keywords into the tags were only interested in making information freely available to their colleagues - they weren't selling anything. For this purpose the keyword tag was very useful and it still has its place in intranet sites.

However, all, I repeat all public internet search engines ignore this tag completely. Having got that out of the way lets look at what they do take notice of. Search engines will rank keywords depending on where they appear on the web page. Keywords appearing in the HTML Title Tag will usually carry most weight. Those which appear in the first few paragraphs of the main text will tend to be given more weight than those appearing later. Depending on the engine, text in links will be given some weight also.

The frequency with which keywords are used in the titles and text is also very important. A search engine will analyse how often keywords appear in relation to other words in a web page. Those with a higher frequency are often deemed more relevant than other web pages.

All search engines use this basic method of location and frequency to rank results. But they all do it in different ways. Some will crawl and / or index more web pages than others. Some search engines will also crawl web pages more often than others. The result is that no search engine has exactly the same collection of web pages to search through. That naturally produces differences, when comparing their results.

To some extent, all search engines will penalise pages or exclude them from the index if they appear to be "spamming." An example of this is when a word is used an excessive number of times on a page in an attempt to increase its ranking.

Off the page factors

Off the page factors are those that you cannot easily influence. The most important is link analysis. By analysing how pages link to each other, a search engine can both determine what a page is about and whether that page is deemed to be "important" and thus deserving of a ranking boost. In addition, sophisticated techniques are used to screen out attempts by webmasters to build "artificial" links designed to boost their rankings.

Some webmasters have tried to boost their rankings by setting up "link farms", groups of spurious sites whose sole purpose is to link to each other and increase the "link popularity" of the main site.

As I am writing this (January 2006) Google is taking around 9 months to rank new sites. This isn't due to any lack of efficiency on Google's part; it is a conscious policy decision. It is thought that this is the way that Google have chosen to even out the effects of the recent proliferation of "link farms".

Another off the page factor is clickthrough measurement. In short, this means that a search engine may watch what results someone selects for a particular search, and then eventually drop high-ranking pages that aren't attracting clicks, while promoting lower-ranking pages that do pull in visitors. As with link analysis, systems are used to compensate for artificial clicks generated by eager webmasters.

What needs to be done?

So what needs to be done to get your website listed high up in the search engine results pages? The short answer is that the site needs to be optimised for a set of keywords and phrases that your potential visitors will enter into the search box of their search engine. The copy on the pages needs to be written in such a way that keywords are used in the correct place with just the correct frequency.

The keywords need to be the right ones - the ones that people actually use when they are searching for the information, service or product your site offers. And the copy needs to read fluently, be informative and interesting. After all your site is meant to be read by real human beings, not machines - high ranking will only bring you visitors, it won't keep them, it won't bring them back again and again and it won't get them to sign up for your service or buy your product.

There are lots of other things you can do to fine tune your ranking and get even better results. These will be covered in future articles in this series.

Submitted by Brian Kilcoyne from No.1 Copywriting.