Spike Milligan: Policemen are numbered in case
they get lost.
(The Goon Show: The Last Goon Show of All, 1972)
I couldn’t come up with a suitable (or should that be
unsuitable?) quote from the Carry on Film’s for this post, so I
hope you’ll forgive me for picking one from the Goons instead.
Last time in Carry on up the Internet: A
bit of an Enigma I talked about how the club’s web site had
managed to knock itself out of Google and other search engines
using a robots.txt file. Having corrected the problem, they could
have sat back and waited for the search engines to add them back
in. With a bit of luck they’d be picked up in about a month. They
asked if there was any way to speed this up.
In the early days of the web, you could submit yourself to
search engines and listings sites (we used to run a service some of
you may remember called Signpost that helped you do that).
Submitting to search engine and directory sites fell out of favour
for a few years.
In the last few years the biggest search engines have brought in
a new way to let them know about your site. Funnily part of it
involves using a robots.txt file on your site. Back in 2005 Google
launched, Google Sitemaps, a way to tell Google about the pages on
your site. Since then other big name search engines like Yahoo! and
Bing have adopted the standard which is now known as XML
Sitemaps.
By adding a file to your web site, usually called sitemap.xml,
to your site you can tell the search engines about your pages to
make it easier for them to find them. You can add a line to a
robots.txt file to say where your sitemap file is to make it easy
for the search engines spiders to find. While it is possible to
write a sitemap file in a text editor there are several web sites
and software that will produce one for you. Many content management
systems, like Joomla and Liberato, can automatically make them behind the
scenes.
When you update your site you can use websites like
Bing’s webmaster tools (link opens
new window), Yahoo! Site Explorer (link
opens new window) and Google’s webmaster tools (link
opens new window). If you're using a content management system,
it can send a ping to the search engines to automatically let them
know whenever your site changes.
So rather than sitting back and waiting for the search engines
to come to you, you can proactively go out there and tell the
search engines about your site.