Spiders, Crawling, and Internal Links

We speak of the internet as a web, and talk about the search engine spiders crawling it. That’s because the internet was originally conceived as a web of hyperlinks interconnecting everything. That’s how it generally is now, too, though it’s a bit more sophisticated than it used to be. Search engine spiders arrive at your website and crawl along all the links to find all the pages. They index the site, which is to say they make themselves a sort of map of the site, and then when someone looks for something you have, they mention you to the searcher.

It’s a great system. However, in quite a few of the sites I’ve worked with recently, I’ve found pages that aren’t linked up to anything.

Normally, you know, you have navigation on the homepage that takes people to your inner pages. You might then have further pages linked from those pages. With luck, they’re all hooked up together in a sensible way that people can easily follow. If so, the search engine spiders probably can do so, too.

But sometimes you find a page which isn’t linked up to anything else. People might find these pages through a search function at your website, or they might never find them at all. They’re not going to do you much good, though.

How does this happen? Lots of ways. Here are some I’ve seen recently:

• The page is private. If you have a secret page (press photos, prices, other things you don’t want people to be able to find by searching at Google), then you may have no links to it and that’s your prerogative. You can then send people a link to the page if you want them to see it.
• The page is left over from the past. It had a link from some page which has been deleted, for example, and now that its parent page is gone, it’s lost in cyberspace. If you don’t need it any more, delete it too. Otherwise, give it a link from an existing page.
• The page should have been a post. I’ve seen several orphan blog pages recently which were made by people who thought they were saving them in a way that caused them to show up automatically on a page. They weren’t. If you want that to happen, make a post, not a page.
• The site was built wrong. It’s sad but true — some sites are built without proper internal links. If yours is one of them, it won’t be properly indexed.

Examine your website, round up your orphan pages, and get them properly linked up.