This post is part of the tenth birthday celebrations of bbc.co.uk

The BBC has two domains in use for public service content - http://www.bbc.co.uk and http://news.bbc.co.uk. It ended up that way more by accident than design. Why not, for example, http://radio1.bbc.co.uk or http://eastenders.bbc.co.uk. Or, for that matter, why not just radio1.co.uk?

Promoting everything on TV and Radio with the mantra "Bee bee cee dot co dot ukay slash whatever" implies that the site is one, rather than a collection of mini-sites. Other UK broadcasters have chosen a different path, using a selection of content or channel specific top level domains like skysports.com, news.five.tv, channel4radio.com and so on.

If you promote one URL - bbc.co.uk - then as a consequence you have to put something for everyone on the front door. Initial homepages for the fledgling BBC web services in 1997 were graphic heavy, and had content areas very much classified by the departments that made them. The labels are not terribly intuitive for the user.

Looking back on it now, I have no concept of what I would actually get if I clicked the link labelled "Technical Services", and why that would be any different from the link labelled "IT" on the other side of the page.

This volume of BBC content online grew and grew. By 2001, the relaunched homepage - now under the BBCi brand - had 21 primary categories with 68 sub-category links.

When a shot of the 2007 BBC homepage design was leaked on Flickr, Matt Jones, who has done a lot of fantastic design work at the BBC over the years, picked 2002 out as his BBC homepage design "classic".

It was the product of a lengthy and engaging design process, which was compiled into a book called The Glass Wall.

(You can download that as an 8meg PDF here, and it gives a great insight into how the page came about.)

The 2002 vintage still had an over-abundance of links, though, and a 2004 re-design didn't exactly get the situation under control - the categories were reduced to 12, but there were still 60 sub-category links listed.

I can't tell you how many phone calls I used to field from producers keen to see a link to their site just squeezed into the available pixels underneath their parent category. My answer was always the same - the categories were chosen following card-sorting exercises with groups of users, and I wasn't messing with them.

The problem though is almost intractable, and is summed up by some of the audience research carried out for Philip Graf's 2004 review of BBC Online for the DCMS:

"The review's audience research presented some reservations about the design and ease of navigation from the BBC Online home page. Users, other than the very inexperienced, tend to be goal orientated, seeking to find a specific service or information as quickly as possible, but members of the public found the BBC Online homepage too cluttered and that it did not adequately serve as a guide to the rest of BBC Online."

People want to find exactly what they want on the homepage, and everything else to them is just "clutter"; yet the page has always had to cater for everyone using it as the front door to the site, and to showcase the breadth of the BBC's service. As ex-homepage picture editor Andrew Webb once put it when talking about re-designing the homepage:

"Consider how we'd handle say the World Cup, if you could have all the BBC entire World Cup output at your fingertips.. Now think that at the same time Wimbledon is on, and there's a dozen other things happening like the Chelsea flower show or local elections, all of these have past and live BBC output.. and people might want all, some or none of these things in different sizes and ways, from live HD video right down to portable sized chunks. And consider that ancillary content is now being generated, so there's world cup blogs, head gardener blogs, interactive guides, player information, and a billion other things... Makes your head hurt sometimes!"

The new 2007 homepage design (still in beta) tries to address this a little by letting the users choose and drag around what is important to them. At the time of writing there's already lively feedback about the design happening on Richard Titus' post for this blog. It will be interesting to watch the reactions develop as people get used to the page.

Martin Belam is a former Senior Development Producer, New Media

This post is part of the tenth birthday celebrations of bbc.co.uk

In his introduction to this set of tenth birthday articles, Nick Reynolds said that he couldn't recall a time when the BBC didn't have a website. This raises the question: if the BBC's Director General woke up tomorrow, and suddenly realised that the BBC had forgotten to ever build a website - what would Mark Thompson ask to be built?

If you set yourself the task of imagining building bbc.co.uk from scratch, there are quite a few things that probably wouldn't look much different. A page for every programme? That makes sense. A place to download TV programmes and catch-up on radio? Likewise - although the way we now take listen again and DRM-free podcast downloads for granted belies the innovation, technical and legal complexities in delivering those services.

You'd probably also think about building something pretty similar to the BBC News site. I suspect, though, that it in the regulatory climate of 2007 it would be rather harder for the BBC to launch. The howls of protest to the BBC Trust Ofcom and the DCMS from the commercial news sector would be deafening.

There are some things that, on reflection, you probably wouldn't build.

Why have BBC only TV listings on bbc.co.uk, when BBC Worldwide has a perfectly good Radio Times site covering much more, and, as the TV promos might have said, "other listings web sites are available"?

The Telegraph recently ran an article with a list of rather odd bits on the fringe of the site that the BBC could perhaps cull - although visiting the ever-entertaining h2g2 in order to look for something obscure or esoteric was rather like shooting Babel fish in a barrel.

Since the breadth of coverage on BBC Online has always been vast, and because in 1997 the BBC was so wedded to the concept of being a linear broadcaster, it has been problematic to organise the site.

But what about Jamie Kane? Where do you put an immersive game experience that was played out on websites, mobile phones and through instant messages?

Or "WW2 People's War"? A great historical resource collection, no doubt - but it's not structured "education" content for schools, nor does it have a television programme to go with it.

What about the BBC's backstage.bbc.co.uk project - should that be filed in a "geek" category alongside bbc.co.uk/opensource, BBC Research, iPM and Pods and Blogs?

Karen Loasby, team leader for the information architects in the Future Media & Technology division, recently defined the problem as trying to "re-brand miscellaneous". Whichever way you sort the BBC's web content, you always end up with a pile of "awkward bits 'n' bobs". Describing her work on a recent project to re-define the site's Information Architecture from scratch, she said:

"Alongside the meaningful stuff like 'programmes' and 'news' we've got 'about' which is just a bucket for corporate information and other pages we have to have on the site but the audience isn't necessarily looking for. At the moment we've also got 'innovation' which is a bucket of new stuff that doesn't fit in the current org structure. And then there is 'products' which wouldn't necessarily be a miscellaneous category for another organisation but for us it means things we make that aren't TV or Radio programmes. Might need to have a re-think."

Deciding how the site's organisation was shaped has always had a massive impact on the design of the BBC's homepage, and in the last of my guest posts for this blog, I'll be looking at ten years of homepage information design.

Martin Belam is a former Senior Development Producer, New Media

This post is part of the tenth birthday celebrations of bbc.co.uk

The freshly redesigned BBC homepage comes complete with a swishy new interface that allows users to customise the way the page appears to them. It isn't the first time the BBC has dabbled with personalising homepages.

For some time now, users have been offered personalised weather on the BBC homepage, and BBC News has had a small panel for UK users allowing them to get their local stories straight away by entering their postcode.

The BBC homepage also used to have "targeted" promotions in the early 2000s. As users visited places around the BBC, a cookie would pick up where they went, and identify them as one of four types of the BBC's web audience. When they visited the homepage, they would see a promotion appropriate to their audience type - i.e. whether they were a soaps/entertainment type, or a Radio 4/factual type.

The biggest attempt at personalisation on the BBC site, though, was a service launched in 2000, called myBBC. In some ways, it was rather ahead of its time.

Of course, compared to modern Web 2.0 services like iGoogle, Netvibes or Pageflakes, it was rather clunky and primitive. There was no AJAX involved; it was all delivered using a Perl back-end and the BBC's Central User Database. It used CSS to do the colouring, which was against the BBC's HTML guidelines at the time (and would still be for a couple of years), as it was not supported across all of the BBC's target browsers.

One of the issues faced by the BBC developers was getting information out of the mostly static HTML sites that were being produced at the time, and into these custom myBBC panels.

Nowadays, the BBC produces hundreds - thousands probably - of RSS feeds, which would be ideal for the job. However, when myBBC was being developed, RSS 0.91 was not yet a year old, and Atom wasn't even a twinkle in Sam Ruby's eye.

That meant that each syndication panel format ended up being pretty much custom built for each area of the page.

Another element of the myBBC service was the way in which it worked across different platforms. The idea was that once you were logged in, you would get the same selection of content whether you were viewing myBBC on your computer, your PDA, or using the little lamented Web-on-TV format.

Despite the cross-platform features, in some ways it was more Web 0.2 than Web 2.0, but myBBC won awards at the time. It was one of the links in the BBC Online global navigation, appearing in the left-hand links column on every page on the site.

It was quietly closed down in 2003. It didn't have a massive user base, although those who did use it were fiercely loyal to it and had grown accustomed to it as their homepage. It seemed that in some ways that level of personalisation was a challenge for the BBC's very mainstream online audience. It will be interesting to see how the new drag 'n' drop BBC homepage interface fares with that same audience nearly five years later.

Martin Belam is a former Senior Development Producer, New Media

In my last post, I was looking at how email and message boards have revolutionised the relationship between the BBC and what used to be a much more passive audience - even if the technology powering those boards was a little temperamental at times.

By 2003, it was clear that the BBC needed to move to a new system, and the decision was taken to migrate the message boards to the DNA software that the BBC had acquired when it purchased the Douglas Adams-inspired h2g2 site in 2001.

Ashley Highfield wrote on this blog the other week about his so-called public "spat" with James Murdoch, but Sky haven't always had a rough deal from the BBC's New Media department. In 2004, one of the first boards to move to the new system was Points Of View, which fell under my remit. In the process, I inadvertently signed off as approved a design that included a very prominent Sky remote control in it, rather than the more neutral "sticky-backed plastic"-type of branding that BBC regulations require.

Opinion among the audience is divided about how effective the BBC's moderation is. Nobody likes it when their message fails to appear, and the BBC News Have Your Say even has an indicator of the status of the moderation queue in a bid to be transparent.

Steve Hermann recently asked for feedback on how users saw the BBC's blogs - and the issue of comments came up over and over again, with opposing views:

"I continue to think that the blogs and HYS need a community moderation scheme akin to Slashdot in order to reduce the number of 'me too' posts and so that thoughtful, worthwhile comments can be given greater prominence." - Kendrick Curtis

"Comments which add to the debate should be published but those which are rambling or simple repetitions should not...Editors should exercise their experience in vetting comments. Wake up BBC!!" - Pancha Chandra

"Disagree with those who opine that the number of comments should be slashed by removing 'repetitions' or rambling comments." - JustSomeCalifornian

"I don't know about the blogs, but drastic improvement is needed on Have Your Say. On the recent France/US Special Relationship topic, only 18% of posts were published!" - Steven Martin

I think opinion is still divided within the BBC as well. Sites like Digg and Slashdot are often cited as examples of good community moderation tools in action; however, they also have a relatively homogenous audience group using them. That is far from the case with the BBC's audience.

In my mind, particularly on the BBC News site, the current model is an improvement. For all the flaws of "Have Your Say", it is vastly better than its closest interactive forerunner, "The Great Debate" message-board - which was, frankly, a bear-pit.

The fact that the current "Have Your Say" feels more like a letters page on a specific topic, rather than a message board for chit-chat, seems to have quietened down the repetitive flame wars between individual regular posters. Despite the number of messages that don't get through the moderation queue, the BBC is now publishing many, many more contributions than when it used to select them by hand in the old "email and publish" model.

There is no doubt that contributions from the audience add value to the BBC News site, make it more entertaining, and allow the BBC to showcase a diverse range of views. However, BBC News receives something astonishing like up to 30,000 contributions to Have Your Say a day. Even using a lower conservative estimate, that adds up to several million messages a year, which is of course, quite resource-intensive to manage and moderate.

Over the last ten years, the BBC has steadily moved from an "email and publish" model, to pre-moderation, through to post-moderation and eventually, in many areas, reactive moderation.

Reactive moderation, however, has its pitfalls. I remember a flurry of panic within the BBC when it was suddenly noticed that the Doctor Who message board was riddled with insulting comments about incoming Chairman Michael Grade, due to his role in the show's demise during the 1980s. Of course, nobody had complained about what was being posted, since most Doctor Who fans agreed with the unkind things being said about him.

There are also plenty of other places on the Internet, like Digital Spy or most newspaper websites, where audiences can gather and discuss the BBC. Interactive tools like blogging have given a voice to campaign groups both for and against the Corporation. Email has proved to be an effective lobbying tool over issues like the BBC screening of "Jerry Springer - The Opera", whilst there is seldom a time when a search for BBC on the Number 10 e-Petitions site won't turn up something to sign.

This represents perhaps the biggest shift over the last ten years. The correspondence between the BBC and the Licence Fee payers who fund it now happens much more out in the open. You used to be able to write to the BBC, and hope that Barry Took might read your letter out, and... erm, well, that was about it. Alternatively you could write to a newspaper about the BBC, and hope they published it. Nowadays, it only takes a couple of comments on the BBC's Points Of View message board, and Emily Maitlis or Fiona Bruce find their legs being debated in the national press.

Martin Belam is a former Senior Development Producer, New Media

This post is part of the tenth birthday celebrations of bbc.co.uk. Part Two is here.

It is easy to forget how rapidly email has revolutionised the way that the public interact with organisations like the BBC. I still remember the phone number of Multi-Coloured Swap-Shop, and the postcode of Radio 1 from when I was a kid. Now a good proportion of interaction is done rather more rapidly via SMS and email. In 2004, Chris Kimber, then Head of Interactive at BBC Radio & Music, said:

"Only 10 years ago, radio was a one-way experience, but digital technology has given the radio ears that provide programme-makers with instant feedback. Before they had to rely on getting letters back but now we have chat rooms, message boards, text messaging and e-mail. Programmes can really connect with audiences in a way that 10 years ago they could not".

The recent furore over the Radio Five Live phone-in about the Madeleine McCann case highlights how real-time this feedback loop is.

The BBC has often struggled to deal with the sheer volume of correspondence it receives.

By the time I left the BBC in 2005, we estimated overall the Corporation was receiving something in the order of hundreds of thousands of emails a week from the public - but that was a bit of a finger-in-the-air estimate to be honest. Whatever the figure, it was far, far, far more than could be economically individually answered.

Email isn't the only "new media" route over the last ten years that has given the audience an opportunity to communicate directly with the BBC - for some time, bbc.co.uk has been hosting message boards.

The early boards used an in-house system called "howerd2". It was named after Frankie Howerd, as in "A funny thing happened to me on the way to the forum..."

Internal BBC urban myth has it that the system was built to last for six weeks to provide a message board to accompany one programme. The BBC ended up using the software for something more like six years.

It was prone to being rather temperamental, and there were frequent problems with keeping the database server in sync with what appeared on the front-end of the web servers. Which is, I believe, the technical way of saying that messages tended to disappear into cyberspace and never be seen again.

This, obviously, infuriated users who had spent time composing their posts. If the message contained a complaint about the BBC, or a political standpoint, and then disappeared due to the bug, it often appeared to them to be the heavy hand of BBC censorship at work.

The message boards used to be closed overnight, but posts could be made and queued for publication. In the end, moderating the backlog of messages each morning became uneconomical, and the boards were just completely shut at night. Well, sometimes. Sometimes the server would get a little confused, and open up for posting seemingly when it fancied it.

In some cases, the BBC's message boards proved too popular for their own good. Not only was the technical system stretched to capacity, but moderation was very expensive and communities took on a life of their own in directions the BBC didn't expect. One of my ex-colleagues, who was a part of the "Community" team, used to say that back in the early 2000s, BBC producers "didn't always understand why people want to visit the programme x board to talk to each other, rather than to talk about programme x."

The decision to later close some boards in the entertainment and drama areas prompted a furious backlash from what The Times called "ladies who lunch on the net":

"They are members of one of the country's most clandestine communities. Predominantly female and in many cases highly educated, they have gathered - unbeknown even to their husbands - under assumed names to meet 'like-minded people' for more than two years. But this morning this mysterious group will wake up to find its cosy world in tatters. Yesterday the BBC axed the majority of its online message boards."

In the next part of this post, I will look at how the BBC migrated to a new message board system, and the ever thorny issue of message board moderation.

Martin Belam is a former Senior Development Producer, New Media

If you extrapolate current statistical trends, by 2025 40% of the UK population will be obese, 1 in 3 of us will be Elvis impersonators, and increased electronic storage will mean that you'll be able to carry a video of your entire life around with you at all times.

Well, maybe not quite, but it's certainly true that over the last ten years physical storage of digital media on servers and removable storage has got cheaper and smaller - and it wasn't exactly big in 1997.

All of which raises the question - why can't I just link to the 1997 version of the BBC website, or the 2000 version, or to exactly what it was like on Christmas Day 2002? A lot of those things are simply not there anymore.

When I joined the BBC, I was always told that Andrew Neil had the first individual programme support site - but that, for example, is no longer on the BBC site. The pages credited the production of the HTML to Mark Himsley, just as the BBC would do for a television programme.

Even in cases where an original BBC website from the 1990s is preserved, there is no guarantee that any of the functionality will survive.

In 1997, the BBC covered a general election online for the first time. The pages are still on the web, but sadly, Peter Snow and company no longer swing like they once did. Clicking the 'Interactive' option from the menu generates a 403 Forbidden error.

Sometimes, although the HTML has been kept live on the BBC's front-end web servers, some of the dynamic applications at the back-end have been decommissioned, accidentally wiped, or have simply broken as the BBC's technical infrastructure evolved.

Sometimes, when sites have been formally closed, the decommissioning of sites has been handled well by the BBC. The much-missed Cult website has a goodbye message on the homepage, and the banner has been altered to reflect that the site is no longer maintained.

At other times, the BBC has been a bit clumsy.

When I started at the BBC in 2000, the online coverage of the France 98 World Cup was already badly broken, because someone had deleted the entire directory with all of the images from the tournaments. I've never established whether it was a mistake, or due to rights issues with the photographs themselves - but either way, it basically destroyed the site.

That isn't to say it wasn't BBC policy to keep anything and everything. One of the cybercrimes that could have your "Trusted User" FTP access to the BBC's master content server revoked was getting caught deleting a large chunk of the site without letting the BBC Broadcast team know first so they could back it up.

The BBC is learning, though. With the new programme support pages, there is the promise that each episode of every show will have a permanent and unique URL for the foreseeable future. The Information & Archives department, home to an astonishing treasure trove of physical archives, now has a system in place to capture copies of files as they are uploaded to the BBC's servers, and to preserve the multi-media elements related to programmes.

There have also been moves outside of the BBC to archive the site. The Wayback Machine has some content from the BBC site that can no longer be accessed. News Sniffer is a favourite site of conspiracy theorists on both the left and right of the political spectrum, whilst the BBC News homepage and BBC homepage archives were prototypes that came out of the BBC's backstage.bbc.co.uk initiative.

Every now and then the BBC runs a treasure hunt or amnesty, asking people to return any old reels of films or home recorded clips of BBC shows that have now been lost to the archives. I sometimes wonder whether they should do a similar appeal to try and get some missing screenshots back of the early web years.

Martin Belam is a former Senior Development Producer, New Media

Although I was Senior Development Producer for the BBC's homepage at the time, I wasn't actually in the office to help with coverage of the July 7th bombings. By the time I'd started travelling that day, television was already reporting "power surges" on the London Underground.

The BBC, of course, has a whole raft of plans and procedures for national emergencies. There was already an established process of handing control of the main picture promotional area of the homepage directly over to BBC News in the event of a major story breaking. However, that didn't seem to go far enough that day.

By the afternoon, the page being served was an experimental XHTML/CSS table-free version that had previously only been used as a 'proof of concept'. It worked in most legacy browsers - although I do recall that the BBC got emails of complaint from some users of Internet Explorer on the Mac OS. BBC News had a further contingency plan in place - by 11am that morning the BBC News site was being served via Akamai, rather than from the BBC's own server farm.

Stuck at home, whilst trying to get on with some other work, I kept taking screen-grabs of the BBC homepage each time it updated, and was uploading them to Flickr to keep a record of the changes throughout the day.

In the middle of it, someone messaged me via Flickr, asking that I stop doing it. It was, apparently, really annoying them and wasting their time, because when they were looking for pictures of the bombings, my screengrabs kept coming up instead of pictures of the actual carnage, which is what they wanted. You do meet some funny people on the internet.

I also recall that during the rush to keep the information updated and the page running, a phone call was taken from a distant BBC policy department, pointing out that the homepage was erroneously referring to the recently re-branded "Action Network" service under the old name "iCan". I understand it was somewhat forcefully explained that this wasn't perhaps the most pressing issue at the time!

The flexible use of the BBC homepage was in contrast to how the page reacted on September 11th 2001. The template wasn't anywhere near as malleable then, and so all that could be done was to edit the three promotional slots on the page to carry news of the unfolding events.

In order to get the information published faster, the team bypassed the CMS, and were simply using FTP to put some small HTML includes live. In fact, until the directories were cleaned up and reorganised in 2005, if you knew the right address, you could still find the HTML snippets with the last news updates from that day on them.

After the July 7th bombings, Doug Graham, who was part of my team, was instrumental in making sure that the BBC published some figures about the sheer volume of data and streaming that had been served that day. The headlines: 40,000 page requests per second on the News site, and BBC bandwidth usage reached 11.1Gb/sec, with an extra 3Gb/sec going to Akamai.

At that time, the BBC didn't have any kind of "blog" outlet like here or over at the BBC News Editors blog. Since it wasn't press release material, the numbers were hastily published in the feedback area of the site.

Incidentally, the little bump you can see on the graph from the day before was the surge of traffic rushing to find out whether London had been awarded the 2012 Olympics, which had set a much happier, but short-lived, bandwidth consumption record just the day before.

Martin Belam is a former Senior Development Producer, New Media

When he was at Ask, Tuoc Luong once put the demands placed on search engines very succinctly:

"We need to read users' minds."

He was referring to the tricky job of interpreting the very small amount of information that the user gives when the type in a search query. Does someone looking for "jam" on the BBC website want some recipes, to listen to a documentary featuring Paul Weller, or to find out what the traffic will be like on their way home?

The "lilac" search results I described in the first part of this post only lasted for a few months, as in 2002 there was a major push to integrate web search within the BBC's search engine. Previously, as part of the BBC's remit to be a "trusted guide to the web for the UK", there was a directory of links called WebGuide - a small-scale Yahoo!-style directory which linked to sites of UK interest.

By now though, Google was indexing billions of web pages, and it was obvious that maintaining a small directory of sites wasn't going to scale up to the growing size of the internet. The BBC turned to search as a tool, trying to provide a web search that was safe for children, promoted UK content, and which, unlike all of the major commercial search engines, didn't take advertising in return for prominent placement.

In the summer of 2002 this launched as the main search box on a newly designed BBCi homepage. Getting involved in web search wasn't a popular move in some quarters. Commercial sites saw it as a threatening land-grab, and the BBC was accused of artificially inflating the ranking of BBC content within the results.

In his 2004 DCMS review of BBC Online in 2004, Philip Graf wrote that:

"Respondents, in our audience research, could not clearly identify the value of the BBC search engine. Many were content with commercial tools, such as Google. On balance, however, I feel the BBC should retain its search engine. Given that search is becoming such a fundamental part of how the internet is used, it is worth keeping a publicly funded, UK competitor in the market place."

The move into web search could not ultimately be described as a success. It is now tucked away behind the fourth tab on a BBC search results page, and has no direct search box entry point on the site. Other search developments have been more welcome, however.

CBBC Search provides a safe environment for kids to start learning about searching the internet, with parents able to feel safe that nothing bad will be lurking in the results. And in 2004, in one of the first examples of a public BBC "beta test", News launched an audio/video component in the search results. With the arrival of the iPlayer, the ability to search through video content is a crucial future development for the BBC.

The last significant redesign of the BBC's search systems took place in 2006. This unified all of the BBC's content in one massive 1.5 million document index, and removed the concept of 'scoped' searching of just one area of the site. Finally, the technical and organisational hurdles had been overcome, and wherever you were on the BBC site, a search should produce the best results, from the whole range of BBC content, including audio and video clips.

It isn't quite a mind-reader yet - but the BBC's search engine has come a long, long, way from the days when I had to edit the results by hand!

Martin Belam is a former Senior Development Producer, New Media

Until a couple of years ago, I was a Senior Development Producer at the BBC's New Media department. Whilst I was there I used to blog rather enthusiastically about my work, and the team at the BBC Internet blog has asked me to contribute some articles here about the history of the BBC's web site.

I first started to work at the BBC in 2000, as a junior member of a small team looking after the BBC's search engine. Back then, searching the BBC site was a bewildering and perplexing experience, as there was no global search across all of the content.

Instead, on the Today site, you could find a small box in the top right-hand corner that only searched the Today site. Or, if you were on the EastEnders site, there was a long search box at the bottom of the homepage, that only searched the EastEnders site, and so on.

As well as being somewhat randomly placed, the search boxes weren't even all using the same technology. BBC News used a product from Autonomy, whilst other bits of the BBC were indexed using software called Muscat. The results could be pretty appalling. One of my first jobs involved artificially putting the right URLs at the top of search engine results.

This wasn't a hi-tech solution. We had a spreadsheet that listed search terms, and the URL that should be displayed if a user employed them. We used to improve it based on the frustrated emails we got from the public. A mail would come in saying "I searched for 'Jeremy Paxman' and I never found the Newsnight site", and the team would dutifully add that 'jeremy paxman', 'paxman' and 'rottweiler' should produce bbc.co.uk/newsnight as the number one result.

The Muscat search engine was also unable to distinguish between different languages, so if you typed in 'Tony Blair' you were just as likely to get a news story mentioning his name from the BBC's Portuguese news site as from the English language site.

It was obvious it needed to be improved, and as part of the re-branding of BBC Online to BBCi in 2001, a new global search was introduced. The grey 'toolbar' was added to the top of (nearly) every BBC web page, placing a search box on every page of the site.

To get around this, the scope of the "Best links" that used to be hand-coded into that spreadsheet was increased. It became a large behind-the-scenes taxonomy mapping relevant BBC content against thousands of keywords and concepts. If you typed in something like "Test The Nation" as your search anywhere on the site, the top results would include a BBC "Best Link" taking you to the national IQ quiz homepage. The new interface wasn't universally acclaimed, but it was a vast improvement, and the indexing of the BBC's content was improving as well.

In my next post, I'll be looking at how the BBC introduced web search to the site in 2002.

Martin Belam is a former Senior Development Producer, New Media

This post is part of the tenth birthday celebrations of bbc.co.uk

I don't think I've been to a meeting, presentation, or read a document from the BBC in the last couple of years that doesn't cite "Doctor Who" as a shining example of something-or-other.

Well, I'm about to indulge in the same vice myself, as I think you can get a pretty good snapshot of the development of the BBC's web activity over the last 10 years by looking at the programme.

Prior to the RTD-inspired revival, 1996 was the last time the BBC tried out Doctor Who on TV, with a one-off movie starring Paul McGann in the role. Promotion was strictly on air and in print - there was no such thing as a BBC Doctor Who web site.

During the gap between the TV movie and the arrival of Christopher Eccleston in the title role, there was plenty of off-air activity around the show. The BBC had a successful range of books based on the subsequent adventures of McGann's 8th Doctor, and Big Finish produced a monthly series of audio drama CDs featuring actors from the Doctor's past, some of which have now made their way onto BBC7.

And then there were the webcasts.

The webcast was basically a radio show accompanied by some animation techniques that older viewers might recognise from the original Captain Pugwash. Well, that perhaps sounds a little harsh, but essentially static drawings of the characters would fade in and out at varying degrees of magnification, over the top of a set of different background images.

During the next couple of years, this technique was repeated for "Real Time" starring Colin Baker (in a rather more sensible cartoon costume than his TV one had been) and "Shada", with McGann reprising an unfinished Tom Baker television story. Whilst it wasn't broadcast quality, it was still 'official Doctor Who from the BBC', and so with no real prospect of it returning to TV, fans were mostly kept happy.

A real leap forward occurred in 2003, though, with "Scream Of The Shalka". Made to celebrate the show's 40th anniversary, the visuals were produced by Cosgrove Hall and demonstrated that the BBC was confident, with increased broadband take-up, that it could stream more demanding content. In fact, the animation was good enough to be broadcast on television via the red button over the Christmas holidays, and the same technique was used to reconstruct two missing episodes of the 1968 Patrick Troughton story "The Invasion" for DVD release.

Shalka remains the only appearance of Richard E. Grant's angular 9th Doctor though, because just prior to the webcast launching, the BBC announced that Doctor Who would be returning to BBC One. The team behind the webcast put a brave face on it, despite their project becoming not the real new Doctor Who before it even started.

In contrast to 1996, when "Rose" was broadcast in March 2005, there was already acres of web support material. In the run-up to the show's return, the Doctor Who site had shown a new cryptic picture from the production team every day for several months, and there were streaming interviews available with key contributors like composer Murray Gold.

The web also took on a role beyond mere programme support, and became part of the programme. The BBC produced fake websites for on-screen organisations like U.N.I.T. and Geocomtex, a 'Who is Doctor Who?' site purporting to be run by one of the characters in the show, and there was a message for Rose on the Bad Wolf site that kept cropping up in BBC Search results. Ultimately, the Daleks even invaded the BBC homepage.

Not only that, but the BBC got a wake-up call about the power of the web as a distribution medium. "Rose" was leaked on the net weeks before it was shown on BBC One, the first time that this had happened to a high profile BBC production. Users couldn't officially watch the programme itself over the web, but they were able to stream the behind-the-scenes "Doctor Who Confidential" show on demand.

Fast-forward to 2007, and now the show, plus spin-offs like Torchwood and The Sarah Jane Adventures, can be downloaded from the BBC's iPlayer, and each Doctor Who episode is accompanied by a free commentary podcast to go with the programme. A couple of weeks ago, you didn't even have to be on the BBC's site to watch the Children In Need special "Time Crash". The BBC themselves had uploaded it to their YouTube channel.

From nothing, to watching TV online, all in the space of ten years - that would have seemed more futuristic in 1996 than anything dreamt up by the script-writers.

Martin Belam is a former Senior Development Producer, New Media