This week, we launched our new site search product. This launch is a big step on our journey to deliver a single, unified and consistent discovery experience for the BBC.

Try it out with the search box in the top right-hand corner of most pages (or, if you can't think of something to search for, try Liz Hurley, Miranda, Unemployment, Spain). Let us know your thoughts using the form at the bottom of this page.

The new search is based on the Search+ platform, which I blogged about earlier in the year, and includes many refinements and enhancements.

This new search supersedes both the standard search product and the three-column Search+ product to offer a single site search experience for BBC users.

Before I go through the new features in detail, I'd like first to give you an insight into our product vision and discuss some of the issues that have shaped it.

To scope or not to scope

When I started in this job, there were hundreds of distinct sites on bbc.co.uk and most of those websites would have their own search "scoped" to their content. Not only was this inefficient and wasteful but also it led to an inconsistent experience in which search boxes in exactly the same position on, say, the Schools website and the History website would produce different results. People were often confused; indeed, I remember analysing the logs for the Gardening website search and discovering that the top result was for [eastenders] and returned no results.

We decided to reduce the number of scoped searches and make the search box in the top right-hand corner of every page return the same results, wherever you were on the site. However, this was not universally popular. People tend to think of bbc.co.uk as only that part of the website that they like to use. Many people only use the News website, or GCSE Bitesize or iPlayer and, not unnaturally, are bemused when they search for something and get results from another area. We get many emails asking for the search box on News to return results from News only. There is even an online campaign dedicated to achieving this goal!

So, on the one hand, people are finding no matching results in scoped searches, even though relevant content exists elsewhere on the website. On the other hand, people like the convenience of scoped searches, particularly when they only use one of the BBC's services.

The table below further illustrates this problem. It is a log of the top queries coming from different BBC websites on one day in June 2010. The three columns show where the searches originated (on the iPlayer, News and Sport websites). Those queries in black are the ones for which the best results will come from that website (so, for a search in iPlayer for [luther], we should expect the best content to be from iPlayer). The red ones are the queries for which the best results will come from other parts of the BBC website (so, for a search in iPlayer for [revision science gcse] we would expect the best results to come from Learning rather than iPlayer).

 

So, as you can see, the picture is pretty mixed, with a closer correlation visible the higher up the search logs one looks between the originating domain of the search and the provenance of the optimal result set.

Clearly, we need to have a search product which recognises that many users want to scope their searches to the domain they are currently in while simultaneously revealing great, relevant content that happens to be in another part of the website.

Product Vision

With this in mind, we created the following Product Vision.

Our aim for search is:

1. To create a scoped search for each BBC product (News, iPlayer, CBBC and so on) that "understands" the domain knowledge behind each product and uses that information to create the best possible sorting rules, display and filtering options within search. In this way, we will provide the richest search experience available anywhere on the web for each of our products.

2. To allow users to search across product scopes to see everything that the BBC has on a subject.

3. To ensure that, regardless of which scoped search someone uses, they are always exposed to the best relevant content the BBC has to offer. This will ensure that, for example, a heavy News user gets to know about iPlayer content relating to their query.

4. To offer related results from non-BBC sources.

5. To make it possible for people to discover trending topics on the BBC site and make feeds of results available to be embedded in other products (on and off the site). To allow search result pages to be recommended from within other BBC products.

We think Search v2 starts to fulfil a number of these ambitions. Let me run through some of the new features and demonstrate how they relate back to the Product Vision.

1: A scoped search for each BBC product

This example is a search for [graeme swann]. The results have been scoped to Sport. This shows the main elements of the page design, which I have highlighted with red numerals.

 

1. The central column will contain the main list of results. The large heading (in this case, Sport) shows that you are searching within the Sport scope.
2. The left-hand column will contain navigation to the other scopes and filtering options such as media type, date range and sorting choices.
3. The right-hand column will contain search results from other parts of the BBC (Elsewhere on the BBC) or other websites (Elsewhere on the web).

The styling of the new search results uses the BBC's Global Experience Language, which brings it in line with the rest of the website and which Bronwyn comprehensively explains here.

Category-specific filters

In the image above, you can see four sets of filters in the left-hand column. The first is the category filter. It shows you how many results there are for your query within each category. Below this are "scope-specific" filters. For the Sport scope they are:

 

1. Allows you to filter your results to show only video, audio or text & pictures. The default setting is All Media.
2. Allows you to set the earliest and latest dates from which you want to find results.
3. Allows you to change the ordering of your results. As I said above, the default order is Newest First, but you can also choose to order by Most Relevant First or Oldest First.

Over time, all scopes will be given filters best suited to that particular type of content, so Learning, for example, will have filters such as Subject, Syllabus and so on.

Context-driven search results

As discussed above, there is a strong correlation between the part of the website that a user is exploring and the type of content they are searching for.

So, we have set the system to return results automatically from the category that relates to the section of the website you are using - if you are on the News site when you enter a search term, you will get results from the News category in the main, central column. Likewise, if you search from Sport, you'll get sports results.

Over time, we will extend this functionality to other sections of the site - iPlayer, Learning, Food and so on. The eventual aim is that all products on the site will scope to their own search category while staying within a single, consistent design. As this happens, we will remove the old "scoped searches" like this one.

From today, however, if you search from any part of the site other than /news or /sport, you will get combined results on our All Results page. You will know which area you are about to search in from the text inside the search box. On News pages the box looks like this:

On pages where you are searching All Results, the box looks like this:

 

2. Allow views across BBC products

Unscoped searches

When you hit the All Results page, the central column will contain the top results from all the matching categories. The categories are ordered by their relevance to the search query, using an adaptation of the Smart Zones process that I discussed in an earlier post. Here is the [graeme swann] query from above, run on the All Results scope.

 

Note that the page layout is the same as in the earlier example. However, because I haven't scoped my search, I will see results from all areas of the site in the central column. In this case, Sport is rightly deemed most relevant to the cricketer, followed by Blogs, News and then TV & Radio sites. A maximum of three results are shown from each category.

Likewise, the query [christmas cake] returns Food as the top category. (Ignore the Editor's Choice result in each of the next three examples - I deal with this feature in detail later on.)

 

Whereas a search for [iceland] returns News as the top category.

 

And a search for [trigonometry] will put Learning content at the top.

 

You can get to all results for a particular category by clicking the links in the category title (or in the navigation in the left-hand column).

Editor's Choice

Many search result pages will also include a selection of editorially chosen results that we believe are most likely to be what you are looking for. If you search for a programme title, say [miranda], you will see a link to the programme website.

 

If you search for a country, say [burma], we will highlight the BBC Country Profile (as well as other editorially selected features) and if you search for a British place, say [blackpool], you'll get a link to the nearest BBC Local website.

 

 

3: Ensure that related content is never hidden

Elsewhere on the BBC

As I said in the Product Vision, our aim is never to hide great, relevant content from users, regardless of the scope within which they are searching. So, whenever you search inside a scope, you will always see the Elsewhere on the BBC module showing results from other parts of the website.

 

4: Offer related results from non-BBC sources

Elsewhere on the web

Below the Elsewhere on the BBC module, you will see the Elsewhere on the web module showing related results from around the web. These are a mixture of editorially selected websites and algorithmically driven results.

 

5: Linking to search results

Linking to search results from within the BBC

In the example below, you can see that iPlayer now links to BBC search. In this case, a programme about Operation Mincemeat links to search result pages for World War II, Adolf Hitler and the show's presenter, Ben Macintyre. This is possible because search produces a controlled vocabulary of around 40,000 topics that are available as links and which I wrote about previously here.

 

Search feeds

These 40,000 topics are also available as RSS feeds. Click on the orange symbol in your browser's address bar to get the feed.

 

How we put this together

For those with a technical inclination, our senior engineer on this project, John Muth, will blog on the technical approach next week and I will add a link from this blog then.

We're hiring!

Finally, a bit of news about the BBC's search team. We are moving to Salford in the summer as part of a BBC-wide relocation. As you can imagine, such a big change has resulted in a fair amount of turnover within the team and so we are now actively hiring software engineers (predominantly Java) and other roles, including my job - Technical Product Manager, Search. You can read about the recruitment process here.

• A brand-new search which intelligently tailors the display to the specific query the user enters
• The addition of featured content to search result pages
• Improved links to non-BBC content
• More accurate results through improved meta-data creation

The richness and diversity of the BBC's internet content (News, Blogs, iPlayer, Weather, Sport, Recipes and so on) places demands on site search that are different from more focused websites and this led us to explore original solutions to enterprise search.

As you will see in this post, our solution is a major departure from traditional enterprise search designs. So, for the rest of this post, I'd like to give you some insight into how we reached our decision to make such a significant change to the way we deliver our search results.

As always we would love to hear your views on these changes.

The challenge of site search

This was a daunting challenge, but as always the first thing to do was to define the problem. This meant analysing what users are searching for and which results they are clicking on.

To explain what we found, consider this list of the 50 most popular searches in December 2009.

The third column shows the click-through rate (%CTR) for each term, i.e. what percentage of people who searched for that term actually decided to click on a result. The higher the %CTR the more satisfied we believe users to be.

Notice that I have colour coded the different search terms. Green are searches where the user is looking for something specific: usually a part of the BBC website (iPlayer, the Weather website, Bitesize) or a named programme (EastEnders, Top Gear, Doctor Who). Red ones are searches where the user is looking for something about a topic - a particular person, a country or city, a subject or news event.

Look what happens when you average the %CTR for these two types of search:

Looking for something specific 83.86%
Looking for something about a topic 63.91%

As you can see when you want to get something specific, the BBC site search is pretty effective. But if you want to find things about a person, place, subject or event the experience is patchy.

So we began to re-develop site search to improve this type of search, which I call Topic Searches.

To start with we asked "why are Topic Searches less successful?" The answer is surely that the intentions of the person doing this type of search are less clear. When you search for merlin or top gear or travel news the intended results are easy to predict. But when you search for iran, what do you want? The latest news about Iran? Some background information on the country? A programme that's available on iPlayer?

What about searches for Delia Smith? Do you want to know when her next programme is being broadcast? Do you want to follow one of her recipes?

Results display: the problem with search result lists

"The search just throws everything at you, you would have to spend ages just looking through the pages to find what you need."

There are two big problems that make a single list approach very difficult:

1. You need to understand what the user means to know what relevance scores to give to their results. Two searchers may want two different things but use the same search term - one user may search for swine flu looking for a description of the symptoms whereas another may use exactly the same search term but be interested in news about the latest outbreaks.


2. Different types of content are relevant for different reasons. The most relevant news article is the most recent one (unless of course someone is looking for a specific article). Whereas for archive programmes or background articles date is much less important, the most relevant thing is the one "most about" the subject.

This means that when you try to produce a homogenised, "relevance ranked" list of results you need to add biases and boosts to different types of content to get the right results to the top of the list. With the diverse content on offer on bbc.co.uk, this is nearly impossible.

(It's worth noting that the major search engines have also been moving from a single list view to a more modular set of results over the last few years. Google for instance include modules of results from their verticals - News, Video, Blogs, Maps and now Tweets - blended into their Web results. Google's Marissa Mayer laid out their strategy in a post in 2007, Danny Sullivan has a great explanation too.)

So our solution rejects the single list approach and instead introduces "Smart Zones".

The New, Enhanced BBC Site Search: Smart Zones

As you can see, the results are split up into sections or "zones" - in this case: News, iPlayer and Knowledge (the zone which holds background and in-depth content). The content in each zone is ordered in the most appropriate way - so that you see the latest News but the most relevant Knowledge items.

Of course zones are nothing new in search result pages but - this is the clever bit - these zones order themselves on the page depending on the search query, the matching content available and what we think users find most important. So every time you search you will see the best possible ordering of available results.

For example, compare the results above with a search for David Cameron I did on Sunday morning just after he had been interviewed on The Andrew Marr Show. You can see that the News and iPlayer zones have swapped places as the system has calculated that the interview is likely to be the most popular result.

When I searched for Ryan Giggs, the Sport zone appears in the results.

Searchers can see all zones or focus in on one particular zone. Here I have searched for Jonathan Ross and I have selected to expand the News zone to see more News content (note all the other zones are still available on the right of the screen reminding you that other relevant content is available).

Other Enhancements to Search: Highlighted Content

Likewise, a search for India returns the BBC Country Profile and the latest weather forecast. (Note that no new content is being produced here, we are just finding new ways to draw attention to our best web pages). We will be working to increase the amount of highlighted BBC content over the coming months and highlighting the best content from other websites.

Other Enhancements to Search: Extra results from around the web

Other Enhancements to Search: Extended query terms

Other Enhancements to Search: Better results through added content structure

Note this system is in its infancy so it will take some time before the improvements in result relevance are apparent for every search.

Other Enhancements to Search: Addressable search results

Phased Roll-out

At the moment only about 11,000 search terms will return the enhanced results detailed above.

Over the coming weeks more and more searches will return the enhancements until every search works in this way. There will be additional zones added into the results and we have a slate of new features that are in development.

We will also be using the technologies we have built here to produce exciting new applications. One of the objectives we are focusing on is how we can use these tools to make you aware that there is new content about things that are of interest to you as soon as it is published. Watch out for something special when the BBC Homepage re-launches in the very near future.

Matthew McDonnell is Head of Search, BBC.

Take a look at the latest version of the BBC's search service.

For the first time, relevant BBC TV and radio programmes are included in your search results alongside news articles, other BBC web pages and links to external websites.

About a third of the most popular searches on the BBC seach engine are for programmes. While selected programme websites have always been available in search, you will now find two new types of results:

• programmes that you can watch or listen to in BBC iPlayer

• and episode pages for all programmes broadcast on radio or television in the last eighteen months (since we began /programmes our new approach to publishing programme information).

Most of these episode pages will not let you watch or listen to programmes, but you can read about that episode's storyline and cast, and in the future these pages will contain richer information including clips, like this.

The search results pages have also taken on the new BBC style and we are experimenting with new ways of displaying the results. I think it looks great - let us know your thoughts.

I am going to post about this in more depth in a couple of weeks - once we have some performance figures and feedback.

This is the first of a series of planned improvements to search in the next few months, so any observations from you are very welcome indeed.

And for anyone who wants a more in-depth view of how BBC site search works, one of our senior software engineers, Andy Webb, will be posting in a couple of days.

Matthew McDonnell works in Search & Navigation, Online Media Group, BBC Future Media & Technology.

Topics are automatically updated web pages, each one covering a different person, country or subject.

This is a beta release so forgive us a few rough edges. They will be smoothed out over the course of the trial.

We wanted to make it as easy as possible to create and maintain pages for large countries as it is for small ones, whether they are currently the location of big news stories or not. Or to build pages for a politicians, famous people and historical figures that alert users to relevant programmes and news stories they might otherwise miss. Or to showcase the best and latest things the BBC has produced about topical issues, historical events, big organisations and popular hobbies. And we wanted to produce a page all aboutdogs.

So /topics uses a variety of search techniques to create feeds of the latest BBC content from news articles, programmes available to watch on iPlayer, weather forecasts, news videos, country profiles and information from the TV and radio schedules. BBC editors then add in hand-picked articles and features from around the BBC and other websites.

Stephen Betts of our technical team will post in the next few days giving more detail on how the feeds are generated and the pages assembled as well as giving an insight into the engineering approaches.

I will of course reply to your comments on a later post, but there are a couple of questions that I can answer up-front.

Why are we doing this?

While it is quite easy to find all the news about, say, anti-social behaviour or all on-demand programmes about India, it has always been more of a challenge to find everything the BBC has recently produced about a topic. Now we can make pages that organise the full range of BBC content around selected subjects like this and this.

And because the overhead involved in maintaining these pages is so low, we can cover many more subjects than we could using traditionally edited pages which had to be manually updated by a human being. As the feeds used in /topics are automatic, we can be confident that all the pages are bang up-to-date.

Finally, these pages will exist forever regardless of whether there is new content being produced about the subject (even if the odd page may look a little bare now and again). So, for the first time, it's possible to link to one permanent page for the topics that we cover on the BBC. At the moment we have a selection, and the beta trial will help us determine the scope of what we might do in the future.

How do we choose which topics to cover?

We take three things into account. Firstly, what people are looking for (we do this by analysing the BBC search logs). Then, what we have content about (we aren't aiming to create an encyclopaedia of all human knowledge; rather to better organise and showcase our content) and lastly, the BBC's editorial priorities.

For the moment a few countries, people and subjects have been chosen to get things off the ground and to test the technology. The list will grow over the coming months.

If it's all automatic, what about editorial priorities and tone?

No new content is produced for our topic pages, so we know that everything we link to meets editorial guidelines. Topic page editors can also add and remove content if necessary.

BBC journalists and content producers use their judgement to assess their importance of their stories and content every time they write a story or publish some content. We're working on systems that will capture that wisdom, infer the relative editorial importance of a piece of content (for instance by checking whether it appeared on the News or iPlayer homepages) and then use this information to influence the ordering of content on the pages.

Is it just going to be News and iPlayer content on these pages?

Far from it. We are developing a BBC blogs feed and planning Sport and other feeds. In addition, we are working on sourcing widgets of content from other areas of the BBC including things like the recipe database.

We are trying out some of these ideas on the topics page for China. Pete Clifton has blogged in more detail about China on the BBC News Editors Blog.

What about related content from other websites?

We want to include high quality content from outside the BBC to enhance our pages. We'll be working on providing feeds of news and blogs from sources other than the BBC.

Can I get the feeds and build them into my own website or personal feeds?

Yes, feeds will be available soon.

This is just the first stage of the project and new features are planned over the course of the next couple of months. We will keep you posted as the service evolves.

Finally for enduring the long days and short tempers that are an inevitable part of product development, I would like to thank all of the talented engineers, editors, designers and project staff who have worked so hard to get us to this stage.

Have a play around with /topics and leave me a comment.

Matthew McDonnell is Portfolio Executive, Search and Navigation, Internet Group, BBC Future Media and Technology.