The typical response, particularly within the BBC, would be a suggestion to re-purpose much of the infrastructure we already have: media ingest, metadata management, transcoding, web publication, device targeting.

There was a snag, though. In fact, there were a couple. First, this wasn't just a pop-up TV channel - this was a "broadcaster in a box", which could later be handed over to arts organisations to pick up and run with. Second, we had to have as little impact upon the rest of the BBC as possible (it turns out that 2012 is quite a busy year for the Olympic Broadcaster).

And it had to go live on the 1st May 2012.

We had to figure out not just how we could get this thing up and running, but how we could do it - and document it - well enough so that unskilled yet motivated people who'd never been anywhere near a traditional broadcasting operation would be comfortable running, allowing arts organisations to play a part in an emerging digital cultural space.

My answer to the question was to build a website targeting what the BBC tends to call these "four screens", but with room for some native applications to augment it.

I didn't plan on actually doing it, though. I was just giving him an opinion. I figured that this would be one of a range of options considered and, given the stretched resources, it would end up being outsourced to a video-on-demand specialist to rapidly put together and operate.

That didn't happen. It turned out that in that brief question/answer exchange, the phrase "'pop-up' television channel" didn't really capture the breadth and depth of what was going to be attempted. This wasn't just going to be another video-on-demand service: it needed to be inherently flexible and able to take a range of media along with specialist propositions from cradle to screen. The ultimate objective wasn't "to run another BBC service", but to have a toolkit containing a "broadcaster in a box". We weren't out to create the next BBC channel, we were capturing the essence of what a broadcaster is and does.

A group of us sat in a small meeting room on the seventh floor of Television Centre to discuss the options we had. We'd spoken to video-on-demand specialist suppliers, and at that time the combination of timescales, required platform flexibility and potential audience reach meant that this route just wasn't feasible. If we weren't going to use an existing platform, somebody would have to build one - and at that point I still didn't think it would be us.

As we reached a quiet moment in the meeting, with everyone in the room considering our limited options, Jake Berger bit the bullet: "why don't we just do it ourselves?"

Another tricky pause, before Tony asked me, "so… if we were to do it your way, how would it work?"

And so, in that room, I sketched out how I would go about building what came to be named "The Space". It went a little something like this:

Take a well-known easy-to-use open source content management system (WordPress), and put it onto Linux machines firewalled to the hilt. Add plugins to it to in order to generate a completely static version of the site, which is then sent over to public-facing Apache web servers or a CDN (content delivery network), and build a set of templates to present the different kinds of media we're presenting.

So long as the CMS is extensible, we shouldn't have too much trouble storing all of the required metadata and relationships. Because the public-facing servers are dealing solely with static resources, making it scale cost-effectively is relatively easy, and because it's not built on any BBC-specific systems, it can be hosted anywhere and subsequently handed over when "our part" of the project ends. And there, in the space (no pun intended) of a few minutes, I'd sketched out the basis of how The Space would - and now does - work.

Jake, Dirk-Willem van Gulik and I were asked to come up with some numbers and a plan that was a bit more tangible than my handwaving in the meeting room. Between us, we came up with estimates of how which kinds of servers we'd need, when (and for how long), how we'd handle the design and markup of the templates, the video encoding, the technical operations, and committed our planned architecture to paper.

From my rather vague idea, Jake added his own project management expertise and Dirk his in-depth knowledge of BBC operational processes and we ended up with something we believed to be realistic and achievable, and that we could send on to the project board for consideration. A little later on, we tasked Jon Stuart, drawing on his experience with the BBC's audience-facing online systems, to refine the numbers and figure out ways by which we could deploy and manage the proposition.

We produced a small working model which gave us an indication of cost against projected given audience size, the number of live events and anticipated hours scheduled. Tony then brought in Vibeke Hansen, who had designed the original iPlayer with him, to come up with the the look and feel of The Space. After some very quick work by Vibeke, Caroline Smith and Nick Clement we had a design we could let our client-side developers loose on. Paul Coghlan and Stephen Collings went to work building and testing templates across a range of connected TVs, mobile devices and desktop browsers which would put the Blue Room to shame, while Aaron Dey, Steve Allen, Robert Gummeson and Dagmara Kodlubanski started putting together an ingest and transcoding chain. Meanwhile, we had the help and advice of Brandon Butterworth and colleagues in BBC Research and Development, particularly around live streaming, and Alex Russell from BBC Distribution to help us with our Freeview HD interface.

One of the dirty secrets of broadcast engineering is that you always assume that things will go wrong - because, in reality, no matter what you do, they always do. That's why everything at the BBC has five layers of backups and contingency planning… but having that degree of fall-back also requires the twenty-four-hours-a-day seven-days-a-week operational support of a major broadcaster. The Space certainly wouldn't have that. It needed to be able to stand alone.

We solved this by doing two things.

First, we captured the essence of our existing online practice and experience, by liberally (and in some cases literally) copying a lot of the provisioning and deployment scripts, approaches and processes that keep the BBC on the Internet. At the same time we have also automated (and now documented) a lot of the knowledge needed to run all of that.

Second, we took the opportunity to refine and rebase our approach. The BBC has been on the Web for quite a while now, and over time its legacy infrastructure has grown, and grown… and grown. Now, this isn't only about hardware - there aren't huge city-sized data-centres dedicated to keeping the BBC online - but anybody who's gone digging through bbc.co.uk will know that it sometimes feels like cutting through an old tree and counting the rings. The older pages were developed to entirely different processes - and are today still hosted today using quite different setups to that of the present systems.

As an early mover, the BBC has invented a lot of complex technology and approaches. The most successful of these are now appearing in hosted services or with cloud providers with features and refinements which we often find we're often unable to adopt ourselves. In short: we were able to cherry-pick the best parts of BBC online infrastructure and get rid of the bits we didn't need, and properly explore how something which technologically looks not a million miles away from the stuff powering bbc.co.uk today can be operated and managed in an off-the-shelf cloud hosting environment.

While we did this, Bill Thomson and Simon Still kept a constant feedback loop going with those commissioning work and mentoring the commissioned artists.

Today, we have a small back-office system which lets us create and transcode content (for all 4 screen sizes and a multitude of browsers) and send it off to cloud-based storage acting as a CDN origin. The content management system allows arts organisations themselves to submit metadata for their pieces, and for The Space's editorial team to review, edit, arrange and publish it.

Yet even after such a breakneck journey, this is still just the beginning (and I have to keep reminding myself of that): The Space is operational within the BBC until October, at which point the real hand-over happens, and then in many ways it's down to Arts Council England to continue operations if they wish.

In the next few weeks, I'll be following up on this post with a more detailed and technical look at how we customised WordPress to handle a content model not a million miles from /programmes, how we bent Apache to our will, and - of course - whether our approach paid off.

Meanwhile, you can visit The Space on your desktop, tablet, mobile, or connected TV with a web browser by visiting http://thespace.org. If you have a compatible Internet-connected Freeview HD television or set-top box, you can access our MHEG service - powered by the same content management system, and developed by specialists S&T - on channel 117 (you may need to re-scan to pick up the new channel).

Mo McRoberts is a Data Analyst in the Digital Public Space project and Technical Lead of The Space. Tony Ageh has also blogged about this partnership with Arts Council England.

Since those posts, our efforts have been focussed upon delivering the technology platform for a joint project with Arts Council England called “The Space” which will be available between May and October this year.

However, we haven’t lost sight of the vision for the Digital Public Space project, and I wanted to share with you a piece of work which has come from that.

As a short-term measure, we were able to take catalogue data in whatever form they were able to provide it in and convert it through a semi-automated process to RDF linked data sets, but this was a fairly resource-intensive process and far from ideal. A longer-term goal was for partners to be able to publish these sets themselves.

To help with this, we’ve produced two short guides which we’ve distributed to partners. The first covers the basics of constructing good identifiers for things and the mechanics of publishing data in a variety of formats (including transparently publishing data alongside human-readable web pages); the second is about publishing data sets specifically.

Both guides seek to capture best practice, and aren’t intended to lay down hard-and-fast rules. Because they’re applicable to anybody wishing to publish linked data, not just the Digital Public Space project partners, we’re making them available to download for anybody who might find them useful.

Naturally, we welcome feedback on these guides, and will look to incorporate suggestions into any future versions that we produce.

• Download “Crafting URIs and Publishing Data” (PDF, 241kB)
• Download “Publishing Datasets” (PDF, 126kB)

Mo McRoberts is an analyst in BBC Archive Development

Bill Thompson and Matthew Postgate chat at Over the Air 2011. Picture by Paul Clarke, used with permission.

This Friday and Saturday I was lucky enough to have been asked to speak at Over The Air 2011, held at Bletchley Park — home to the team of code-breakers who cracked the Engima and Lorenz ciphers during the Second World War.

The topic of my talk was our prototype implementation of the Digital Public Space. Last week, The Guardian’s Tech Weekly podcast went into detail about the aims of the project and talked with Richard Ranft of The British Library and Francesca Franchi of The Royal Opera House, as well as my colleague Bill Thompson, who is Head of Partnerships here in BBC Archive Development.

I’m pleased to say that my talk seemed to be well-received, with a good twenty minutes of Q&A from an interested audience, covering various areas that the project including approaches to digitisation, rights and the expression of metadata.

Over The Air is billed as “36 hours of mobile development” and much of the time spent there by developers involved taking full advantage of the glorious weather and spacious grounds, hacking on prototypes to be shown off on Saturday afternoon.

On the Friday evening, Bill Thompson kicked off the first ever Ignite Bletchley Park talks with a whirlwind tour of the nature of reality itself, why the notion of “augmented reality” doesn’t make sense, and how the Digital Public Space fits into our evolving sense of what constitutes “real”.

Having slaved over their hacks for a day and a night, come Saturday afternoon it was time for the developers to show them off and be judged in several categories. A number of the hacks were pitched squarely at the Bletchley Park Challenge — a plea to developers to find a way to replace the audio tour “wands” which visitors carry with them as they visit different areas of the estate.

This challenge was of particular interest to us as a demonstration of the ways in which physical cultural assets and structured, machine-readable data about those assets can be brought together to create new kinds of journeys and make our cultural heritage accessible to wider audiences.

The challenge was won by Michael Deales for his entry “Bletchley Park Guide”. In other categories related to the concept of the Digital Public Space, Team WhyMCA won in the best android category for “Hack the Mansion” — a geeked-up version of the classic board game, Monopoly; and Melinda Seckington won the prize for most cultural entry with “MuDo's”, a museum to-do list. While not a winner, Cristiano Betta did show off a version of his “Mobile History of the World in 100 Objects” webapp — something which is, of course, close to our hearts. He’d originally built the app as part of History Hack Day earlier in the year, and has released the code on GitHub.

Of course, no event would be complete without a big finish! Over The Air’s came courtesy of Ewan Spence (accompanied by some pyrotechnics), who had over the course of the weekend built a KerPlunk-styled Enigma machine out of bits of wood, some garden hose, plastic cups, and… some of those little silver balls of fondant that people use as cake decorations.

Mo McRoberts is a Data Analyst in the Digital Public Space project. Tony Ageh, the BBC Controller of Archive Development, leads the project.

Yesterday, the BBC's Director of Future Media, Ralph Rivera, gave a speech at the newly-opened offices of the World Wide Web Consortium (W3C) in Oxford.

His speech emphasized the BBC's support for the organisation and its philosophies in the context of the BBC's work on a 'new broadcasting system' that can reach everyone, is free at the point of use and makes BBC programmes available to all those who can benefit from them. The speech also discussed the ways the BBC is seeking to get the maximum value from its archive and asked the audience 'what good is it to retain this archive it if can't be shared?' before describing the 'digital public space' within which the BBC now sees itself as operating as it delivers its services online.

As Ralph noted, the digital public space can mean different things to different people. To some it's a philosophical ideal, the belief that UK citizens have the right to access and interact with the countries social and cultural assets online. To me in my role as Data Analyst within the BBC's small Archive Development team it's something very specific.

I and a couple of colleagues work on the Digital Public Space project. This is a partnership between the BBC and other cultural institutions in the UK, including museums, archives, libraries, galleries and educational bodies, all of whom share a vision of not simply using Internet technology as a distribution channel, but instead being part of that digital environment as it evolves: being part of the Web, rather than just on it.

It aims to be an access point for all of the UK's cultural archives, marrying together both the rich information which has been carefully collated, checked and double-checked over the years by experts in their respective fields, with the more immediately-accessible higher level information and audio-visual material, both from the partners and around the Web.

The first step along the way in achieving this is a prototype which is being developed that brings together the archives and catalogues of some of the partnering institutions (including the BBC's) within an 'Umbrella' data model and creates a platform on which applications and interfaces for navigating, annotating and curating them can be built. Eventually, you would be able to access and add to this information through an online gateway, but there could also be specialist entry-points.

For example, there might be an iPhone or Android app for exploring the history of your local area, or a YouView interface focussed on "British Ballet". Part of what makes the project so exciting is that we really don't know what kinds of interfaces and applications will end up being developed for the platform.

The Semantic Web lies at the very heart of this. It provides the toolkit for describing real-world things in a machine-readable way, just like ordinary web pages describe those things in a human-readable way. Like the "Web of documents" we are generally used to, the Semantic Web is built on the fundamental principle that anybody can publish anything about anything else, without having to go through layers of bureaucracy and paperwork. Even the language used to describe these things -- RDF -- uses vocabularies which are often developed independently of one another, and come into existence by being published somewhere on the Web, and having RDF documents begin to use them. There is no central "ontology authority" who decides what does and doesn't form part of the Semantic Web's vocabulary: if there isn't an ontology in existence which is able to describe the things you need to describe, there's not much, beyond time and effort, standing in the way of you creating one.

Within the digital public space prototype, RDF gives us a common language that institutions can use to describe their catalogues in their own terms. The prototype aggregates these catalogues, finding areas of overlap, and presenting the things described by them in a unified manner, not organised in terms of the catalogue entries that are best suited to archivists, but instead in terms of the people, places, events, things and collections which those entries describe.

First and foremost, the aggregated information is itself published as RDF. Being intended for consumption by software, RDF isn't terribly exciting for most people to look at, so as part of the prototype we're also developing a number of user interfaces to explore different ways in which the catalogues can be navigated.

The aggregation engine doesn't have any special knowledge about the partnering catalogues, though. As far as it's concerned, there's no fundamental difference between an expert institution and anybody else. There's a language for making statements about things (RDF), a way of identifying the things in the catalogues (URIs -- of which what we know as "Web addresses" are a subset), and a way to publish those things (the Web).

There are some practical hurdles to be overcome, however.

With institutions, it's quite easy to mandate that the software that feeds catalogue information to the aggregator must push RDF documents to a RESTful Web service, using a digital certificate which provides a strong identity so that the information can be attributed to them. For individuals, things get a little more complicated. We know that user interfaces can be built to take care of the heavy lifting of generating RDF and pushing it to the aggregator, but that still leaves problems with certificates -- most people don't really use public-key cryptography on a day-to-day basis, and so we need to settle upon an approach to identity that everybody can get to grips with.

Beyond that, there are aspects of RDF which haven't been finalised yet -- attaching digital signatures to different parts of an RDF document, and specifying the source of a set of statements ("named graphs"). With all of these issues, we're looking forward to working with the Web community to find solutions.

You're probably wondering when you'll get to experience the digital public space, and in particular this prototype. The answer is "it depends". This phase of the project is due to end in June, at which point we will have something tangible that can be shared amongst select individuals in the partnering organisations, to act as a proof of concept. While the details have yet to be finalised, we hope that the next stage after that will be to make it available to everybody in each of those organisations on a permanent basis. If that's successful, then we are looking to open it up to the many schools, colleges and universities in the UK.

As you can imagine, the legal and rights issues surrounding both the catalogue information and associated digital media are complex and varied, and navigating them means working closely with rightsholders and industry bodies, and will take some time. However, the BBC remains committed to the aim set out in Putting Quality First ("Opening up the BBC's library of programmes") -- and this is a vision shared by all of the project partners -- of providing permanent access to the UK's cultural archives in a digital environment that's available to everybody.

We know that the digital public space can only become a reality if we build on open technologies and standards as championed by the W3C -- the digital environment in which we're creating this already exists, and so co-operation and partnership is absolutely key to the success of the project.

Mo McRoberts is Data Analyst, BBC