The moment when the Globe to Globe production of Venus and Adonis finally finished transcoding just after midnight on May 1st.

Last August the BBC and Arts Council England formed a partnership to deliver a new platform for digital arts, The Space.

My team started to build it during August last year, and we launched thespace.org on May 1st, delivering a multimedia, multi-platform,cross-genre, global arts service that supports video, audio, articles, image galleries, games, interactive applications and live streams and is available on smartphones, tablets, computers, smart TVs, and Freeview HD (Channel 117 if you want to have a look).

It was a bit of a rush.

From nothing a small group of us created a design and user experience, developed a content management system based on a very customised Wordpress instance, coded responsive design templates to work on a variety of screen sizes, built a transcoding system, procured a cloud-based content management and content delivery network, went through a cycle of user testing, contracted two live streaming suppliers, procured a bespoke VOD (video on demand) application for Freeview HD, launched a Freeview HD channel, and went live with a global site that works on more-or-less any modern browser and quite a few older ones.

After forty days in my own special wilderness, I'm coming up for air after what were certainly the most exciting days of my life at the BBC. On a few occasions I've found myself wondering what on earth I was thinking on the fateful day last year when I muttered 'why don't we build this ourselves' to Tony Ageh and Mo McRoberts, but I don't regret any of it.

We took on the task knowing it would at best be very difficult and at worst prove to be impossible. There have been many late nights and early starts. So many screens, so many refreshes. So many contracts, so many lawyers. So many cables, so many encoding profiles.

Launch Day

On the night before launch, Paul Coghlan, Dirk-Willem Van-Gulik and Jon Stuart had been working their way through some last-minute video transcodes, builds, changes and fixes; Mo McRoberts was online and about to hop on the train from Glasgow to London; and I was waiting for the transcoding of the Globe to Globe production of Shakespeare's Venus and Adonis in six South African languages to complete. And we'd just watched the News At Ten announcing our go live the next morning, meaning that an audience of millions would be expecting us to be there when they woke up...

The picture at the top of this post shows the progress window of our custom built local transcoder based on the open source ffmpeg/x264 encoders and Manzanita transport stream multiplexer to provide all the formats needed for both web streaming and Freeview HD MHEG broadcast. It runs on bootcamped Win 7 iMac, and turns a single HD video file into six profiles for all web devices and TV, and then uploads it to the cloud.

It shows the moment when the final go live video transcode queue completed after midnight, meaning we had a full set of content for launch. I'd been waiting for it to complete for hours.

The site went live when Mo took down the holding page on thespace.org. He actually did it at about 0130 on May 1 from the dining car of the Glasgow to London Sleeper train, just after he passed Carlisle. He'd promised it'd happen before Carlisle, but I was willing to forgive him.

I arrived home in the early hours, excited by the knowledge that we were live, but aware of the 'known unknowns' and 'unknown unknowns' that we would face in the coming months.

The following morning I attended the launch event at the Royal Festival Hall, bleary-eyed and unshaven, and then headed to Television Centre and checked with Alex Russell that Freeview HD channel 117 was working, before firing up the array of devices in the office to make sure all was well across our promised platforms, devices and browsers.

We were getting 800 requests per second during much of the morning, and the cloud hadn't complained one bit. Jon Stuart had done well.

What Do You Reckon?

Soon after launch we started to receive a variety of messages from people who had come to The Space to have a look around. These offered a healthy mix of general and specific positive feedback; and a smattering of snags, bugs, and suggestions. But the response from around the globe was overwhelmingly positive, which made our efforts over the previous months feel totally worthwhile

We don't have a system for logging compliments, but we do have one for logging bugs. You can email us with your bugs at 'contactus@thespace.org'. Every single issue is logged on JIRA, and every single one will be addressed. We need users of thespace.org to help identify bugs and make suggestions for future enhancement. This is invaluable to us, so I thank all of you who have done so, and ask that you continue.

With a small team with a constant flow of stuff to do, prioritisation of these issues is one of the hardest parts of our job. Our mission is to provide a good experience across all devices, browsers, operating systems, in the midst of unpredictable OS and browser upgrades.

And we are striving to make The Space as usable as we can for users who have sight, hearing, cognitive and motor accessibility requirements.

So, how did it go? We had over 250,000 users in the first week. Within the first three days we managed to get a good experience on 95% of browsers. And so far we have had only one hour when part of the site went down. I won't tell you whose fault that was because I'm a nice person.

Coming Live from The Space

As well as the recorded performances and interactive artworks we have also hosted seven live streamed events, ranging from a two hour performance of Britten's War Requiem from Coventry Cathedral to three sets of seven hour streams over three days from Vanilla Galleries.

These required close and sometimes hair raising coordination across my team, the venues, and third party suppliers to the venue and to The Space. We worked Skype group messaging to the bone. If I ever need a reminder of what working to a live broadcast deadline feels like, I have those chat logs to refer to. Steve Allen worked magic with the live events, facing chaos and adversity with levels of calm only found in someone who is planning to launch a lunar explorer in his spare time.

The Space Team

My team are the real stars in the firmament of The Space, and I knew we could pull it off once I'd pulled this small group of very talented people together, gave them a mission, asked them to trust each other, and added a bit of fuel and the occasional drop of oil.

Mo, Paul and Steve have been absolutely heroic, and without them there would be a void rather than a space.

Vibeke Hansen and Caroline Smith continue to evolve and enhance the user experience and design of The Space. You'll see some significant changes appear in the next month or so.

Aaron Dey made all of our machines talk to each other, despite them being from many different generations and being across the OS divide.

Robert Gummesson keeps the transcoders firing on all cylinders, and has recently configured them to wrap subtitle files in to the video profiles.

Jon announced he was moving to Vietnam, but pointed out that we rarely saw him anyway, and that we wouldn't notice the difference, and he continues to keep the machines humming away nicely in the cloud.

And last but by no means least, Dirk-Willem Van-Gulik and Brandon Butterworth - our 'tech overlords' - keep an eye on the big picture and keep us connected to the rest of the BBC's technology folk.

Running The Space

The team that operate The Space - Hilary Bishop, Sally Taft, Mike Osborn, Ana Lucia Gonzalez and Dora Somerville - turn the empty vessel that we have built into a fresh offering every day, using the amazing variety of material that we receive from the commissioned arts organisations. They greet with joy the green 'queue completed' message on our transcoders and fear its evil twin, the red 'failed to upload' error...

We've made a few substantive changes recently, with new releases every week, and we have many more up our sleeves as we make our way towards the end of phase one in November. We've upgraded our search function quite substantially, and have started to create collections pages to house the ever-increasing volume of material. We'll also be implementing page templates that offer a greater number of items and build on our accessibility features.

We hope you continue to enjoy The Space, and i thank you again for the comments and suggestions. My favourite so far comes via Facebook: 'pretty much the best thing online EVER'.... which is over-egging it a little, but gives us something to aim for in the best way we can.

PS, iOS users - we've found that in several cases, users need to clear their cache and / or delete and reinstall the web-app icon on iOS to pick up the new code and be able to enjoy the fixes and features that we have deployed!

Jake Berger is Head Of Technology and Distribution, thespace.org and Programme Manager, Digital Public Space, BBC Archive Development

The remains of the Metroon. Photo by bokur.net, used under licence.

The story begins about 2,500 years ago, in Athens.

Around 500 BC in the Ancient Greek city-state of Athens the state archive was housed in a building called the Metroon, or ‘mother building’. This temple, dedicated to the goddess Demeter, was filled with papers relating to the day-to-day civic, legal, commercial and cultural life of its citizens.

The Metroon was open to every citizen, and all were entitled both to read and to make copies of anything held there, giving them a level of access to the building blocks of their society that is unrivalled in the modern age despite our Freedom of Information laws and open data initiatives.

Today, there are simply too many Metroons, even if we had permission to enter all of them. The vast majority of current archives remain undigitised and available only by visiting a physical building.

But the bigger challenge is that public archives are run by organisations and institutions that have collected, crafted and labelled their archives to meet their operational needs rather than general access. Gathering a comprehensive or authoritative set of materials from different archives becomes an Olympian task: there is simply so much stuff in so many places described and recorded in different ways using different systems.

Digitisation does not solve this problem, since the different databases are not necessarily compatible. Even though we are entering an age of mass digitisation, the challenges in achieving Athenian levels of access to digitised material are many, massive and varied.

Fortunately, the emergence of the semantic web and linked data practices and standards means that a modern Metroon – a digital public space – is, at last, a possibility.

I'm Jake Berger, the Programme Manger for the Digital Public Space project. Essentially this means that I have to work out and describe the scope and challenges of the overall vision, then try to break down the big challenges of the wider Digital Public Space project into smaller ones and work with my BBC colleagues and external partners, (like those mentioned by Bill earlier in the week) to get these smaller projects off the ground. I also try to make sure that our project’s thinking aligns with other thinking around the BBC and beyond

Mo McRoberts introduced the Digital Public Space project in a blog post in April.

In summary, our ambition is to create an online space in which much of the UK’s publicly-held cultural and heritage media assets and data could be found - connected together, searchable, machine-readable, open, accessible, visible and usable in a way that allows individuals, institutions and machines to add additional material, meaning and context to each other’s media, indexed and tagged to the highest level of detail.

A couple of weeks ago, Bill Thompson talked to Jemima Kiss about the Digital Public Space project in the Guardian’s TechWeekly podcast. When Jemima Kiss described DPS as a big library Bill explained how the shared metadata would work:

The stuff’s not in the library. You have the best catalogue ever, and when you want something there’s an instant delivery service. Those organisations that want to keep their material in the library can do so. Those that want to keep it to themselves because they’re worried about rights issues or whatever can keep it to themselves and only make it available when they’re asked for it to people they’re sure will look after it.

Bill wrote earlier this week about how the BBC was working with partner organisations, and you can see a visualisation of how the partners, the catalogue, the assets, and the products and services all fit together on Bill’s post.

Data model and reference implementation

As Mo wrote, an ‘umbrella’ data model is being developed. This brings together a number of catalogues - data sets describing the holdings of a range of partners - classifying their contents in a consistent way, identifying themes and types, and mapping out connections and associations across diverse data sets.

The data model is lossless - it does not attempt to truncate or simplify any of the extensive detail within partners’ catalogues. There may be elements or fields within each catalogue that do not currently have an obvious connection to any other field in any other catalogue, but the ‘lossless’ approach ensures that any such connections can be found and mapped in the future.

So far the data model can encompass catalogue information from the BBC, British Library, The National Library Of Scotland, The Royal Opera House, Royal Botanic Gardens at Kew, The National Archives, The Science Museum and The Arts Council’s Arts On Film collection.

Early versions of this data model indicate that - as hoped - there will be many, varied and often unexpected journeys that can be made through these catalogues and the material they describe. For example, a user starting out by watching a film of a production of Macbeth from the Royal Opera House might then look at a scan of a rare musical manuscript from The National Archives, then browse similar manuscript scans held at the British Library, watch a clip from a BBC documentary about how paper was produced in Shakespeare's era, before ending up learning about the plants used to make the paper using information from The Royal Botanic Gardens At Kew. In a DPS, all of this could happen in the same online space.

Clearly, a ‘critical mass’ of data and material needs to be brought together before we see such innovative journeys emerge. So we are now beginning to assemble this critical mass.

Screen grab from a Metabroadcast prototype to help folk navigate a large dataset.

Dealing with Complexity

The project throws up a lot of complex questions and we have already started projects to address some of them.

Bill recently mentioned a collaboration between JISC and BBC Northern Ireland, which combined digitised video with tools to search and tag it for research or teaching.

There are a whole series of challenges around the user experience and navigation through vast and diverse data sets. We have worked with Metabroadcast to try out a couple of approaches to this

Mo blogged about the development of a web browser-based user interface, which navigates through these catalogues using the concepts of “people”, “places”, “events”, “things” and “collections”. I soon hope to share some other User Interfaces that we’ve developed within the Archive Development team

Of course these are only the beginning; soon we will launch projects with other collaborating institutions which will explore issues around rights, identity, access, privacy, provenance, persistence, user–generated content and data, augmentation and amplification.

As an insight into these issues, it will be a large task to administer huge numbers of rights holders when the current rights situation is so complex. Earlier this year a study by our Rights and Business Affairs Department, submitted to the Hargreaves Review, revealed an average of 85 separate rights-clearance transactions per episode of Doctor Who.

Another question is to track contributions, usage, and amendment whilst preserving the privacy of contributors. This leads to the question of provenance and how we can preserve this information over time.

When more projects to address these complex questions are off the ground, I will blog about them too.

Rome wasn’t built in a day. Neither was Athens, and nor will the Digital Public Space. But, I hope that the blueprint we are beginning to develop, the plan that will deliver it and the Digital Public Space itself are as valuable to every modern-day citizen as the Metroon was to the citizens of Athens.

I look forward to blogging again when these projects have delivered, as part of consulting our partners and audience over the next steps.

Jake Berger is the Programme Manager for Digital Public Space in the BBC Archive Development.