As some of you will have noticed, we suffered a major failure of BBC Online last night. The site started to fail at 20:10, and by 20:25 was completely down. It stayed down until 21:10, when it started to recover, and by 21:30 the site was back. Some of you may then have experienced problems accessing some pages between 21:55 and 22:10 as we restored full resilience, and from 22:10 onwards we were back to full operation.

The problem was caused by a failure of the traffic managers in both our data centres.

These traffic managers are a critical part of our infrastructure, responsible for handling all requests to the site, and routing those requests to the right servers. They are designed to be highly reliable, and have served us very well to date.

We are still investigating the root cause of this incident, and I would like to apologise for any inconvenience that this outage may have caused. We are working hard to make sure that the causes of the issue are addressed, and that this does not happen again. I will keep you updated on this blog in the coming days.

Richard Cooper is Controller of Digital Distribution, BBC Future Media

Like everyone else in the UK involved with bringing the games to a digital audience, the BBC is preparing for the increased amount of people watching and engaging with events online. Our comprehensive live online coverage will offer Olympics audiences, for the first time ever, the chance to watch every Olympic sport, live from every venue - from the big events, to minority disciplines which normally wouldn't be broadcast.

As with all big sporting occasions, our traffic will vary during the day depending on the event, competitor, team and timing. Where we need to prepare most is around the big moments - for example, when Team GB is (hopefully) in the running for a medal - which have the potential to drive a lot of people to our online streams. This is not new for us. The BBC has proven it can cope well with big event streaming: in 2010 we brought the World Cup live to online audiences, and dealt with increased traffic comfortably.

We did this by expanding our streaming capacity, which we do every year to accommodate inevitable increases in people watching our content online and across connected devices. We've already expanded our capacity this year in preparation for the Olympics. As in 2010, we've ensured this increase will enable us to handle the high levels of traffic we expect to see during the games.

We're aiming to deliver more content from the Olympics than ever before, live and on catch-up, and we're fully prepared to meet the demands of that pledge.

Richard Cooper is Controller, Digital Distribution, BBC Future Media

This has come into sharp relief this year, with the last remaining blocks of IPv4 addresses being allocated by the Internet Assigned Numbers Authority (IANA). These addresses pass down to the Regional Internet Registries (the one that applies to the UK is RIPE NCC), which then allocate them to ISPs and other companies that need them.

The internet won't stop working on the day that the last IPv4 address is allocated, but a solution is required to allow the internet to continue to expand.

That solution is IPv6.

IPv6 provides a mind-boggling number of addresses (3.4 x 10^38). It's hard to find a meaningful analogy for a number this large, but if every man, woman and child on Earth had a billion devices each with an IPv6 address, you haven't even come close to scratching the surface of the number of addresses available.

Unfortunately, IPv6 is not backwards compatible with IPv4, so there's a lot of work required before end-to-end IPv6 operation becomes the norm. In practice, this means that IPv4 and IPv6 will have to co-exist with each other for a long time. As content providers, content distribution networks, network operators and equipment manufacturers introduce IPv6 there are plenty of opportunities for problems to appear that interfere with a user's experience of using the internet.

And that's where World IPv6 Day comes in - it's a 24 hour "test flight" intended to motivate relevant companies to "prepare their services for IPv6 to ensure a successful transition as IPv4 addresses run out".

In the spirit of this, the BBC is also making BBC Online available over IPv6. We're doing it in two ways:

1) We've created a new address: http://ipv6.bbc.co.uk. When you try to access this address over an IPv4 network, it simply won't work (your browser will produce an error saying something like "The requested URL could not be retrieved"). However, if you access this address over a network that supports both IPv4 and IPv6 you will see the home page as normal.

2) We've also, just for today, made the usual BBC Online address (https://bbcstreaming.pages.dev) enabled for both IPv4 and IPv6. You shouldn't notice the difference, with the site working as normal for you. In fact, unless you're a bit of an expert, you won't be able to tell whether you're accessing the site over IPv4 or IPv6.

Very few of you will have an IPv6-capable network. If you want to try accessing the BBC's site (or any of the sites participating in World IPv6 Day), you may find that the only way, today, is to set up an IPv6 tunnel (which allows you to make an IPv6 connection over an IPv4 network). There are a number of companies that provide this service, some of which are listed here.

Under the hood, we still have a lot of work to do at the BBC to make BBC Online fully IPv6 capable.

To make the site available over IPv6 to the extent that we have, we've added a separate (much smaller) network that is both IPv6 and IPv4 enabled, and connected it to the site through a pair of load balancers that we normally use for testing purposes. The host name ipv6.bbc.co.uk only has an IPv6 address registered on it, and this is the IPv6 address of the load balancers. Through this you can access pages on the BBC Online site, but note that most objects referenced by these pages are on other domains that are IPv4 only (so to see the pages correctly you must be on a both IPv6 and IPv4 enabled network). Our DNS is IPv4 only, as is our IP geolocation system that we use to serve different editions of the site as appropriate to your country, and restrict access where we only have content rights for the UK.

We'll be upgrading the site to be fully IPv6 enabled as part of our normal technology refresh programme, and we expect that will be complete sometime next year.

Meanwhile, what we've done for today allows you to see if you can make an IPv6 connection, and for us to see the problems that we have to solve and determine the work remaining for us to do to be fully IPv6 enabled.

Many thanks go to colleagues in Future Media, R&D and Siemens who pulled this all together.

Richard Cooper is Controller, Digital Distribution, BBC Future Media

Between the hours of 0200-0500 (the quietest time for BBC Online), the equipment that failed on Tuesday night will be replaced, and if you happen to be on the BBC website at that time, you may experience some disruption in service.

Here's what's happening, in case you want more details.

Part of the network will be shut down to enable the faulty equipment to be replaced, and this will result in some route re-convergence as the core network works out the best path from the hosting centres to the internet. This will happen twice: once when the equipment is shut down, and a second time when full resilience is restored. Each re-convergence could take a few minutes, and while this is happening you may experience some interruption to service.

Despite some reports, the technical teams within the BBC and Siemens worked very well on Tuesday night to rectify the situation swiftly and effectively. There are incident management processes in place to handle failures such as this one, and while we hope not to have to use them too often (!), this was a good example of close collaborative working.

Richard Cooper is Controller, Digital Distribution, BBC Future Media.

Our systems are designed to be sufficiently resilient (multiple systems, and multiple data centres) to make an outage like this extremely unlikely.  However, I'm afraid that last night we suffered multiple failures, with the result that the whole site went down. Enough of the systems were restored to bring BBC Online pretty well back to normal by 23:45, and we were fully resilient again by 04:00 this morning.

For the more technically minded, this was a failure in the systems that perform two functions.  The first is the aggregation of network traffic from the BBC's hosting centres to the internet.  The second is the announcement of 'routes' onto the internet that allows BBC Online to be 'found.'  With both of these having failed, we really were down!

We'll be taking a very hard look at what we need to do to make sure that this doesn't happen again.

Richard Cooper is Controller, Digital Distribution, BBC Future Media.

In a nutshell, a 160 core fibre trunk (a bundle of cables) was cut.

Normally this would not have affected what you would see on the BBC's websites, but in this case the resilient routes did not take up the strain in some cases. The consequence was that we lost a number of our services for a few hours until the fibres were re-spliced.

The main thing that you may have noticed was some of our content becoming stale as updates were delayed. We also had interruptions to many of the live audio and video streams on bbc.co.uk.

So we have a lot of work to do to ensure that next time we have a significant cut like this (these things happen), the resiliency works and there are no interruptions to our services.

In the meantime, we'd like to apologise for the problems.

Richard Cooper is Controller, Digital Distribution, Future Media & Technology.