Reflecting on the ‘great summer outage’…
The major BT outage from the summer just gone epitomises the struggle that all MSPs and resellers face. The ability, or lack thereof, for a connectivity giant to lose all connectivity for more or less a full day, not face any real consequences and not seem to make any real effort to take responsibility is incredibly frustrating.
What was the issue?
As a result of a major outage at a London BT data centre on the 20th and 21st of July, over 10% of BT customers across the UK were unable to access the internet. A number of internet banking customers, including Lloyds, HSBC, Barclays and Natwest, were also affected by not being able to access their online accounts. Although BT was not disclosed as being the ISP, it has since become industry knowledge that this is the case.
According to the London Internet Exchange (Linx), the cause of the outage was due to the Harbour Exchange data centre owned by the California-based data centre operator Equinix and came just over six months after the data centre involved was taken over from Telecity Group. Although Equinox declared that the issue only affected a small number of customers and was solved within minutes, the repercussions had an effect on BT for the majority of the day. BT’s issues turned out have been caused by a 20 minute power cut at a data centre in London’s Docklands.
According to Equinix, the internet did not just ‘stop’ — over 80% of traffic continued to flow and the connection dropouts began to recover even before the power was restored. Linx’s 700 members include major internet providers such as BT and Virgin Media, as well as other content providers. Indeed, other operators in the same building also lost power to their equipment, however, did not face the same outage levels as BT. Virgin Media said it had received no reports that it had been affected by the outage. It is sure to say that the incident severely showed up the vulnerability of the internet’s architecture in the UK.
What are the implications for Brigantia partners?
The vast majority of Brigantia partners provide connectivity services to their customers and so unfortunately the implications of this vast outage effected many. Because BT own most of the UK’s broadband infrastructure, this issue was unavoidable (as most of the leading banks also discovered). Without connectivity, many MSP and resellers’ smaller customers cannot conduct their day-to-day business functions as a result of the reliance on FTTC and, in some cases, ADSL connections.
Although largely unavoidable, unless the customer has the budget or requirement to lease a private connectivity line, the role of the MSP is still important in ensuring confidence and managing the situation as best as possible. Risk management, couple with the expected elements of a coherent disaster recovery and data backup plan should of course be second nature. However some MSP and resellers may consider reviewing those customers most heavily effected and reassessing their needs as a business. Not only could this open an opportunity to sell a more high-end connectivity solution, protecting by strong SLAs and 99.9% uptime guarantees (available via Brigantia), but this will also develop a stronger element of trust with regard to the business relationship. Of course, failure to repair the phone lines and broadband will enable customers to claim to BT for compensation, however there appears to be no provision for consequential losses — merely service credits. As an example the BT Business Broadband compensation scheme of will pay out “£25 per fault fixed late”.
Ultimately, most MSPs and resellers will just have to accept what conditions are on offer. BT are big enough that they can dictate to suppliers how things are done. With the summer outage, it may be a supplier’s problem, but BT are not in a position to use that as an excuse because they’ve either audited the setup and were happy with it, or they didn’t audit it in which case they can’t be said to have done due diligence. Either way, from a PR perspective, it’s still BT’s name in the headlines.
What is being done by regulatory bodies?
Ironically, the outage came just a day after BT’s Openreach internet infrastructure business was criticised by MPs over its alleged failure to “adequately” invest in fast fibre. Going forward, one of Ofcom’s key pledges concerns what happens if broadband users are affected by breaks in service. Currently, users have to lodge a formal complaint with their provider and then escalate that complaint to the Ombudsman after eight weeks if they’re not happy with the response – with no guarantee they’ll get any compensation. Instead, Ofcom says it will “seek to ensure that, if things do go wrong, consumers and small businesses receive automatic compensation for any loss of or reduction in service”. According to Ofcom, this will provide a significant incentive for providers to improve service and fix faults quickly.
Ofcom has since published its consultation on automatic compensation for consumers and smaller businesses who experience disruption to daily activities caused by problems with their communications services. It was emphasises in the report that there is a need to provide consumers with a significantly better quality of service. Introducing new rules on automatic compensation when consumers experience problems was a key measure for Ofcom to take action on. It is worth noting though that these new measures would only impact a small proportion of consumers and businesses. As stated in the consultation document, the vast majority (80% of fixed line, broadband and mobile consumers) are satisfied with their services and typically consumers consider that their services are working as expected.
Building on measures introduced by Ofcom in 2014, Openreach will be held to higher minimum requirements to repair faults and install new lines more quickly, as well as improve performance more generally, for example by reducing the number of faults. These new tougher minimum standards are due to be brought in later this year. Ofcom will also publish detailed performance tables ranking broadband providers by service quality – which is intended to help users make better informed decisions about which firm to go with. And Ofcom will also provide clearer coverage info, adding broadband to the mobile coverage map it introduced last year./