Wednesday 2nd February 2022

DCN1 - Network Issue

Our monitoring systems have alerted us to an issue affecting our core network at DCN1 (Nottingham) - we are currently investigating. Connectivity to servers within this facility may be intermittently affected, we apologise for any inconvenience this may be causing.

We will provide updates regarding this issue as they become available.

== Update 16:18 ==

We are continuing to investigate the incident and deploy a solution to the network connectivity issue.

== Update 16:40 ==

Service has now been restored, we are monitoring the situation and starting work to identify the root cause and produce a report.

== Investigation==

We became aware of an issue with one of our core firewalls, which resulted in disruption to network connectivity to our DCN1 (Nottingham) facility. As part of that investigation we performed diagnostics and resolved an identified issue, this brought the firewall back online. However, we quickly identified that this only restored partial service, further investigation of our core network identified that the fault had cascaded to one of our core distributions switches. We initiated a forced failover across our other core switches, restoring network connectivity.

The core distribution switch is still isolated from the network, and we are going to conduct an analysis of its logs to determine any underlying fault that would prevent us it being placed online. If no fault is found we will return the switch to our network and confirm routing of traffic is behaving as expected – therefore, we are undertaking emergency maintenance at 00:01 Thursday 3rd February, we do not anticipate any further disruption to connectivity.

We apologise for the inconvenience this incident has caused, and a full RCA will be produced made available upon request.