Summary:
An internal routing issue developed in our network between our edge routers and our core distribution routers.
Diagnosis and Resolution:
During regular network maintenance Blacknight staff were moving a customer from the shared vlan to their own VLAN. During this move we were forwarding IP packets from their old IPs to their
new IPs. Normally this should not be a problem. However in this case, the rules caused OSPF on the primary distribution router to flap.
As a result of the session flapping the secondary router was not able to take over correctly. We manually failed over to the secondary router and at this point the network stabilised.
We are still investigating this issue, but we believe a router upgrade that is planned for later in the year will fix this issue permanently.