On 10:04am on 4/2/2008 an ethernet card failed in a device on one of our metro-e providers Layer 2 connectivity device in DEG. Immediately (within 50ms) our kit failed over to our backup route into DEG. There was no service disruption during this window due to our resilient network design. At 12:00 the card was replaced and this link came back up and we flipped our traffic back over to our primary link. Again service was unaffected.
We received the RFO from our metro-e provider yesterday afternoon that basically said what I’ve described above. A card failed and it was replaced within 2 hours.