June 17, 2004

Service Outage

If you're reading this, it's because we're back up after several hours of T1 downtime. We went down hard at 1050hr PDT, and I opened a Speakeasy ticket at 11:05.

They, in turn, opened a Covad ticket after it was clear that there wasn't a software problem, and Covad got an ILEC dispatch out here at about 1400hr PDT. The SBC tech said that our new Speakeasy T1 was not provisioned cleanly when it was first installed; that there was a split (tap) between the NOC and the neighborhood box, and that the circuit had dropped below tolerance.

After some head-scratching, I convinced him that he didn't need to provision a new set of pairs for us because we had a perfectly good set from the old XO T1 that was no longer in use. He was (to his credit) very, very reluctant to reuse pairs that still had voltage across them, but I showed him the disconnect order that was executed June 1, and he agreed to swap the pairs. This worked; service was restored at 1530hrs PDT.

Since the Speakeasy circuit went live, I did notice some minor errors in the DSU interval logs which are explained by this theory; however we are not sure what caused the (previously usable) signal to suddenly go bad. I do not expect a recurrence of the problem.

Posted by eric at June 17, 2004 04:32 PM

Post a comment