EGAD

Internet is Down

Grr, the internet is hard to use tonight.

Some quick probes indicate that it's the Level3 network links in Chicago making trouble. According to postings on the NANOG list the entire Level3 backbone collapsed at 2:00 EDT sharp, suggesting that maintenance or an upgrade somewhere backfired. Probably router trouble; given that only one company's network is affected, but the outage is affecting basically all of North America, it's clearly not a fiber-vs-backhoe type of issue.

[Update: 2:25 CDT -- Indeed, it's actually worse. I just saw a note complaining that backbone links in Europe are going down, too.]

Okay, while I've been writing this, the network is starting to stabilize, at least in the Midwest, which is what I can directly probe. Which is mostly to say, there is at least one working router back online in Chicago. Probably due to many poor network engineers frantically backing out whatever they changed that broke the internet.

NANOG traffic indicates that things are coming back to life in some places, while elsewhere the problem is getting worse. Like ripples on disturbed water. Looks like the 'net will have sorted itself out by morning, though. It's late, so I'm turning in.