{"id":9910,"date":"2017-11-06T15:22:12","date_gmt":"2017-11-06T21:22:12","guid":{"rendered":"https:\/\/thinkerventures.com\/article\/\/"},"modified":"2017-11-06T15:22:12","modified_gmt":"2017-11-06T21:22:12","slug":"level-3-issues","status":"publish","type":"post","link":"https:\/\/thinkerventures.com\/level-3-issues\/","title":{"rendered":"Netflix not working properly? It’s not just you"},"content":{"rendered":"

Comcast suffered a strange semi-brownout today here in the Rockford area. A full outage is easy to diagnose: Your Internet just doesn\u2019t work. This outage was weird, as the issue was between Comcast and a third of the Internet, and took about 1\u00bd hours to resolve. Afterward, Comcast seemed to point blame at Level 3, a Colorado-based Internet interconnection company that connects service providers.<\/span><\/p>\n

Level 3 is one of those companies that are crucial to a smoothly operating Internet \u2014 and therefore a smoothly operating workplace \u2014 but few people know because we never see its name on a bill.<\/p>\n

Our first notification came when our telephone servers, hosted at Google, lost connection with their handsets all over Rockford. Our monitoring software checks the connection between every server and a VoIP phone onsite with our clients. Whenever a problem arises, an email is fired off for us to investigate, and a second email is sent once it\u2019s resolved. We quickly determined that the issue was localized to Comcast by calling one of the associated telephone numbers and hearing the correct outgoing message. This meant the phone servers were operating correctly.<\/span><\/p>\n

During this stage, the Bose music system here at Thinker continued playing Spotify and Gmail continued to work, but all my SSH sessions into respective servers closed down. We discovered that all of our servers with Google were unreachable, but all of the servers with Amazon were fine. \u00a0<\/span><\/p>\n

Our phone servers are generally configured to offer an option to call people by extension and even have links to a phone book. Most extensions fail over to cellphones, so people could still be reached.<\/span><\/p>\n

At this stage, my next tool is the great Down Detector website, <\/span>http:\/\/downdetector.com\/status\/comcast-xfinity<\/span><\/a>. The website was, however, unreachable by Comcast. \u00a0A quick look using my phone showed us where the fault lay.
\n\"\"<\/span><\/p>\n

Using my phone\u2019s hotspot mode, we determined that our websites at Google were working fine for the rest of the Internet and our phone servers continued to work, so we alerted a few of our clients to the issues before they realized what was happening. At this stage, we also discovered that Facebook was unreachable. With Facebook down, we realized that Comcast would be well aware of the outage and would be prioritizing a fix.<\/span><\/p>\n

We used our online monitoring tools to keep track of which phone extensions across town were disconnected and watched them come back online (at first, with very high latency, which then returned to normal speed over the next hour or so).<\/span><\/p>\n

By 2 p.m., the outage was mostly resolved in Rockford. Some reports are indicating that Comcast has bypassed Level 3 for the time being. One-third of the Internet traffic normally goes through Level 3, so you may continue to experience some services, such as Netflix, loading much more slowly until the issue is fully fixed. Essentially, it would be like the state shutting down Interstate 90, forcing all of us to take U.S. 20 to Chicago.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

Comcast suffered a strange semi-brownout today here in the Rockford…<\/p>\n

Read more<\/a><\/p>\n","protected":false},"author":5,"featured_media":9913,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"episode_type":"","audio_file":"","cover_image":"","cover_image_id":"","duration":"","filesize":"","date_recorded":"","explicit":"","block":"","filesize_raw":"","footnotes":""},"categories":[13],"tags":[3],"_links":{"self":[{"href":"https:\/\/thinkerventures.com\/wp-json\/wp\/v2\/posts\/9910"}],"collection":[{"href":"https:\/\/thinkerventures.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/thinkerventures.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/thinkerventures.com\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/thinkerventures.com\/wp-json\/wp\/v2\/comments?post=9910"}],"version-history":[{"count":0,"href":"https:\/\/thinkerventures.com\/wp-json\/wp\/v2\/posts\/9910\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/thinkerventures.com\/wp-json\/wp\/v2\/media\/9913"}],"wp:attachment":[{"href":"https:\/\/thinkerventures.com\/wp-json\/wp\/v2\/media?parent=9910"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/thinkerventures.com\/wp-json\/wp\/v2\/categories?post=9910"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/thinkerventures.com\/wp-json\/wp\/v2\/tags?post=9910"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}