How Pakistan knocked YouTube offline (and how to make sure it never happens again)

This graph that network-monitoring firm Keynote Systems provided to us shows the worldwide availability of YouTube.com dropping dramatically from 100 percent to 0 percent for over an hour. It didn't recover completely until two hours had elapsed.

(Credit: Keynote Systems)

A high-profile incident this weekend in which Pakistan's state-owned telecommunications company managed to cut YouTube off the global Web highlights a long-standing security weakness in the way the Internet is managed.

After receiving a censorship order from the telecommunications ministry directing that YouTube.com be blocked, Pakistan Telecom went even further. By accident or design, the company broadcast instructions worldwide claiming to be the legitimate destination for anyone trying to reach YouTube's range of Internet addresses.

The security weakness lies in why those false instructions, which took YouTube offline for two hours on Sunday, were believed by routers around the globe. That's because Hong Kong-based PCCW, which provides the Internet link to Pakistan Telecom, did not stop the misleading broadcast--which is what most large providers in the United States and Europe do.

This is not a new problem. A network provider in Turkey once pretended to be the entire Internet, snarling traffic and making many Web sites unreachable. Con Edison accidentally hijacked the Internet addresses for Panix customers including Martha Stuart Living Omnimedia and the New York Daily News. Problems with errant broadcasts go back as far as 1997.

It's also not an infrequent problem. An automatically-updated list of suspicious broadcasts created by Josh Karlin of the University of New Mexico shows apparent mischief--in the form of dubious claims to be the true destination for certain Internet addresses--taking place on an hourly basis.

So why hasn't anyone done something about it? False broadcasts can amount to a denial-of-service attack and, if done with malicious intent, can send unsuspecting users to a fake bank, merchant, or credit card site.

To understand why this is both a serious Internet vulnerability and also difficult to fix requires delving into the technical details a little.

How to pretend to be YouTube.com
When you type a domain like "news.com" into your Web browser, it uses the Domain Name System to cough up a numeric Internet address, which in our case is 216.239.113.101. That IP address is handed to your router, which uses a table of addresses to figure out the next hop toward the news.com server.

Network providers--called autonomous systems, or ASs--broadcast the ranges of IP addresses to which they'll provide access. One of the functions of the Internet Corporation for Assigned Names and Numbers is managing the master list of AS numbers, which it does by allocating large blocks of 1,000 or so at a time to regional address registries.

Kim Davies, ICANN's manager of route zone services, says ICANN isn't able to revoke the AS number of a misbehaving network provider. "It's best to think of them as similar to post codes or ZIP codes," Davies said. "We maintain a registry of them to ensure that they aren't conflicting."

If the address information provided by AS is reliable, all is well. But if an AS makes a false broadcast, because of a configuration mistake or for malicious reasons, all hell can break loose.

This is what happened with YouTube, which Pakistan's government ordered blocked because of offensive material, apparently a video depicting the cartoons about Muhammad that had been posted in a Danish newspaper. Some reports have said the video featured several minutes of a film made by Dutch politician Geert Wilders, an outspoken critic of Islam.

A spokesman for the Pakistani embassy said on Monday that the order to block access to YouTube came from the highest levels of the government. It would have been passed along to Pakistan's Electronic Media Regulatory Authority and then to Pakistan's telecom authority, the spokesman said, which in turn would have issued the formal order to the Internet providers.

Pakistan Telecom responded by broadcasting the false claim that it was the correct route for 256 addresses in YouTube's 208.65.153.0 network space. Because that was a more specific destination than the true broadcast from YouTube saying it was home to 1,024 computers, within a few minutes traffic started flowing to the wrong place.

A timeline created by Renesys, which provides real-time monitoring services, says that it took about 15 seconds for large Pacific-rim providers to direct YouTube.com traffic to the Pakistan ISP, and about 45 seconds for the central routers on much of the rest of the Internet to follow suit.

YouTube took countermeasures within minutes, first trying to reclaim its network by narrowing its 1,024 broadcast to 256 addresses. Eleven minutes later, YouTube added an even more specific additional broadcast claiming just 64 addresses--which, under the Border Gateway Protocol, is more specific and therefore should overrule the Pakistani one. Over two hours after the initial false broadcast, Pakistan Telecom finally stopped.

How could this have been prevented? First, Pakistan Telecom shouldn't have broadcast to the entire world that it was hosting YouTube's IP addresses. Second, Hong Kong-based PCCW could have recognized the broadcast as false and filtered it out.

An employee of PCCW, who wished to remain anonymous because he is not authorized to speak for the company, said that as soon as the false broadcast occurred, PCCW started receiving a flurry of phone calls from global ISPs wondering what had gone wrong. A YouTube representative also called.

Even Pakistan Telecom contacted PCCW. "I don't think they understood what was going on," the employee said. A spokesman for PCCW's U.S. operations, based in Herndon, Va., declined to provide details.

At the moment, large network providers tend to trust that other network providers are behaving reasonably--and aren't intentionally trying to hijack someone else's Internet addresses. And errors that do arise tend to be fixed quickly by manual intervention.

But as the number of suspicious broadcasts grows, and the potential for fraud increases, so does the justification for more aggressive countermeasures. (Besides, some government will eventually order its network providers to broadcast false information about the Internet addresses of "offensive" Web sites. We've already seen domain name blocking in Finland and Web page blocking in the United States, both supposedly enlightened Western democracies.)

One way to handle this is for network providers to be automatically notified when the virtual location of an Internet address changes, which is what some researchers have suggested in the form of a "hijack alert system." Another is to treat broadcasts with changes of addresses as suspicious for 24 hours and then accept them as normal. Simple filtering of broadcasts may not always work because some networks provide connectivity to customers with thousands of different routes.

Probably the most extensive countermeasure would be a technology like Secure BGP, which uses encryption to verify which network providers own Internet addresses and are authorized to broadcast changes. But Secure BGP has been around in one form or another form since 1998, and is still not a widely-used standard, mostly because it adds complexity and routers that understand will add additional cost.

At least that's been the conventional view. A high-profile incident like YouTube being knocked offline may accelerate this process, said Steven Bellovin of Columbia University. "I know there are serious deployment and operational issues," Bellovin said. "The question is this: When is the pain from routing incidents great enough that we're forced to act? It would have been nice to have done something before this, since now all the world's script kiddies have seen what can be done."