Black Hat: An interview with Dan Kaminsky, the DNS dude who saved the Internet

Dan Kaminsky showed up at the Black Hat conference in a Pac-Man T-shirt and jeans. But he was the man of the hour at a presentation yesterday that held 1,000 people spellbound during his ninth talk in 10 years. The 29-year-old self-described DNS guy talked about the flaw he discovered earlier this year and managed to keep secret as security researchers prepared a patch for it, thereby allowing the Internet to avoid a train wreck. Kaminsky covered a lot of ground about how he discovered the flaw in the Internet’s Domain Name Server infrastructure for keeping the addresses of web sites and how it can be tricked into sending users to fake web sites. He was about to collapse with relief and take a nap. But I caught him for an interview.

VB: What’s a good way to describe this bug?

DK: We always knew we were doing bad. We knew there was a one in 65,000 chance we were hosed. But we thought that you could only try to attack this once a day. When you have 65,000 days, that’s a lot of protection against someone using a random attack. We held back a paper that was coming out. But this thing that limited how many times someone could try this attack wasn’t secure. There were 15 ways around it. Some didn’t work on every attempt. With this new way to attack, someone could randomly attack 65,000 times in ten seconds. It became easy for the bad guys to win and redirect users to fake pages.

VB: Do you feel a bit like a celebrity security researcher here?

DK: Look. There was an unusual amount of noise necessary to get this thing fixed. If you find a bug that affects this many people, you have to do three things to protect them. First, you have to find it. Finding it was easier. Not that much time investment. Then you have to get the bug fix written. Since this one crossed so many company boundaries, we had to figure out how to get all of these companies to talk to each other. Then we really brought in a lot of people and made it clear how important this was and got that working. But that’s not enough. We don’t have really good resources to patch the infrastructure. And this was an infrastructure flaw. So we had to do the third step. That was to evangelize outside of the security community. The security community is not the one doing the patching. I was not asking security researchers to patch anything. I was asking the network operators and engineers. Guys, there is a problem with your network. This last month has been all about getting network engineers enough information so they can protect their users. We built a patch. We needed them to test it. We didn’t want them to take down their networks while they did it. But we also didn’t want their emails getting misdirected to China (or any other place where bandits might misdirect them).

VB: How did you get started in security?

DK: I got offered a really boring job at Cisco.

VB: What led to that?

DK: I’ve been a geek for a long time.

VB: What was your first security-related experience?

DK: I discovered that the way that everyone was signing up for printers gave the school code-execution access to everyone’s desktop. I was really bothered by that. I built a system called Docsprint. You wouldn’t log into the school and thus be exposed to code execution. Instead, you would send data to my server and my server would log into the school. For a little while, I had half of the print jobs in the dorms in Santa Clara University going through my dorm room. That was a lot of fun. I showed up at Cisco to get a job being a network monkey. I was working on stuff that was undocumented and it was a crappy language. It looked like there were things wrong with the code. So I just started fixing bugs. I started studying security. Really getting into it. Reading all these books. I got an email from a friend named Ryan. He said I was a good writer (I did debate in high school). He asked me to write a chapter in a book. He said my name wouldn’t go on it. It was on “hack-proofing your network.” I didn’t tell anyone. That year there was a security treasure hunt. You either won a T-shirt or a ticket to Black Hat. I’m not actually kidding. I went for the ticket to Black Hat. I was already coming for Defcon. The book was there and it had my name on the cover. Not only that. It had Dan Kaminsky of Cisco Systems on the cover. I hadn’t told Cisco I was writing a book with their name on it. Who was this college intern? I went to a panel with Mudge on it. He was one of the oldest hackers on the scene. He asked a question on a panel. I answered it. He asked how old I was. I said “20.” He said, “Never tell anyone how old you are. If you do, they will believe only so much of what you have to say.” That was my first Black Hat, and now this one is my tenth.

VB: So how old are you?

DK: I can’t tell you that. I’m 29.

VB: What was it in your background that made it more likely you would find this bug?

DK: I had been doing Domain Name Server (DNS) research for a long time. This is my third talk on it. My first talk was all about it and how I could tunnel data over it, store data in it and use it as a general communication channel. Last year, I talked about how I could use DNS interactions with web browsers to get around firewalls, basically using DNS to manipulate the web browser. I wasn’t looking to break DNS. I was trying to build a new content distribution network. I’m trying to build a new CDN. It seems like a great idea. I need a way to move someone from a slow server to a fast server. Can I use DNS? Well, I had this tool from last year’s talk. If I used it, I’d have to deal with the thing called the TTL (Time To Live, or a built-in delay), which makes it too slow. I thought about that. It was supposed to be slow to protect against cache poisoning. But it totally worked. My jaw was agape. I contacted Paul Vixie, who has been doing DNS for 20 years and is the president of ISC, the makers of BIND. I said, “Paul, we got a problem.” It was bad. We started contacting people we know. Everyone was on board and very helpful.

VB: Wasn’t it hard to find 16 people you trusted with this information?

DK: The DNS community is small. We all had things to lose.

VB: People understood the implications of it?

DK: There was no confusion. This breaks everything. It’s embarrassing how many things this breaks. We should not be in a position where a bug this simple causes this much damage.

VB: You mentioned that this flaw has been around since 1983 in the infrastructure. Does this mean the Internet needs a fundamental makeover?

DK: Security did not exist as a design consideration in 1983. Not even 1993. It wasn’t until the late 1990s that people saw we are investing a lot of money in this and this is where bad guys could steal assets and cause damage. There has been a lot of retrofitting of security on top of the system. But at the end of the day, we are addicted to using the Internet in unsecure ways. Every time you use the Secure Socket Layer on the web browser, there is usually something broken. It’s getting better in some ways but a lot worse in others.

VB: You said that the bug has broader implications. Why?

DK: It’s fascinating. There is no code execution because of this. But the accuracy of DNS is important to everything. When you undermine it, you mess up everything. The initial push on July 8 was to point out that all clients browsing the web are not necessarily seeing what they think they see. And not all emails go to where they are supposed to go. That was the first domino here. Those are bad enough for security. That was all I needed for the information technology world to be concerned. The rest of the dominoes are all the other things that are affected. DNS is at the heart of the infrastructure. SSL requires you to get certificates. That’s how it works. But you get the certificate via DNS and you get it via email. If that’s corrupted, SSL doesn’t work securely.

VB: You also said that the “forgot my password” feature on almost every login page is also compromised. Can you explain that?

DK: What happens when you forget your password? You don’t prove anything. You just give your email and they send you a link to reset the password. They don’t validate who you are. The email either has the password, a link or maybe it will ask for more information. What if that email goes to someone else? You’re hosed.

VB: You said it was lucky a security researcher found this and not someone else.

DK: Serviceability is one of the most ignored aspects of security. That is one of the strongest lessons I learned in the last year. You have to expect that a piece of your infrastructure will fail and you have to be prepared to fix it rapidly. People did heroics here. But that doesn’t scale. We need a process where we get a warning and need as many days as possible to figure it out. People really worked on fixing this with a patch. I believe that people should take this into account when they are shopping for infrastructure technology. The system should make it easy for patching to happen. Infrastructure that can be patched in eight hours instead of 90 days will be more likely to survive an attack. The difference between a random hacker and the security professional is the awareness of disaster planning and mitigation.

VB: More crticial flaws are going to be found?

DK: The money is too good. So yes. One of the talks here is about the opportunity. This bug was very monetizable.

VB: There is evidence that attacks are in progress?

DK: We know that something went down in Austin. We believe it was with Google and was a case of click fraud. But we’re awaiting more data. There is stuff going on that I can’t talk about. I have to say that I have data and we’re still analyzing it.

VB: But you’re confident that industry-wide collaboration will yield great results in the future?

DK: I do think we have a model that works for fixing something like this. I don’t think we did it perfectly. There were noticeable gaps in peer review, and we missed the NAT and firewall vendors. That was an entire industry we missed. We should have had them involved because we were changing their nature. But you can’t argue with the numbers. Today, 120 million broadband customers are protected. About 84 percent of the computers that were tested on my web site were vulnerable at first. Now it’s 30 percent. That’s what we got as a result.

If you liked this Q&A, please check out our others:

Byron Acohido, author, “Zero Day Threat”, on who to blame for identity theft

Bob Aniello, marketing chief at THQ, on mass market video games

John Antal, chief of staff and military/historical director at Gearbox Software, making “Brothers in Arms: Hell’s Highway”

Wagner James Au, author “The Making of Second Life”, on life in a virtual world

Robbie Bach, president of Microsoft’s Entertainment & Devices Group, on Zune

Robbie Bach, president of Microsoft’s Entertainment & Devices Group, on Xbox 360

Robbie Bach, president of Microsoft’s Entertainment & Devices Group, on Windows Mobile

Mark Bernstein, CEO of the Palo Alto Research Center, on life beyond Xerox

Jeff Boyd, CEO of Miles Electric Vehicle, on the future of cars

Lucy Bradshaw, vice president in charge of production at EA Maxis, on making Spore

Jim Crowley, CEO of Turbine, on keeping the online game machine humming

Gonzague de Vallois, senior vice president of publishing at Gameloft, on iPhone gaming

Vinod Dham, father of the Pentium, on a life of technology and venture investing

Cammie Dunaway, sales and marketing chief at Nintendo’s U.S. unit, on broadening the game market

Jon Goldman, chairman Foundation 9, on game development as a model

Seth Goldstein, CEO Social Media, on social networking’s future

Mike Gallagher, head of the Entertainment Software Association, on E3 2008

Bing Gordon, former chief creative officer, Electronic Arts, future partner at Kleiner Perkins, on leaving EA

Todd Howard, executive producer of Fallout 3, on making a big role-playing game

Mark Jacobs, general manager of EA Mythic, wants to go to WAR

Steve Jurvetson, partner at Draper Fisher Jurvetson, on the cleantech revolution

Max Levchin, CEO of Slide, on social networking

John Lilly, CEO of Mozilla, on the hybrid nonprofit-for-profit business model

Marissa Mayer, vice president for search at Google, on social search

Paul Marcoux, Cisco vp of green engineering, on making data centers energy efficient

Ray Muzyka and Greg Zeschuk, co-founders of BioWare, on making great games

David Nordfors, director of Innovation Journalism program at Stanford, on teaching new journalism

PopCap Games top executives Dave Roberts and John Vechey, on making games fun

Steve Perlman, CEO of Rearden, on funding R&D for startups

Ted Price, head of Insomniac Games, on expanding a high-quality game development studio

Jeff Pulver, VOIP pioneer, on the future of voice

John Riccitiello, CEO of Electronic Arts, on EA’s focus on higher-quality games

Gordon Ritter, Emergence Capital, on software-as-a-service

Henk Rogers, Tetris pioneer, on saving the earth

John Schappert, corporate vice president at Microsoft’s game division, on changes to Xbox 360 gaming

Curt Schilling, founder of 38 Studios and Boston Red Sox pitcher, on starting a fantasy online game

Dwayne Spradlin, CEO of InnoCentive, on expanding R&D crowdsourcing

Bill Watkins, CEO of Seagate, on the storage business and roughing it with Eco Seagate

Shu Yoshida, president of Sony’s worldwide game studios, on being an underdog again

Strauss Zelnick, CEO of Take-Two Interactive, on managing creativity

Mark Zuckerberg, CEO of Facebook, on hiring an outsider as COO