Owning IOS by Michael Lynn

1 August 2005 16:14 UTC

How are you going to ship out patches when every router is dead?

Michael Lynn used to be an employee of Internet Security Solutions. He looks the typical stateside programmer - a bit worse for wear for the sedentary job and too much junkfood - but he's also quite the hacker and now a whistle blower too.

Cisco routers run the Internet. They're assumed to be extremely reliable and many people wrongfully assume they're totally hardware. But the Cisco routers are extremely dependent on software and in particular their own operating system IOS (Internetworking Operating System).

IOS is a careful critter. It continually checks to make sure there's nothing wrong, no intrusions, no memory mess-ups, and so forth. If it detects something is not as it should be, it automatically sets the router to crash down and come up with a fresh reboot. Minimal loss of time and connectivity.

So far so good.

But there are a lot of things going on concurrently in a router and it is possible more than one check routine sees it's time to shut down and start again. And so if the router interrupted whatever it was doing - eg crashing - to crash, things would take forever.

Thus the crashing_already_ flag.

crashing_already_?

When a Cisco router decides it has to do the Phoenix thing, it first checks the crashing_already_ flag. Any routine given the go-ahead to crash and reboot the router will set this flag. If a subsequent routine finds this flag set, it backs off: the router is already going down, so there's no reason to start the process again.

So far so good again - except for one small problem.

It's possible to hack the system and trick it into setting the crashing_already_ flag without the system really being set to go down. Then you have a small time frame where you can insert your code and spread it to other routers - and on the Internet, at that level, things spread like wildfire.

Which is what Michael Lynn demonstrated at the Black Hat Briefings on 27 July in Las Vegas.

It could be a 'digital Pearl Harbour', he said, and asked rhetorically:

How are you going to ship out patches when every router is dead?

Lynn also showed the assembled hackers how to 'wipe their tracks' once they'd hacked into the system. He claims he came forward, and was aware he would have to resign from ISS, because it was a matter of 'national security'.

But neither ISS nor Cisco appreciated his gesture. Within hours of his speech he was formally relieved of his duties at ISS and served with a walloping lawsuit.

Cisco Powered Backlash

But the news traveled fast on the Internet - thanks to the Cisco routers - and opinion was not in favour of ISS and Cisco, Bruce Schneier calling them 'thugs'.

Schneier also pointed out that trust in Cisco will not be great after the event.

Cisco had already released several patches to fix the flaw; ISS and Cisco have also settled with Lynn, essentially withdrawing their suit if Lynn says no more.