Interview with a Ninja, Part I

It's a strange world we live in now, isn't it? We communicate through machines as often, and with more different people, than we do in person. Some of us have careers that didn't exist when we were in college. Some of us even have careers that involve doing things we used to do secretly, possibly even illegally.

Consider the professional penetration tester, who breaks into his clients' systems and applications for the purpose of testing and improving their security. To me, this is one of the most remarkable professions a person can have in information security.

Some people are skeptical of the value of this line of work. What does it prove? If penetration testers succeed, have they really learned more than, or even as much as, they would have learned by performing a careful analysis of system and application configurations, and maybe by analyzing some source code? If penetration testers fail, did they simply have a bad day, or is the system in fact secure?

Although I'm sure there is such a thing as a redundant or otherwise unhelpful penetration test, I can assure you that in my own role as a network security architect, there are plenty of situations where a rigorous, professional penetration test is essential. Source code isn't always available, configurations don't always tell the full story of system behavior, and some things are too new for there to be any “conventional wisdom” of how secure they are, or how they can be made secure. A good penetration test can help me determine which of a vendor's claims hold up, and which system administrator practices are being adhered to.

Therefore, I work closely with and proudly count as friends some outstanding penetration testers. Luckily for you, gentle reader, this month, one of them, who for our purposes here I'll simply call Ninja G (for reasons that will be revealed shortly), graciously agreed to a Linux Journal interview, in which we discussed his singular career, what he finds interesting in computer security, and what he sees as notable trends in the current state of Linux/UNIX security.

MB: Thanks so much for taking the time to chat. By way of an introduction, can you explain what your official duties are and how you spend a typical day?

NG: I work in the finance industry for a Fortune 500 (or less) company. The duties of those in my group include penetration testing, enterprise-wide network security scanning and real-time security event response. My days vary greatly from one week to the next, which is a wonderful thing about this sort of work. One week I am testing a small embedded Linux device, the next week it could be a remote server or Web site, corporate Wi-Fi or VoIP network.

I believe this classifies me as a generalist, so it probably isn't surprising that a large part of my days are spent coming up to speed on the target platform. This can include securing logical or physical access, reading all the vendor-supplied documentation, researching all currently known and historical vulnerabilities and then actually using the platform or device (as intended) to gain a holistic view of how things normally work. Only then can I start to ask myself all of those evil little questions concerning how the designer decided to handle situations that (under normal circumstances) were never intended to happen.

MB: How did you arrive at this kind of a job? I'm assuming you didn't major in penetration testing in college!

NG: College? Heh, no. I was an illicit computer/phone hacker during the 1980s and early 1990s. I don't have a criminal record, but I can't say that I was never caught; rather, at one point I was the focus of a multi-agency investigation. I never was prosecuted, but my “close call” officially ended my desires to intrude illegally into systems as a hobby.

It was at that point that I started looking into jobs in the Information Security field. I was hoping to find an outlet where I could continue the passion of my past hobby, yet without the risk of prosecution. The problem was trying to find just the right company that not only had an Information Security department, but also found value in beating up proposed solutions to look for potential security problems. I knew that large financial organizations had an interest in this, but unfortunately, I lived in small town, which was exactly the wrong area for this type of work.

So I “followed my bliss” and ended up in a large city, working for a financial company whose products are offered globally. I wouldn't say that my first job in this industry was exactly what I wanted, but often your job is what you make it, so I slowly started revealing my skill level in the arcane “hacking” arts. Word gets out, rumors spread, and very slowly more and more of the work that interested me started to come my way. Fast-forward 16 years, and I'm working for a larger financial company in an even larger city. Now the bulk of my time is spent on penetration testing, so I can't complain at all.

MB: What do you find most interesting in your job and your career? What sorts of problems are the most fun, and why?

NG: Learning new things every day. Difficult problems are always the most rewarding.

In this line of work, you are only as good as your last major find, and people tend to forget quickly about past (now-mitigated) vulnerabilities. It isn't uncommon to go for days without stumbling upon new vulnerabilities of any significance, which is horribly depressing. Perseverance at these times often cuts deeply into my own personal time, but it almost always pays off.

All it takes is discovering one new zero-day (vulnerability), and I'm back on top of the world—for a day or so. Then, the process starts all over again with something else.

As a generalist, I really don't have any preference on what types of devices, software or systems I test. Each presents their own lessons and potential for discovery.

MB: WLAN, as a target, seems to be especially popular in the criminal community. Does WLAN stand out, one way or another, in your own work, or is it just the same types of problems and vulnerabilities, only over radio waves?

NG: No, I would say that wireless presents its own unique set of security ramifications and even heavily trained corporate WLAN administrators sometimes don't comprehend concepts as simple as malicious server certificate re-use. If you're unfamiliar with this style of “insider” attack, it can be summed up in one sentence. An authorized Web server administrator creates a malicious (evil twin) wireless access point that mimics the corporate WLAN (and which they load with the valid certificate from their Web server from the same name-domain); however, they configure it to merrily accept (and log) all authentication credentials provided, regardless of what they are.

For those wanting to play with this sort of attack, Josh Wright's WPE (Wireless Pwnage Edition) patch to FreeRadius is a nice shortcut to RADIUS server impersonation. Of course, back when I was first playing with this, that didn't exist, so I manually patched HostAPd to do the same thing. In more sophisticated attacks, I have “Frankensteined” a modified HostAPd to Xsupplicant, and successfully performed full man-in-the middle attacks against WPA2/AES with hardware one-time password tokens.

There is also a bunch of older attack code floating around that doesn't really work well anymore, as it was written before QoS was a common feature in access points. I have updated a lot of this code for my own use, plus I have written several of my own wireless attack tools. My favorite sort of wireless exploit to write would probably be best categorized as “blended surgical attacks” where the real action happens at layer 7, and forensics after the fact is next to impossible, as the attackers aren't connected to any WLAN, nor do they ever expose their own MAC address.

MB: I've noticed that a lot of guys on your team study ninjutsu, the martial art practiced by real-life (not metaphorical) ninjas. Is this a coincidence? What's the ninja-hacker connection?

NG: Oh, so you noticed that! Probably just a coincidence. To be fair, I would say that I study a system of Japanese martial arts that contains nine different budo taijutsu koryu (old-school combat arts), only three of which are classified as ninjutsu.

The connection? Well, I can't speak for any other martial artist, as everyone has his or her own reasons for pursuing such a hobby, but I do have a few theories on why people who are interested in computer hacking or penetration testing may be attracted to specific martial arts, such as ninjutsu.

First, both hackers and ninjas benefit from the mythology spun around their mystique. Often our labels for these individuals come along with a lot of misconceptions, which usually don't exactly hurt the people being labeled, as often they are attributed with near super-human abilities and skills. (Win! Unless these labels are used in a court of law, then: fail!)

Second, as my teacher often says, “We must learn the techniques and tactics of the bad guys, if we are to defend against them.” The same holds true for anyone who is skilled at breaking or securing systems. Knowledge of what the other guy is likely and capable of doing at any point can make a huge difference.

During martial training we learn to do some rather nasty things effectively, and then we explore the areas of space between attacker and defender, which, if occupied, could lead to either an optimal or sub-optimal outcome for the defender. We “play” in this way until the mind starts to form almost a three-dimensional model of what space leads to which outcome. My teacher often says, “Move through the 'safe shaped' space!”, which is bewildering to those unfamiliar with this concept. A lot of budo taijutsu training is counter-intuitive like that. Which brings me to my last point.

I once read that people who play mahjong, or other puzzle-like games can keep their mind sharp well into their autumn years—provided that the puzzles are hard. I'm pretty certain that those who exercise their minds by hacking are probably naturally drawn to other activities that are equally stimulating and challenging on a mental level.

Of course, hacking problems are way different from budo taijutsu problems, as they use completely different sections of the brain. When hacking, you have the luxury of being able to stop and think for a bit. Stopping occasionally to scratch your head, ponder and plot can go a long way to moving you forward to your end goal. Now imagine that your problem isn't breaking into a system, but rather surviving a knife attack. There simply is no time to stop to think; you just have to move and trust that your mental map of interactive space, balance taking, weapon usage, striking and so on is sound.

Training is often chaotic, with the teacher varying the conditions (such as weapon type or length, distance, number of attackers and so on) frequently. At higher levels of training, things start moving and changing so quickly that there really is never a time set aside to stop mentally to comprehend your experience. So instead of having a cognitive understanding of exactly what happened each time, you have only a general “feeling” of what is happening.

MB: I know that you work with Linux in at least two different contexts in your day job: as a platform from which to conduct attacks and also as a target, since so many of the things we ask you to test nowadays seem to run on Linux! What sorts of trends, good and bad, do you see in Linux security?

NG: On embedded Linux systems, I often find the sorts of problems that plagued larger UNIX systems decades ago, except now they can't be blamed on a clueless or lazy system administrator. Instead, it is often an embedded Linux developer who doesn't understand things like file ownership and permission bits, temporary file name prediction, proper usage of hard and soft links, command chaining, race conditions and exactly why it may be a good idea to update that prehistoric version of BusyBox.

One trend I see is that some embedded developers have gone to great lengths to attempt to enhance the security of the embedded Linux environment (to varying degrees of success). Their efforts range anywhere from a secondary password challenge-response scheme for root access, to neutering root completely, to only executing signed binaries, to removing all login services and shells entirely. A lot of these same things hold true for non-embedded Linux appliances, where users are not generally traipsing through the filesystem with shell access.

Of course, all newer Linux platforms have benefited greatly from some of the more recent developments in memory protection. There is still more work to be done, but I definitely would classify things like executable space protection as a good trend.

MB: By saying “neutering root completely” are you referring to SELinux and other role-based approaches to security? The “root is omnipotent” aspect of Linux's security model has always been its soft spot, hasn't it?

NG: SELinux is sometimes used (usually in full-size appliances); however, you probably would be surprised at some of the one-off, homegrown solutions that some vendors have implemented to attempt to rein in the almighty “root” in embedded Linux solutions.

Some vendors simply decide that root has no real purpose as a user account, so they cut it off at the knees (by giving it an invalid shell and a locked-out password). The system still will function perfectly, but whenever there is a need to upgrade system files, it usually involves completely reflashing the firmware for the device.

Other vendors choose to keep the root account, and then build their own mechanisms where upgrades are handled by root, but practically everything else runs as a nonprivileged user. In theory, this is a good approach, but again, even a simple misunderstanding of how hard and symbolic links work can spell disaster for upgrade routines that expect only vendor-provided (properly formed) upgrade files. If you can predict ahead of time where a temporary file used in the upgrade process is going to exist (say, /tmp), and you get there first with a link, many times you can end up clobbering the target of your link as root. If you get lucky, the script also may have a more-permissive umask setting, which would end up modifying and maybe lowering the file permissions to something less than before.

Dear readers, at this point in the interview, I've got good news and bad news. The bad news is, I'm out of space for this month's column and will have to wait for next time to share the rest of this fascinating and fun conversation with you.

The good news is, in Part II, you'll learn about Ninja G's opinions on the role of firewalls, the corrupting influence of hacking skills, the importance of responsible disclosure and his predictable yet surprising answer to the eternal question of who is more elite, pirates or ninjas. I hope you look forward to it!