How secure is your shell? At many enterprises, not very

How secure is your shell? At many enterprises, not very

Summary: We talk to Tatu Ylonen, inventor of the SSH data-in-transit security protocol, about how (surprisingly!) sloppy large companies are in granting access to their most sensitive systems.


In 1995, young Finnish computer scientist Tatu Ylonen invented the SSH data in-transit security protocol, or "Secure Shell" for short. His goal was to make it more secure for authorized users to access the information stored on a server.

If you've been using a computer sometime over the last decade and a half, chances are high that you've stumbled across the protocol somehow. It's a favorite of large companies.

The enterprise (particularly the IT organization) is vastly different from what it was in 1995, yet Ylonen continues to work on the same key issue: secure access to sensitive information. (His company, SSH Communications Security, just released a new SSH key identification and removal service.) It turns out that it matters more than ever in an always-on world.

We spoke to him about what companies are doing to keep their data secure -- and how they're failing.

ZDNet: What are you working on? What problem are you trying to solve?

TY: The problem we are trying to solve is these keys that grant access to servers in the IT infrastructure that haven't really been managed properly. We've implemented services that mitigate these risks; we just last week closed a $2.5 million deal with a major bank.

What we're basically providing them is expertise on how to get SSH keys under control -- knowledge on back doors they have into the IT environment -- and then help them eliminate all those keys. There are thousands of such keys today in the environment. It's to make sure that traders or anybody else can't get unauthorized access to systems.

We want to start changing keys automatically so that if there are any copies or an employee leaves the organization, access is terminated.

We've been digging into the SSH key management issue for the last couple of years. We were getting more requests around it from customers. We have one of the top three retail chains [as a customer] and began digging through the problem with them. None of these major customers actually knew how many SSH keys they had in the environment. They didn't know who could access what. They had no control.

ZD: Which is a crazy thing to say, really.

TY: This problem has been hidden inside IT. These keys are used typically for automating administrative responsibilities. These companies have something like 20,000 servers in their environments. There's nothing wrong with [corporate] protocol, it's just that the management of these controls has been sloppy.

One bank estimates they have 400,000 access keys that they have been accumulating over the last 10 years. They don't know which ones to remove because they don't know which keys allow what. They don't want to close them down because they don't want to shut the door on access to a critical system.

At one global bank, we found over a million keys that grant access to the server. Ten percent gave access to the root. The big risk here is that...imagine that these keys connect different servers to each other in a kind of mesh. If you break into one server, there's a substantial risk that you'd be able to get into other servers -- you could continue to attack all of the other servers. You could potentially knock down a very substantial part of an organizations infrastructure in a matter of minutes. It seems silly, but an extremely similar attack happened in 1988.

ZDNet: How proactive, or reactive, are enterprises on this issue? Do they wait for a crisis before they're compelled to address these things?

TY: An IT administrator only sees a small corner of the infrastructure. They only see a part of the problem; no knowledge of the size of the problem or how many keys they have. And they're so extremely busy, especially with the downsizing going on over the last few years. They just don't have the bandwidth. Even auditors haven't woken up. Some have.

There are lots of administrators involved in creating these keys. They can read and copy private keys stored on that account. Some keys are short -- you could write them on the back of your hand.

There's a cost for creating keys. At one business, 200 administrators were spending 10 percent of their time doing this. Another company had a 15-person dedicated team set up for security administration. If you estimate what's the cost of doing that kind of work per year, it amounts to something near the $2 to $3 million range for these companies. That's easily paid back through software.

It helps people justify doing something.

The problem exists in every Fortune 500 company. Every one of them has Linux and Unix servers. It's also mainframes. There are routers that use SSH underneath for management. This applies to the largest cloud services providers. More than half of the world's websites run on Apache, which runs on Linux or BSD, typically, and those machines are typically managed using SSH.

ZDNet: Are they listening?

TY: If we talk to the right people, they almost always recognize that they probably have the problem. They almost always never realize the scope of the problem. There's a lot of educating that has to be done.

I believe that most public companies in the U.S. are out of compliance with Sarbanes-Oxley [The U.S. law that tightened regulation on the accuracy of corporate finances. --Ed.]  because they're not properly controlling who accesses what, and they're not properly terminating access when an employee leaves, and they're not properly auditing those keys.

We have customer cases starting with 20 servers. There's a substantial service component in the process. We need to engage other consulting companies and educate internal IT departments so that they would be able to do this.

The problem is so large that it goes into how auditing guidelines are implemented. Auditors don't know that SSH keys exist.

It's a challenging problem. For us, it helps that we have so many large customers using SSH commercially -- more than half of the 10 largest U.S. banks and seven of the global 10 companies use our commercial SSH clients and servers. We can talk to them about what the real issues are.

ZDNet: In my recent experience, the attitude toward cybersecurity has veered toward, "It's not a matter of if you're hacked, but when." Do you find that corporate executives are thinking differently about security?

TY: The boundaries of organizations have become so complex. Multiple sites. People working from home using mobile devices. Bringing their personal laptops to the office.

You won't be able to control everything. You must have multiple lines of defense. You can't allow another server to be compromised because one person broke into a server. IT infrastructure is becoming an amoeba tied into all sorts of partners. The protocols of that have become so complex. Javascript, Flash, even PDFs have become extremely complex documents. Even in-house, the's pretty complex semantics. The more complexity you have, the more potential exposure you have. The more code, the bigger the server, the more holes.

The approach we've developed with customers is discovering the keys in the environment -- itself pretty tricky to do properly -- and tracking the environment -- that is, which keys can be used -- so that we'll know which keys are never removed. When we want to remove the keys to the root -- ordinary users can't create keys for other users, which especially applies to automated processes -- we automate the management of keys, eliminating 15 or 20 man-years in the organization, giving them very substantial cost savings. Then we start rotating the keys so that every key gets changed every one or three months.

You get cost savings. You reduce the number of admins. You get operational efficiency savings. The goal is to minimize the risk.

Topics: Security, Data Management, IT Priorities

Andrew Nusca

About Andrew Nusca

Andrew Nusca is a former writer-editor for ZDNet and contributor to CNET. During his tenure, he was the editor of SmartPlanet, ZDNet's sister site about innovation.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • SSH Is Great

    And Microsoft has nothing like it.
  • Perhaps, Microsoft doesn't have anything like it

    But SSH Communications Security (linked in the article) does:

    The Tectia SSH Server is an enterprise product supported on the following platforms:
    o Unix
    o Linux
    o Windows
    o IBM z/OS
    o Linux on IBM system z

    P.S. Andrew, from the article, "we just last week closed a $2.5 billion deal with a major bank". Billion or million?
    Rabid Howler Monkey
    • Re: The Tectia SSH Server

      Can you SSH into a Windows box and execute the same PowerShell commands remotely that you can locally?

      No, you can't.
      • Powershell over SSH

        Actually you can do this (and I do). There might be more than one way but the software I am using is Bitvise SSH Server. The license is approx $100 USD/server.
        • Re: The license is approx $100 USD/server.

          My condolences.
          • Condolences?

            From the article:
            "we have so many large customers using SSH commercially -- more than half of the 10 largest U.S. banks and seven of the global 10 companies use our commercial SSH clients and servers.

            The Tectia SSH server license is $1050 U.S. per instance and the Tectia SSH client license is $183.60 per instance. Why are these enterprises using SSH Communications Security's commercially-licensed products when the SSH product that you are defending, presumably OpenSSH, is free?

            The fact is that many, if not most, enterprises have heterogeneous environments.
            Rabid Howler Monkey
      • Windows is miles ahead of SSH

        PowerShell's remote connections are *always* encrypted. Automatically.

        Your attempt at dissing Windows actually shines light on where Unix and Linux are sorely lagging behind Windows.

        SSH may be secure and simple to use for computer-to-computer connections that you manage yourself. But in an enterprise where security and regulations demands that keys are not shared, that they conform to certain standards (key length), that keys have a set expiry and are renewed regularly etc, key management quickly becomes a nightmare.

        So much of a nightmare that a bank is willing to pay $2.5 billion to help reign in the chaos.

        Enter Windows Active directory with certificate services. *All* network traffic are *automatically* encrypted, keys are *automatically* renewed. Windows comes with multiple certificate stores on each computer where keys can be stored securely (encrypted and isolated). Keys can be stored in a store available to all authenticated users/processes on a computer, in a store specific to each user account, and even in a store only available to each individual service - even if the service is executed under the same account as other services..

        Public Key Infrastructure (PKI) is an area where Windows absolutely blows Unix/Linux out of the water.

        Oh, and PowerShell can be accessed remotely through a web browser. Authenticated with your usual credentials, available commands/functions/connections limited by policy settings. And secured by SSL. Can you do that with *sh?
        • Windows Is Miles Behind SSH

          Windows has no separation between shell and terminal. On Linux, if I can run a local command "cmd", then I can run the same thing remotely on another machine "host" by doing little more than "ssh host cmd".

          On Windows, that doesn't work; remote commands have to be done in a completely different way, using the much more complicated WMI system.
        • @Honey, The bank did not take your solution

          “we just last week closed a $2.5 billion deal with a major bank.”
          That looks to be a typo, an article on ssh site listed it as a $2.5 million deal.

          Interesting reading.
          Case studies: Large US Federal Government agency selects SSH for IBM z/OS.

          The decision to deploy SSH was based on these factors:
          -Robust Enterprise Security
          -Centralized Management
          -Multi-platform Support: (UNIX, Linux, Windows and IBM mainframe)
          -Easy Integration and Implementation

          Send email to Larry to suggest an article on Powershell.
        • Uh yeah

          You can. But why would you access it through a webserver? SSH is *always* encrypted, automaticly. It does allow you to set up "policy" for restricting access. You are either a tool or a troll.
    • You don't need SSH on Windows. Connections are already encrypted

      When you set up a domain (or a forest) and enroll computers or even set up trust relationships, *all* connections are *inherently* secure and encrypted.

      The difference is that on Linux/Unix you need to manage that by yourself. On Windows it is already integrated with the network stack.

      The only time you need to set up an SSH server is when someone from a less sophisticated operating system need to connect securely.

      The only time you need to use an SSH client is when you need to connect to a less sophisticated operating system in a secure and encrypted fashion.
      • Re: You don't need SSH on Windows.

        Which, interestingly, contradicts the commenter above who said you CAN get it for Windows...
        • Not really a contradiction

          I get things I don't need all the time.
          • Why?

            Why do you allow.
          • Given that this article is concerned with enterprises

            A better response would have been something like: "I bet that your company's CFO is delighted that you purchase software that the business does not need."
            Rabid Howler Monkey