Network security blunders - how to recover

Commentary - We’ve all made one in our career - I’m talking about that blunder you thought would cost you your job. My first major blunder was rebooting all the campus router pairs at one time, not one by one, all at once. I had written a script to install a security update on all the routers and reboot them all one by one…. or so I thought. Turns out my script had an error and didn’t wait between routers.

I thought for sure I was fired, but thankfully, I wasn’t. What could have been a major disaster turned out it was a great learning experience for everyone involved. We all learned a little about crisis management, and as everything came back up online, my boss took a few hours to teach me how to verify the network was working properly.

The good news is that most of the time, our blunders are not so severe. The bad news is that oftentimes they are not instantly noticeable, which means they can remain undetected for weeks, months, or even years until one day they either cause an outage or an auditor calls us on them. On the network security front, firewall management is one area where a simple mistake regarding a rule or configuration change can come back to haunt you – here are some of the most common errors:

Creating firewall groups with no meaning
One firewall administrator had a certain network object in over half the rules. It was named after a famous football star; let’s call it “Joe_Montana.” Whenever they needed to allow access, they added the object IP address, which was used in many of their permissive rules, to this group. The result is that the rule base looks OK to an auditor (because there are no rules that contain ANY) but it's actually a huge security hole. The rules become meaningless and if it was flagged in an audit, the cleanup of the rule base would be a thankless task, requiring many months of sorting out a rule base that securely and appropriately maps to needs of the business

Failing to upgrade your firewall software
A surprising amount of organizations run outdated firewall software. When asked why, it’s the same story about keeping the version fixed for stability; the firewalls cannot be taken down for upgrades, etc… The fact is, firewall vendors upgrade their software for a reason. You don’t need to be on the latest greatest release, but if you are running a version that is 15 or 20 releases ago and 7 years old, stop complaining and start upgrading!

Using the wrong technology
We’ve all heard of ways in which the square peg is pounded into the round hole, but here’s one of my favorites. One network security administrator was arguing with their auditors that because they have a firewall in front of their secure webservers, it formed a second layer of authentication which meant they were using two-factor authentication: a password and a firewall. This guy deserves an A for creativity,, but a firewall (by itself) is not a two-factor authentication solution. Two-factor authentication requires your users have something; it’s something they Know and something they Have, a token and password for example.

The accidental outage
I heard a story of being about one such outage where the firewall administrator was working on the production firewall server gathering some data for a support case. The admin reached across the table and accidentally leaned on the mouse, which was over the Start Menu. As fate had it, the mouse activated the Start Menu and was unbelievably over the shutdown menu item when it popped up. Yep, right there in the middle of production this financial corporation watched their production firewall shutdown.

Poor documentation
All too often you hear about firewall administrators trying to understand what the heck all their rules do. By allowing ourselves to become busy or stressed to take the time to create the proper documentation, we create a time bomb waiting to blow up. Ask anyone that has been involved in managing firewalls how often they have heard “I’m afraid to make changes on my firewall now, all the senior guys have left and we don’t know what most of these names mean or what these rules do.”

Using excessive Drop rules
Often when we are in a hurry we create rules with very excessive accesses, and place a rule just above it dropping the access we don’t want to allow. We do this because we don’t want to figure out how to write the correct rule. For example: “allow All DMZ devices to All Internal devices ACCEPT,” with a rule right above it that says “All DMZ devices to Secure Network device DROP.” The two rules look OK at first, but it is really an ugly hack because we didn’t write out the business need in the first rule. When we operate like this over time we have a rule base with lots of the rule ‘pairs’ and reordering the rule base or editing rule bases is likely to expose more risk or block necessary traffic. Either way, we have a mess we must rewrite at some point.

Using routing as your security policy
I’ve seen many a firewall where rule base changes need routing changes to accompany them. It’s understandable when the change involves a new network on the firewall. There are two very common versions of this blunder that happen all the time. The first is the firewall with no default route. Every route is added to the firewall by hand and the smallest netmask possible is used so traffic will not reach unintended devices if the firewall has no policy. Wow, that sounds great, but it’s totally unnecessary - if you remove the policy on modern firewalls they revert to DENY ANY. This design becomes so hard to manage that the entire team begins to dread making changes. Soon every change needs an engineer to examine the routing, thus every change takes too long and impacts servicing the business in a timely fashion so there is no real value in added security.

The second version of this blunder is most often seen on Cisco devices where admins have ACLs between two interfaces that include the source or destination ANY. They don’t actually mean ANY, they mean all the addresses behind this interface, but I’m too lazy to type the addresses in. This leads to a rule base that can be understood only by knowing the routing table along with the firewall and trying to do the match in your head – way too complicated for a junior firewall admin to take over this firewall.

Using DNS objects in a rule base
One of the options many firewalls provide is inserting a source or destination as a DNS object like www.google.com. It sounds great because google.com can use so many IP addresses and this allows my firewall to always pass the traffic as google.com changes IP addresses. This blunder leads to many risks most organizations should consider unacceptable. First your firewall is now more susceptible to Denial of Service attacks. What happens when it can’t resolve google.com? Second your firewall is going to waste CPU, memory, and network IO on doing DNS lookups for every packet trying to decide if it might belong to google.com. Third, what happens if your DNS is poisoned and malicious addresses for the command and control of the attest botnet are returned with the google.com addresses? You now just allowed all the botnet command and control traffic though your firewall, and logged it as google.com.

Making changes in panic mode
Imagine that something goes wrong and you lose one of the RAID disks. You replace it and while the RAID is rebuilding, the service is slowed down – but you don't realize that it's because of the RAID. At this point, your customers have been denied service for 40 hours and you're losing a lot of money every minute, not to mention customers who are leaving you for alterative services from your competitors.

You go into panic mode and start changing configurations: switches, routers, load balancers and firewalls – anything that you suspect may be causing the problem. After another 24 hours, another sleepless night and many hours of expensive consultants, someone figures out the real cause of the problem. Now you want to revert all the changes you made on the switches, routers, load balancers and firewalls but no one knows what they were because they were made in a rush without any documentation. So you spend another 3 days figuring out how to revert the system back until it's fully operational.

I hope you don’t see any of these in your organization. But if you do, rest assured you are not alone. The best run organizations can find these blunders or others lurking in their firewall rules. The good news is that what experience can’t rectify automation can. The secret to knowing whether your company can benefit from automating elements of your network security - or any other aspect of security, for that matter - is if the ROI is clear and easily quantified. After all, if you can’t quantify how any security investment will impact the bottom line, how can you expect anyone else to?

biography
Michael Hamlin is chief security architect for Tufin Technologies