DEP - A missed opportunity to protect millions of Windows users

Imagine coming across someone who had both an antivirus package and firewall software installed on their PC and yet both were switched off.  You'd think that they were pretty dumb, way too brave or a little bit crazy (or they are antivirus researchers!).  But the fact is that there are literally millions of Windows XP SP2 users who have a defense mechanism in place that would protect them against many of the vulnerabilities that threaten them, but that protection is, by default, partly disabled.

I'm talking about a technology called Data Execution Prevention.  DEP is a feature that is built into Windows (you must have SP2 installed to take advantage of this) that prevents an application or service from executing any code that resides in a non-executable memory region.  The idea is that this technology will halt buffer overflows in their tracks.

There are two kinds of DEP:

• Hardware-enforced
• Software-enforced

By far the most effective form of DEP is hardware-enforced DEP.  This relies on having a CPU that supports the NX or XD bit.  Modern AMD processors support NX (which stands for No eXecute) while modern Intel CPUs support XD (which stands for eXecute Disable).  Both features carry out the same function and differ only in name.  If you don't have a CPU that understands NX/XD then you are limited to the inferior software-enforced DEP and you'd have to upgrade the CPU or buy a new PC if you wanted to use hardware-enforced DEP. 

It's important that I point out that DEP will not secure you from any malicious applications that you yourself choose to run, it only offers a defense against buffer overflows found used by hackers to run malicious code.

OK, so what's the problem with DEP?  Why is it a missed opportunity? Well, by default it isn't set up to offer you the best protection.  Instead it only monitors and protects you from malicious applications trying to leverage essential Windows programs and services.  In the default DEP configuration doesn't offer a great deal of protection.  Will things get better when Windows Vista is out?  Nope.  The defaults are set up in exactly the same way.

Why is DEP set up this way by default?  Because there are a lot of badly written applications out there that routinely execute data as code.  This triggers false alarms, which can be annoying.  Fortunately you can add exceptions for any misbehaving applications you come across.

So how do you fully enable DEP? You can do it with just a few clicks on Windows XP:

• Click Start > Control Panel
• Click on Performance and Maintenance (if you are in Classic View, skip this step)
• Click on System
• Click on the Advanced tab
• In the Performance group, click on Settings
• On the Performance Options dialog, click on the Data Execution Prevention tab
• Click on Turn on DEP for all programs and services except those I select
  

• Click OK
• Click OK to confirm that the system will need to be restarted
• Finally, reboot the system

Here's what you'll see if your CPU doesn't support hardware-enforced DEP:

The process for fully activating DEP on Windows Vista is a little different:

• Click Start > Control Panel
• Click System and Maintenance
• Click on System
• Click on Advanced system settings
• Click Continue on the User Account Control dialog that will be generated
• In the Performance group, click on Settings
• On the Performance Options dialog, click on the Data Execution Prevention tab
• Click on Turn on DEP for all programs and services except those I select
• Click OK to confirm that the system will need to be restarted
• Click OK
• Finally, reboot the system

Once you’ve rebooted, you can test that DEP is working by downloading and running a small utility called NXTEST by Robert Schlabbach.

So, what can DEP protect you against?  Well, there have been three big security scares this year that have been stopped in their tracks by hardware-enforced DEP.  These include the WMF vulnerability from the beginning of the year and the latest VML vulnerability affecting Internet Explorer.  You should never rely solely on hardware-enforced DEP to protect you against malicious code, but given that the detect rate for the VML vulnerability is still pretty awful, it's a handy safety net to be running.