Linux Mint: The right way to react to a security breach

Just over a week ago, the Linux Mint servers were penetrated, and a hacked ISO image was offered for download instead of one of the genuine Mint images.

For almost the entire week after the incident, there was a lot of discussion about how bad the Mint developers were, how stupid and lazy and careless they were, how no one in their right mind should ever use Mint under any circumstances anyway, and on and on and on.

MAJOR BREACH

Hacker explains how he put "backdoor" in hundreds of Linux Mint downloads

The hacker said their prime motivation for the backdoor was to build a botnet.

Read now

In my opinion it was a classic case of a what my father used to describe as, "Some people can only feel big if they are standing on someone else".

Clem posted the monthly update to the Mint Blog yesterday, and devotes almost all of it to discussing the breach and their actions afterward. It is a very interesting, enlightening, and encouraging description, and I strongly encourage anyone with an interest in this incident, or in security incidents in general, to read it.

What I think is relevant about the incident, and the Mint developers handling of it:

• Stop the bleeding, as quickly as possible (take down any and all affected or even potentially affected servers)
• Be open about what happened, as quickly as possible, as publicly as possible, and as thoroughly as possible.
• Don't be afraid or embarrassed to admit things you don't know, and don't hesitate to ask for help or to accept offers of help.
• Don't try to just fix what got hacked. If you got hacked, you have a problem - probably a very big problem. Your objective is to end up at a situation that is much better than the one you were in before the incident.
• Don't stop communicating while you are fixing the problem. This is a big one, and it is an easy trap to fall into. Clem talks about how hard and how long they all worked. It is very easy to just say "we have to get this fixed, then we will explain", but the resulting lack of communication at a critical time is invariably misunderstood by the outside world.
• While you are cursing the evil-doers, don't forget to be thankful for your friends. Especially in this community (Linux and FOSS), there are a lot of people who are willing to help, to be patient, and to do anything else they can to support someone else.
• Finally, a corollary to the "don't just try to get back to where you were" point above, once you are immersed in this kind of situation, don't limit your work and your attention to only the area that was compromised. Widen your scope, look at other activities and other areas, and try to use the knowledge you are gaining to improve them too - as the Mint developers are doing by not only fixing their own servers and distribution system, but also by examining and considering improvements for the distribution itself.

Those are all good things, and I admire the efforts that Clem and the developers have put into it. But I think we all need to look at the other side of the coin as well - the user's side. If you download and install Mint (or any other Linux distribution):

• Always verify the checksum on the downloaded file. Duh, I know, but if I were to be completely honest, I would have to say that I am also guilty of just downloading and installing a few times when I was in a hurry. This is so far beyond a bad idea that I can't even think of a way to describe it. I am ashamed of myself - and if you are guilty of doing it, you should be too.
• Don't blindly use the simplest verification possible. A lot more people (me included) are guilty of this. The MD5sum is the most obvious, so we just run that and assume we are good to go. If there is an sha256sum available, use it.
• Don't assume that a system which installed properly doesn't have a problem. Keep your eyes open for things that are suspicious. One thing I have not seen discussed is how the users who first reported this compromise discovered it - what did they see which made them check it? I work in network security, and I can tell you from first-hand experience that a LOT of problems are actually seen first by ordinary users who simply notice that something "doesn't seem right" or is somehow different. A lot of problems could be found and fixed a lot sooner if people would speak up when this happens.

I hope that the Mint developers get the credit that they deserve for their response to this problem. They certainly got more criticism than they deserved for it.

Read more about Linux and open source:

• Linux and breakfast cereal: Why are there so many flavours?
• Hands-on with Kali Linux Rolling
• How to customise your Linux desktop: MATE
• How to customise your Linux desktop: Cinnamon
• How to customise your Linux desktop: Xfce
• Hands-On with openSuSE Leap RC1: A walk through of the installer
• Hands-On: KaOS Linux 2015.10
• Thus versus Calamares: Comparing Manjaro 15.09 installers
• Upgrading my Linux-Windows multi-boot system to Windows 10
• Hands-On: Linux UEFI multi-boot, my way
• Hands-On: Linux UEFI multi-boot, part two