Red Hat's ​Ceph and Inktank code repositories were cracked

Red Hat reports that the Ceph community project and Inktank download sites were hacked last week and it's possible that some code was corrupted.

Red Hat had a really unpleasant surprise last week. Its Ceph community site, which hosts development for the open-source Ceph distributed object store, and the Inktank download site, the commercial side of Ceph, had both been hacked.

ceph-logo.jpg
What happened? Was the code corrupted? We still don't know. Red Hat reports, "While the investigation into the intrusion is ongoing, our initial focus was on the integrity of the software and distribution channel for both sites."

The good news is "To date, our investigation has not discovered any compromised code available for download on these sites." The bad news is that Red Hat "can not fully rule out the possibility that some compromised code was available for download at some point in the past."

Putting salt into this storage software wound, this crack has opened the door not just to Red Hat's CentOS Ceph, but to Ubuntu Linux's Ceph as well. Both rely on the code from download.inktank.com. The CentOS and Ubuntu versions were signed with an Inktank signing key (id 5438C7019DCEEEAD). In addition, ceph.com provided the upstream packages for the Ceph community versions signed with a Ceph signing key (id 7EBFDD5D17ED316D).

Red Hat security states that they "no longer trust the integrity of the Inktank signing key, and therefore have re-signed these versions of the Red Hat Ceph Storage products with the standard Red Hat release key. Customers of Red Hat Ceph Storage products should only use versions signed by the Red Hat release key."

This intrusion did not affect other Ceph sites such as download.ceph.com or git.ceph.com. It's also not known to have affected any other Ceph community infrastructure. There is no evidence that build systems or the Ceph github source repository were compromised.

According to Ceph, "New hosts for ceph.com and download.ceph.com have been created and the sites have been rebuilt. All content available on download.ceph.com has been verified, and all ceph.com URLs for package locations now redirect there. There is still some content missing from download.ceph.com that will appear later today: source tarballs will be regenerated from git, and older release packages are being resigned with the new release key."

Red Hat Ceph Storage on Red Hat Enterprise Linux (RHEL) are not affected by this issue. Other Red Hat products were also unscathed.

Mark Shuttleworth, Canonical and Ubuntu founder, added, "For clarity, if you are using Ceph from the Ubuntu repositories then you are not affected at all."

Use the following instructions to download, verify, and install the known clean versions of Ceph.

Replace APT keys (Debian, Ubuntu)

sudo apt-key del 17ED316D

curl https://git.ceph.com/release.asc | sudo apt-key add -

sudo apt-get update

Replace RPM keys (Fedora, CentOS, SUSE, etc.)

sudo rpm -e --allmatches gpg-pubkey-17ed316d-4fb96ee8

sudo rpm --import 'https://git.ceph.com/release.asc'

Reinstalling packages (Fedora, CentOS, SUSE, etc.)

sudo yum clean metadata

sudo yum reinstall -y $(repoquery --disablerepo=* --enablerepo=ceph --queryformat='%{NAME}' list '*')

For a blessing, "Customer data was not stored on the compromised system. The system did have usernames and hashes of the fixed passwords we supplied to customers to authenticate downloads."

Red Hat isn't sure how this crack happened. On the other hand, the hacked sites were "hosted on a computer system outside of Red Hat infrastructure." The rebuilt sites are now safely within Red Hat's corporate confines.

Related Stories: