Microsoft takes wraps off new Windows file system

Microsoft has given its first detailed description of ReFS, the new file system that will gradually succeed the now-venerable NTFS in Windows systems.

The Resilient File System or ReFS, the successor to NTFS, will make its first appearance in Windows Server 8.

The Resilient File System (ReFS) will make its first appearance as a storage system in Windows Server 8, after which it will evolve into a system for client storage, then ultimately for boot volumes. On Monday, Microsoft used a post on its Building Windows 8 blog to lay out details on ReFS.

"Along with Storage Spaces, ReFS forms the foundation of storage on Windows for the next decade or more," Microsoft development manager Surendra Verma wrote. "We believe this significantly advances our state of the art for storage... we expect that we will see ReFS as the next massively deployed file system."

The "staged evolution" of ReFS, beginning with its use as a file server, is the same approach Microsoft has taken with previous file systems.

According to Verma, ReFS has many of the same features and semantics as NTFS, which was introduced in 1993. These include "BitLocker encryption, access-control lists for security, USN journal, change notifications, symbolic links, junction points, mount points, reparse points, volume snapshots, file IDs and OpLocks".

If a file-access API can access an NTFS volume, it will be able to access data stored on ReFS.

"Rewriting the code that implements file system semantics would not lead to the right level of compatibility, and the issues introduced would be highly dependent on application code, call timing and hardware. Therefore in building ReFS, we reused the code responsible for implementing the Windows file system semantics," Verma said.

On-disk storage engine

What has changed is the on-disk storage engine underneath the reused code. The ReFS engine exclusively uses the so-called B+ tree structure to represent stored information. Verma indicated this will mean a simpler system, with the choice of structure designed to be as scalable as possible.

Along with Storage Spaces, ReFS forms the foundation of storage on Windows for the next decade or more.

– Surendra Verma, Microsoft

On the subject of detecting and fixing corruption, Verma explained that all ReFS data is check-summed at the level of the B+ tree page, with the checksum — an algorithm for checking data integrity — being stored away from that page.

"This allows us to detect all forms of disk corruption, including lost and misdirected writes and bit rot (degradation of data on the media). In addition, we have added an option where the contents of a file are check-summed as well," Verma wrote. 

The Storage Spaces feature will be available for NTFS and, unlike ReFS, for client PCs from the start. It is supposed to improve performance and protect data by maintaining copies on multiple disks.

When ReFS is used alongside mirrored Storage Spaces, corruptions will be "automatically and transparently fixed", Verma added, noting that ReFS also had a 'salvage' feature for making sure that non-repairable corruption does not affect the availability of "good data".

Scrubbing

ReFS offers a new way of dealing with 'bit rot', or data decay, in that it uses a system task to "periodically [scrub] all metadata and Integrity Stream data on a ReFS volume residing on a mirrored Storage Space", Verma said.

"Scrubbing involves reading all the redundant copies and validating their correctness using the ReFS checksums. If checksums mismatch, bad copies are fixed using good ones," he said, adding that "the file attribute FILE_ATTRIBUTE_NO_SCRUB_DATA indicates that the scrubber should skip the file".

Responding to readers' questions about the blogpost, Verma also stressed that Windows 7 users would be able to read ReFS-formatted partitions from a Windows 8 Server, either by using a new file system driver or by sharing a folder out from the server.

"Note that support for NTFS is going to be present in Windows for the foreseeable future, so you should always be able to access all your NTFS data across versions without any problems," he added.