AWS introduces Amazon MemoryDB for Redis
Amazon Web Services, which already offers a portfolio of 15 databases, is now releasing one more. Amazon MemoryDB for Redis is a real-time database implementation that is based in part on the open source Redis platform, but adds durability and persistence. AWS positions this as targeted for customers who require a full-blown database service rather than a cache for Redis based on the assumption that the use cases will be quite different.
We view it as a continuum, with Amazon MemoryDB being a service that picks up where Amazon ElastiCache leaves off, and as direct competition with Redis Enterprise Cloud offered by Redis, the company. The difference between Amazon ElastiCache and MemoryDB is that the former is intended as an in-memory cache that works alongside a primary database, whereas MemoryDB is a full database service in itself that is designed to operate on its own.
Until now, AWS's ElastiCache service offered a choice of Redis caching, or Memcached. The new service is AWS's answer to Redis Enterprise. Consider Amazon MemoryDB is, in essence, a premium tier of ElastiCache for Redis; functionally, it is a database rather than cache, and technically, it is a modified implementation of the Redis engine that is compatible with it.
The backdrop to all this is that open source Redis is an incredibly popular platform. Ranked among the top 10 databases by db-Engines, according to Sumologic, it is also the most popular database deployed on AWS. And pouring it on, Stack Overflow surveys of its community listed Redis as the "most loved" database by developers for the last five years running. Each of the major cloud providers offer their own Redis-based caching services based on the open source version.
Developers love Redis because it meets developers where they live. It supports the data types that developers frequently work with, such as hashes, lists, sets, strings, and sorted sets. And, secondly, Redis supports the languages that are popular with developers, such as C, C++, C#, JavaScript, Java, Go, Objective-C, Python, and PHP.
Versatility has long been key to Redis' success. It's most commonly used as an in-memory cache that sits alongside a transactional database, such as SQL Server, MySQL, or Oracle. It offloads the real-time processing from back end databases. That's a task that's as old as the transaction monitors of the 1990s and the in-memory object stores (which were primitive key/value data stores) of the early 2000s.
The challenge, however, is when developers use Redis cache for purposes that it was not intended, with time series databases for IoT data being a prime example – and popular use case. Redis cache ingests the data rapidly, but the problem is that it lacks durability and persistence; when the node goes down, the data is lost. Open source Redis can write data to an append-only file on disk, but that measure is similarly vulnerable if the node goes down.
Until now, when it came to Redis as a full-fledged in-memory NoSQL or multimodel database, Redis Enterprise was the only game in town. Redis Labs has staked its business on offering answers for developers requiring more than cache. Compared to open source Redis, Redis Enterprise supports more granular high availability approaches; tiering of data to SSD Flash storage; fully geodistributed (multimaster) database operation through active-active replication; multi-cloud support; and support of Redis Modules, which add extensibility to the database (e.g., support of search, graph data, embedded SQLite, JSON and other capabilities). Redis Enterprise Cloud runs on all three major public clouds – through the marketplaces on AWS and Azure; and as a jointly supported offering on Google Cloud.
And the need for a Redis database is what prompted AWS to introduce MemoryDB – although at this point, the capabilities of MemoryDB and Redis Enterprise markedly differ. For instance, Redis Enterprise offers tiered storage to Flash, a capability that is not on the initial release. A major differentiator with MemoryDB, however, is its approach to data persistence. It employs a multi-AZ transaction log to keep data durable and enable replacement and recovery with no data loss. Importantly, the log does not run on MemoryDB's database nodes. Once MemoryDB commits a write to the underlying transaction log, it is propagated to replica nodes with guaranteed eventual consistency.
Customers can launch MemoryDB clusters using AWS Graviton2 instances. It will perform microsecond reads and single-digit millisecond writes thanks, not only to relying on memory, but with the lightweight transaction log approach. As noted, while tiering cooler data to Flash is not currently supported, we wouldn't be surprised if it gets supported in a future release.
Coming out of the gate, Amazon MemoryDB will support all of the core data APIs of open source Redis, including strings, lists, sets, sorted sets, hashes, streams, geospatial, bitmaps, and hyperloglogs. It will support up to 128 TBytes of memory per cluster, with one replica for each shard. Besides tiered flash storage, other Redis Enterprise features such as support for Redis modules, some of the extended data types, and active-active geo-replication are not currently supported by MemoryDB; many of these features are available only through the Redis Source Available License, which prohibits third parties from selling them if they are offered as a database or other verboten uses. On the other hand, Redis Enterprise, which is currently working on Redis Raft for durability, has not brought that yet to general availability (we expect that to change soon).
By offering a service that AWS terms Redis-compatible supporting the open source Redis APIs, there are parallels with AWS's approaches to Amazon Aurora (for MySQL and PostgreSQL) and Amazon KeySpaces (for Apache Cassandra): they support the same APIs as the popular open source data store with under-the-hood improvements to support a fully managed experience. For MemoryDB, users will be able to build applications using the same Redis data structures and APIs that they already use, but with the added benefits of a fully managed database.
Amazon MemoryDB for Redis begins to fill the gap for customers seeking a durable counterpart to ElastiCache. At this point, it does not have the same breadth of functionality as Redis Enterprise or Redis Enterprise Cloud, at least where it to comes to support of Redis modules or geodistributed read and writes. And while this is not the first foray for AWS with adding in-memory capability to its databases, given the popularity of Redis, it fills a significant gap in its lineup. Unlike most AWS database introductions that typically begin with a preview, coming out of the gate, MemoryDB is now generally available.
Disclosure: AWS is a dbInsight client.