1. Using Memory as a Storage Medium

The image above is the storage pyramid, showing the relationship between the price of a storage medium and its speed.

Today, enterprises widely adopt tape and disks as storage media mainly because of their price advantage. Market prices are driven by supply and demand: even for an expensive storage medium, if the production cost is low, the market is large enough, and the yield rate in the production process is high, its market price will come down.

The equilibrium point of the market is where the storage medium can meet consumers’ needs while the price consumers are willing to pay covers the production cost. This equilibrium, however, is not reached overnight — bringing capacity online takes a long cycle, while price competition shifts from moment to moment, and this mismatch between long and short cycles causes market prices to fluctuate.

Our demand for the speed and capacity of storage media keeps growing; we always pursue faster, cheaper, and larger-capacity storage solutions. This growing demand will push the market equilibrium point to move, give rise to new demand, and at the same time drive innovation and development in storage technology.

Using memory as a storage medium faces two major challenges: high cost and data loss after power failure.

The price problem is a question for demand — if the demand side explodes, the price of memory as a storage medium will come down. Volatility on power loss can be addressed at both the software and hardware levels. At the software level, data reliability can be improved through redundancy, erasure-code reconstruction, and similar techniques; at the hardware level, Intel’s Optane stands as a martyr we can learn from. Of course, we could also leave the volatility problem unsolved — losing data on power loss is only a drawback in traditional scenarios. What about a scenario that requires secrecy? Volatility on power loss might become a very good solution. Looking for new scenarios and high-value-added demand is also a good line of thinking.

Based on the above considerations, I believe that using memory directly as a storage medium is an optional approach for the future. This does not mean it will become the mainstream approach — only that in certain scenarios it will bring considerable convenience or benefit.

2. Distributed In-Memory Storage

If we were only creating a storage area on a single device, tmpfs would do the job.

Therefore, being distributed and supporting horizontal scaling is a necessary characteristic of MemoryFS.

The image above shows MemoryFS’s storage architecture with metadata separated from data.

Using Redis directly to store metadata is actually a very good choice. When using the JuiceFS community edition, my first choice for metadata storage was also Redis; Redis needs only 1GB of memory to support roughly 2TB of data storage.

Another approach is to implement a distributed metadata store using RocksDB and Raft, which would bring more convenient deployment and a more customizable storage solution.

3. Supporting the POSIX Protocol

On Linux systems, storage that supports the POSIX protocol can be mounted into the file system, providing remote access capability and expanding the application scenarios.

The common way to support the POSIX protocol is to use Fuse; JuiceFS and SeaweedFS also provide the POSIX protocol based on Fuse.

In https://github.com/torvalds/linux/tree/master/fs you can see that general file systems such as ext4 and btrfs are all implemented in the kernel. Fuse provides a way to implement a custom file system without modifying the kernel.

As shown above, using Fuse mainly involves the following steps:

• Mount into the file system. Run ./hello /tmp/fuse to mount the file system at /tmp/fuse. The figure uses the libfuse library, but there are libraries for other languages too, such as the Go library https://github.com/hanwen/go-fuse. In the ./hello program you need to implement the specified interfaces and connect them to the external storage medium.
• Use the file system. Run ls /tmp/fuse to see the list of files in the file system. When the command runs, a system call is made through libc, forwarded through the Fuse module in the kernel to the hello program, and the hello program then responds to the file system API corresponding to the ls command.

As shown above, you need to create a cluster composed of MemoryFS Workers to provide storage services; to use it, you simply mount it into the current directory through a local Fuse program.

4. Summary

This post mainly records some ideas about providing external storage capability with memory as the storage medium. The main points are as follows:

• Memory as a storage medium is an optional storage solution for the future
• Being distributed and supporting horizontal scaling is a necessary characteristic of MemoryFS
• MemoryFS needs to support the POSIX protocol and provide remote access capability