arXiv:2408.01805cs.PF
Billion-files File Systems (BfFS): A Comparison
Early discovery. Linked by one identifiable commenter. This paper is outside the main ranking; a mention is not an endorsement.
Abstract
As the volume of data being produced is increasing at an exponential rate that needs to be processed quickly, it is reasonable that the data needs to be available very close to the compute devices to reduce transfer latency. Due to this need, local filesystems are getting close attention to understand their inner workings, performance, and more importantly their limitations. This study analyzes few popular Linux filesystems: EXT4, XFS, BtrFS, ZFS, and F2FS by creating, storing, and then reading back one billion files from the local filesystem. The study also captured and analyzed read/write throughput, storage blocks usage, disk space utilization and overheads, and other metrics useful for system designers and integrators. Furthermore, the study explored other side effects such as filesystem performance degradation during and after these large numbers of files and folders are created.
XFS is well regarded. It is kind of a second-generation of UINX filesystem that was designed from ground up with journaling and extents and btrees and xattrs. EXT4 is well regarded too, it has similar goodies now but it evolved from a first-generation stile without journaling…
starspangledView on HNParent context
From this paper https://arxiv.org/html/2408.01805v1 (2024) > EXT4 is the general-purpose filesystem for a majority of Linux distributions and gets installed as the default. However, it is unable to accommodate one billion files by default due to a small number of inodes…
starspangledView on HNParent context