Patil et al., 2011 - Google Patents

Scale and concurrency of {GIGA+}: File system directories with millions of files

Patil et al., 2011

View PDF
Document ID
51049064502085198
Author
Patil S
Gibson G
Publication year
Publication venue
9th USENIX Conference on File and Storage Technologies (FAST 11)

External Links

Snippet

We examine the problem of scalable file system directories, motivated by data-intensive applications requiring millions to billions of small files to be ingested in a single directory at rates of hundreds of thousands of file creates every second. We introduce a POSIX …
Continue reading at www.usenix.org (PDF) (other versions)

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/30—Information retrieval; Database structures therefor; File system structures therefor
    • G06F17/30286—Information retrieval; Database structures therefor; File system structures therefor in structured data stores
    • G06F17/30312—Storage and indexing structures; Management thereof
    • G06F17/30321—Indexing structures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/30—Information retrieval; Database structures therefor; File system structures therefor
    • G06F17/30067—File systems; File servers
    • G06F17/30182—File system types
    • G06F17/30194—Distributed file systems
    • G06F17/30212—Distributed file systems implemented as replicated file system
    • G06F17/30215—Details of management specifically adapted to replicated file systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/30—Information retrieval; Database structures therefor; File system structures therefor
    • G06F17/30286—Information retrieval; Database structures therefor; File system structures therefor in structured data stores
    • G06F17/30575—Replication, distribution or synchronisation of data between databases or within a distributed database; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/30—Information retrieval; Database structures therefor; File system structures therefor
    • G06F17/30067—File systems; File servers
    • G06F17/30129—Details of further file system functionalities
    • G06F17/3015—Redundancy elimination performed by the file system
    • G06F17/30156—De-duplication implemented within the file system, e.g. based on file segments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Error detection; Error correction; Monitoring responding to the occurence of a fault, e.g. fault tolerance
    • G06F11/16—Error detection or correction of the data by redundancy in hardware
    • G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
    • G06F11/2053—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant
    • G06F11/2056—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant by mirroring
    • G06F11/2071—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant by mirroring using a plurality of controllers
    • G06F11/2076—Details of synchronous techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/30—Information retrieval; Database structures therefor; File system structures therefor
    • G06F17/30067—File systems; File servers
    • G06F17/301—Details of searching files based on file metadata
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/30—Information retrieval; Database structures therefor; File system structures therefor
    • G06F17/30943—Information retrieval; Database structures therefor; File system structures therefor details of database functions independent of the retrieved data type
    • G06F17/30946—Information retrieval; Database structures therefor; File system structures therefor details of database functions independent of the retrieved data type indexing structures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for programme control, e.g. control unit
    • G06F9/06—Arrangements for programme control, e.g. control unit using stored programme, i.e. using internal store of processing equipment to receive and retain programme
    • G06F9/46—Multiprogramming arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from or digital output to record carriers, e.g. RAID, emulated record carriers, networked record carriers
    • G06F3/0601—Dedicated interfaces to storage systems
    • G06F3/0628—Dedicated interfaces to storage systems making use of a particular technique
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00—Data processing: database and file management or data structures
    • Y10S707/99951—File or database maintenance
    • Y10S707/99952—Coherency, e.g. same view to multiple users

Similar Documents

Publication Publication Date Title
Patil et al. Scale and concurrency of {GIGA+}: File system directories with millions of files
US11995043B2 (en) Map-reduce ready distributed file system
Ren et al. IndexFS: Scaling file system metadata performance with stateless caching and bulk insertion
Thomson et al. {CalvinFS}: Consistent {WAN} replication and scalable metadata management for distributed file systems
Escriva et al. HyperDex: A distributed, searchable key-value store
Welch et al. Scalable Performance of the Panasas Parallel File System.
Tantisiriroj et al. On the duality of data-intensive file system design: reconciling HDFS and PVFS
Thanh et al. A taxonomy and survey on distributed file systems
Wang et al. Cfs: Scaling metadata service for distributed file system via pruned scope of critical sections
US10387384B1 (en) Method and system for semantic metadata compression in a two-tier storage system using copy-on-write
Zheng et al. DeltaFS: a scalable no-ground-truth filesystem for massively-parallel computing
Pacheco et al. GlobalFS: A strongly consistent multi-site file system
Patel et al. Improving data transfer rate and throughput of hdfs using efficient replica placement
Avilés-González et al. Scalable metadata management through OSD+ devices
Fan et al. Tengine: A native distributed table storage engine
Vaidya et al. Critical study of performance parameters on distributed file systems using MapReduce
Nakamura et al. MyCassandra: A cloud storage supporting both read heavy and write heavy workloads
Shu Distributed storage systems
Chen et al. PeakFS: an ultra-high performance parallel file system via computing-network-storage co-optimization for HPC applications
Ali et al. Revisiting the metadata architecture of parallel file systems
Luo et al. Comparing hadoop and fat-btree based access method for small file i/o applications
Huang DNN: a distributed namenode filesystem for Hadoop
Patgiri et al. A Survey on Large Scale Metadata Server for Big Data Storage
Arteaga et al. Towards scalable application checkpointing with parallel file system delegation
Weil et al. Ceph: A scalable object-based storage system