Presentations

Hdfs Design

Description
Hdfs Design
Categories
Published
of 12
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Related Documents
Share
Transcript
  The Hadoop Distributed File System:Architecture and Design by Dhruba Borthakur Table of contents 1  Introduction .......................................................................................................................3 2  Assumptions and Goals .....................................................................................................3 2.1  Hardware Failure...........................................................................................................3 2.2  Streaming Data Access .................................................................................................3 2.3  Large Data Sets .............................................................................................................3 2.4  Simple Coherency Model .............................................................................................3 2.5  Moving computation is cheaper than moving data .......................................................4 2.6  Portability across Heterogeneous Hardware and Software Platforms ..........................4 3  Namenode and Datanode ..................................................................................................4 4  The File System Namespace .............................................................................................5 5  Data Replication ................................................................................................................5 5.1  Replica Placement . The First Baby Steps ....................................................................5 5.2  Replica Selection ..........................................................................................................6 5.3  SafeMode ......................................................................................................................6 6  The Persistence of File System Metadata .........................................................................7 7  The Communication Protocol ...........................................................................................8 8  Robustness ........................................................................................................................ 8 8.1  Data Disk Failure, Heartbeats and Re-Replication .......................................................8 8.2  Cluster Rebalancing ......................................................................................................8 8.3  Data Correctness ...........................................................................................................8 Copyright © 2005 The Apache Software Foundation. All rights reserved.  8.4  Metadata Disk Failure .................................................................................................. 9 8.5  Snapshots ......................................................................................................................9 9  Data Organization .............................................................................................................9 9.1  Data Blocks .................................................................................................................. 9 9.2  Staging ........................................................................................................................10 9.3  Pipelining ....................................................................................................................10 10  Accessibility ..................................................................................................................10 10.1  DFSShell ...................................................................................................................11 10.2  DFSAdmin ................................................................................................................11 10.3  Browser Interface ......................................................................................................11 11  Space Reclamation ........................................................................................................11 11.1  File Deletes and Undelete .........................................................................................11 11.2  Decrease Replication Factor .....................................................................................12 12  References .....................................................................................................................12 The Hadoop Distributed File System: Architecture and Design  Page 2 Copyright © 2005 The Apache Software Foundation. All rights reserved.  1. Introduction The Hadoop File System (HDFS) is as a distributed file system running on commodityhardware. It has many similarities with existing distributed file systems. However, thedifferences from other distributed file systems are significant. HDFS is highly fault-tolerantand can be deployed on low-cost hardware. HDFS provides high throughput access toapplication data and is suitable for applications that have large datasets. HDFS relaxes a fewPOSIX requirements to enable streaming access to file system data. HDFS was srcinallybuilt as infrastructure for the open source web crawler Apache Nutch project. HDFS is partof the Hadoop Project, which is part of the Lucene Apache Project. The Project URL is here. 2. Assumptions and Goals 2.1. Hardware Failure Hardware Failure is the norm rather than the exception. The entire HDFS file system mayconsist of hundreds or thousands of server machines that stores pieces of file system data.The fact that there are a huge number of components and that each component has anon-trivial probability of failure means that some component of HDFS is alwaysnon-functional. Therefore, detection of faults and automatically recovering quickly fromthose faults are core architectural goals of HDFS. 2.2. Streaming Data Access Applications that run on HDFS need streaming access to their data sets. They are not generalpurpose applications that typically run on a general purpose file system. HDFS is designedmore for batch processing rather than interactive use by users. The emphasis is on throughputof data access rather than latency of data access. POSIX imposes many hard requirementsthat are not needed for applications that are targeted for HDFS. POSIX semantics in a fewkey areas have been traded off to further enhance data throughout rates. 2.3. Large Data Sets Applications that run on HDFS have large data sets. This means that a typical file in HDFS isgigabytes to terabytes in size. Thus, HDFS is tuned to support large files. It should providehigh aggregate data bandwidth and should scale to hundreds of nodes in a single cluster. Itshould support tens of millions of files in a single cluster. 2.4. Simple Coherency Model The Hadoop Distributed File System: Architecture and Design  Page 3 Copyright © 2005 The Apache Software Foundation. All rights reserved.  Most HDFS applications need write-once-read-many access model for files. A file oncecreated, written and closed need not be changed. This assumption simplifies data coherencyissues and enables high throughout data access. A Map-Reduce application or aWeb-Crawler application fits perfectly with this model. There is a plan to supportappending-writes to a file in future. 2.5. Moving computation is cheaper than moving data A computation requested by an application is most optimal if the computation can be donenear where the data is located. This is especially true when the size of the data set is huge.This eliminates network congestion and increase overall throughput of the system. Theassumption is that it is often better to migrate the computation closer to where the data islocated rather than moving the data to where the application is running. HDFS providesinterfaces for applications to move themselves closer to where the data is located. 2.6. Portability across Heterogeneous Hardware and Software Platforms HDFS should be designed in such a way that it is easily portable from one platform toanother. This facilitates widespread adoption of HDFS as a platform of choice for a large setof applications. 3. Namenode and Datanode HDFS has a master/slave architecture. An HDFS cluster consists of a single Namenode, amaster server that manages the filesystem namespace and regulates access to files by clients.In addition, there are a number of Datanodes, one per node in the cluster, which managestorage attached to the nodes that they run on. HDFS exposes a file system namespace andallows user data to be stored in files. Internally, a file is split into one or more blocks andthese blocks are stored in a set of Datanodes. The Namenode makes filesystem namespaceoperations like opening, closing, renaming etc. of files and directories. It also determines themapping of blocks to Datanodes. The Datanodes are responsible for serving read and writerequests from filesystem clients. The Datanodes also perform block creation, deletion, andreplication upon instruction from the Namenode.The Namenode and Datanode are pieces of software that run on commodity machines. Thesemachines are typically commodity Linux machines. HDFS is built using the Java language;any machine that support Java can run the Namenode or the Datanode. Usage of the highlyportable Java language means that HDFS can be deployed on a wide range of machines. Atypical deployment could have a dedicated machine that runs only the Namenode software.Each of the other machines in the cluster runs one instance of the Datanode software. The The Hadoop Distributed File System: Architecture and Design  Page 4 Copyright © 2005 The Apache Software Foundation. All rights reserved.
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks