apache
/
hadoop
mirror de https://github.com/apache/hadoop.git


			
				
					
						
						
							123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672
							<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<!--
 | Generated by Apache Maven Doxia at 2025-05-02
 | Rendered using Apache Maven Stylus Skin 1.5
-->
<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <title>Apache Hadoop 3.5.0-SNAPSHOT &#x2013; HDFS Namenode Fine-grained Locking</title>
    <style type="text/css" media="all">
      @import url("./css/maven-base.css");
      @import url("./css/maven-theme.css");
      @import url("./css/site.css");
    </style>
    <link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
        <meta name="Date-Revision-yyyymmdd" content="20250502" />
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
                </head>
  <body class="composite">
    <div id="banner">
                        <a href="http://hadoop.apache.org/" id="bannerLeft">
                                        <img src="http://hadoop.apache.org/images/hadoop-logo.jpg" alt="" />
                </a>
                              <a href="http://www.apache.org/" id="bannerRight">
                                        <img src="http://www.apache.org/images/asf_logo_wide.png" alt="" />
                </a>
            <div class="clear">
        <hr/>
      </div>
    </div>
    <div id="breadcrumbs">
            
                                     <div class="xright">            <a href="http://wiki.apache.org/hadoop" class="externalLink">Wiki</a>
            |
                <a href="https://gitbox.apache.org/repos/asf/hadoop.git" class="externalLink">git</a>
            |
                <a href="http://hadoop.apache.org/" class="externalLink">Apache Hadoop</a>
              
                                   &nbsp;| Last Published: 2025-05-02
              &nbsp;| Version: 3.5.0-SNAPSHOT
            </div>
      <div class="clear">
        <hr/>
      </div>
    </div>
    <div id="leftColumn">
      <div id="navcolumn">
             
                                                   <h5>General</h5>
                  <ul>
                  <li class="none">
                  <a href="../../index.html">Overview</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-common/SingleCluster.html">Single Node Setup</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-common/ClusterSetup.html">Cluster Setup</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-common/CommandsManual.html">Commands Reference</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-common/FileSystemShell.html">FileSystem Shell</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-common/Compatibility.html">Compatibility Specification</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-common/DownstreamDev.html">Downstream Developer's Guide</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-common/AdminCompatibilityGuide.html">Admin Compatibility Guide</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-common/InterfaceClassification.html">Interface Classification</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-common/filesystem/index.html">FileSystem Specification</a>
            </li>
          </ul>
                       <h5>Common</h5>
                  <ul>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-common/CLIMiniCluster.html">CLI Mini Cluster</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-common/FairCallQueue.html">Fair Call Queue</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-common/NativeLibraries.html">Native Libraries</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-common/Superusers.html">Proxy User</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-common/RackAwareness.html">Rack Awareness</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-common/SecureMode.html">Secure Mode</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-common/ServiceLevelAuth.html">Service Level Authorization</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-common/HttpAuthentication.html">HTTP Authentication</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-common/CredentialProviderAPI.html">Credential Provider API</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-kms/index.html">Hadoop KMS</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-common/Tracing.html">Tracing</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-common/UnixShellGuide.html">Unix Shell Guide</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-common/registry/index.html">Registry</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-common/AsyncProfilerServlet.html">Async Profiler</a>
            </li>
          </ul>
                       <h5>HDFS</h5>
                  <ul>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsDesign.html">Architecture</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html">User Guide</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-hdfs/HDFSCommands.html">Commands Reference</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html">NameNode HA With QJM</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithNFS.html">NameNode HA With NFS</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-hdfs/ObserverNameNode.html">Observer NameNode</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-hdfs/Federation.html">Federation</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-hdfs/ViewFs.html">ViewFs</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-hdfs/ViewFsOverloadScheme.html">ViewFsOverloadScheme</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsSnapshots.html">Snapshots</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsEditsViewer.html">Edits Viewer</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsImageViewer.html">Image Viewer</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html">Permissions and HDFS</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html">Quotas and HDFS</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-hdfs/LibHdfs.html">libhdfs (C API)</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-hdfs/WebHDFS.html">WebHDFS (REST API)</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-hdfs-httpfs/index.html">HttpFS</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html">Short Circuit Local Reads</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-hdfs/CentralizedCacheManagement.html">Centralized Cache Management</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html">NFS Gateway</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html">Rolling Upgrade</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-hdfs/ExtendedAttributes.html">Extended Attributes</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html">Transparent Encryption</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsMultihoming.html">Multihoming</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html">Storage Policies</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-hdfs/MemoryStorage.html">Memory Storage Support</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-hdfs/SLGUserGuide.html">Synthetic Load Generator</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html">Erasure Coding</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-hdfs/HDFSDiskbalancer.html">Disk Balancer</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsUpgradeDomain.html">Upgrade Domain</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsDataNodeAdminGuide.html">DataNode Admin</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-hdfs-rbf/HDFSRouterFederation.html">Router Federation</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-hdfs/HdfsProvidedStorage.html">Provided Storage</a>
            </li>
          </ul>
                       <h5>MapReduce</h5>
                  <ul>
                  <li class="none">
                  <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html">Tutorial</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredCommands.html">Commands Reference</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html">Compatibility with 1.x</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html">Encrypted Shuffle</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html">Pluggable Shuffle/Sort</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/DistributedCacheDeploy.html">Distributed Cache Deploy</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/SharedCacheSupport.html">Support for YARN Shared Cache</a>
            </li>
          </ul>
                       <h5>MapReduce REST APIs</h5>
                  <ul>
                  <li class="none">
                  <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredAppMasterRest.html">MR Application Master</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-hs/HistoryServerRest.html">MR History Server</a>
            </li>
          </ul>
                       <h5>YARN</h5>
                  <ul>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/YARN.html">Architecture</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/YarnCommands.html">Commands Reference</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html">Capacity Scheduler</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/FairScheduler.html">Fair Scheduler</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html">ResourceManager Restart</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html">ResourceManager HA</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/ResourceModel.html">Resource Model</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/NodeLabel.html">Node Labels</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/NodeAttributes.html">Node Attributes</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/WebApplicationProxy.html">Web Application Proxy</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/TimelineServer.html">Timeline Server</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html">Timeline Service V.2</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html">Writing YARN Applications</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html">YARN Application Security</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/NodeManager.html">NodeManager</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/DockerContainers.html">Running Applications in Docker Containers</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/RuncContainers.html">Running Applications in runC Containers</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/NodeManagerCgroups.html">Using CGroups</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/SecureContainer.html">Secure Containers</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/ReservationSystem.html">Reservation System</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/GracefulDecommission.html">Graceful Decommission</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html">Opportunistic Containers</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/Federation.html">YARN Federation</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/SharedCache.html">Shared Cache</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/UsingGpus.html">Using GPU</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/UsingFPGA.html">Using FPGA</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html">Placement Constraints</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/YarnUI2.html">YARN UI2</a>
            </li>
          </ul>
                       <h5>YARN REST APIs</h5>
                  <ul>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html">Introduction</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html">Resource Manager</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/NodeManagerRest.html">Node Manager</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/TimelineServer.html#Timeline_Server_REST_API_v1">Timeline Server</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html#Timeline_Service_v.2_REST_API">Timeline Service V.2</a>
            </li>
          </ul>
                       <h5>YARN Service</h5>
                  <ul>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/Overview.html">Overview</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/QuickStart.html">QuickStart</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/Concepts.html">Concepts</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/YarnServiceAPI.html">Yarn Service API</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/ServiceDiscovery.html">Service Discovery</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/SystemServices.html">System Services</a>
            </li>
          </ul>
                       <h5>Hadoop Compatible File Systems</h5>
                  <ul>
                  <li class="none">
                  <a href="../../hadoop-aliyun/tools/hadoop-aliyun/index.html">Aliyun OSS</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-aws/tools/hadoop-aws/index.html">Amazon S3</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-azure/index.html">Azure Blob Storage</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-azure-datalake/index.html">Azure Data Lake Storage</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-cos/cloud-storage/index.html">Tencent COS</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-huaweicloud/index.html">Huaweicloud OBS</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-tos/cloud-storage/index.html">VolcanoEngine TOS</a>
            </li>
          </ul>
                       <h5>Auth</h5>
                  <ul>
                  <li class="none">
                  <a href="../../hadoop-auth/index.html">Overview</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-auth/Examples.html">Examples</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-auth/Configuration.html">Configuration</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-auth/BuildingIt.html">Building</a>
            </li>
          </ul>
                       <h5>Tools</h5>
                  <ul>
                  <li class="none">
                  <a href="../../hadoop-streaming/HadoopStreaming.html">Hadoop Streaming</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-archives/HadoopArchives.html">Hadoop Archives</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-archive-logs/HadoopArchiveLogs.html">Hadoop Archive Logs</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-distcp/DistCp.html">DistCp</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-federation-balance/HDFSFederationBalance.html">HDFS Federation Balance</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-gridmix/GridMix.html">GridMix</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-rumen/Rumen.html">Rumen</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-resourceestimator/ResourceEstimator.html">Resource Estimator Service</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-sls/SchedulerLoadSimulator.html">Scheduler Load Simulator</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-common/Benchmarking.html">Hadoop Benchmarking</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-dynamometer/Dynamometer.html">Dynamometer</a>
            </li>
          </ul>
                       <h5>Reference</h5>
                  <ul>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-common/release/">Changelog and Release Notes</a>
            </li>
                  <li class="none">
                  <a href="../../api/index.html">Java API docs</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-common/UnixShellAPI.html">Unix Shell API</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-common/Metrics.html">Metrics</a>
            </li>
          </ul>
                       <h5>Configuration</h5>
                  <ul>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-common/core-default.xml">core-default.xml</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-hdfs/hdfs-default.xml">hdfs-default.xml</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-hdfs-rbf/hdfs-rbf-default.xml">hdfs-rbf-default.xml</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml">mapred-default.xml</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-yarn/hadoop-yarn-common/yarn-default.xml">yarn-default.xml</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-kms/kms-default.html">kms-default.xml</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-hdfs-httpfs/httpfs-default.html">httpfs-default.xml</a>
            </li>
                  <li class="none">
                  <a href="../../hadoop-project-dist/hadoop-common/DeprecatedProperties.html">Deprecated Properties</a>
            </li>
          </ul>
                                 <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
          <img alt="Built by Maven" src="./images/logos/maven-feather.png"/>
        </a>
                       
                               </div>
    </div>
    <div id="bodyColumn">
      <div id="contentBox">
        <!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at

      http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->
<h1>HDFS Namenode Fine-grained Locking</h1>
<ul>
<li><a href="#Overview">Overview</a></li>
<li><a href="#Design">Design</a>
<ul>
<li><a href="#RPC_Categorization">RPC Categorization</a></li>
<li><a href="#Full_Path_Lock">Full Path Lock</a></li>
<li><a href="#INodeFile_Lock">INodeFile Lock</a></li>
<li><a href="#DN_Lock">DN Lock</a></li>
<li><a href="#Global_Lock">Global Lock</a></li>
<li><a href="#Lock_Order">Lock Order</a></li>
<li><a href="#Lock_Pools">Lock Pools</a></li>
<li><a href="#Lock_Modes">Lock Modes</a></li></ul></li>
<li><a href="#Roadmap">Roadmap</a>
<ul>
<li><a href="#a"></a>
<ul>
<li><a href="#Stage_1:_Split_the_global_lock_into_FSLock_and_BMLock">Stage 1: Split the global lock into FSLock and BMLock</a></li>
<li><a href="#Stage_2:_Split_the_global_FSLock">Stage 2: Split the global FSLock</a></li>
<li><a href="#Stage_3:_Split_the_global_BMLock">Stage 3: Split the global BMLock</a></li></ul></li></ul></li>
<li><a href="#Configuration">Configuration</a></li>
<li><a href="#Adding_RPC">Adding RPC</a></li></ul>
<section>
<h2><a name="Overview"></a>Overview</h2>
<p>HDFS relies on a single master, the Namenode (NN), as its metadata center. From an architectural point of view, a few elements make NN the bottleneck of an HDFS cluster: * NN keeps the entire namespace in memory (directory tree, blocks, Datanode related info, etc.) * Read requests (<code>getListing</code>, <code>getFileInfo</code>, <code>getBlockLocations</code>) are served from memory. Write requests (<code>mkdir</code>, <code>create</code>, <code>addBlock</code>, <code>complete</code>) update the memory state and write a journal transaction into QJM. Both types of requests need a locking mechanism to ensure data consistency and correctness. * All requests are funneled into NN and have to go through the global FS lock. Each write operation acquires this lock in write mode and holds it until that operation is executed. This lock mode prevents concurrent execution of write operations even if they involve different branches of the directory tree.</p>
<p>NN fine-grained locking (FGL) implementation aims to alleviate this bottleneck by allowing concurrency of disjoint write operations.</p>
<p>JIRA: <a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-17366">HDFS-17366</a></p></section><section>
<h2><a name="Design"></a>Design</h2>
<p>In theory, fully independent operations can be processed concurrently, such as operations involving different subdirectory trees. As such, NN can split the global lock into the full path lock, just using the full path lock to protect a special subdirectory tree.</p><section>
<h3><a name="RPC_Categorization"></a>RPC Categorization</h3>
<p>Roughly, RPC operations handled by NN can be divided into 8 main categories</p>
<table border="0" class="bodyTable">
<thead>

<tr class="a">
<th> Category                               </th>
<th> Operations                                                                                                                                                                                                                                                                                                                                                                                                                                                                </th></tr>
</thead><tbody>

<tr class="b">
<td> Involving namespace tree               </td>
<td> <code>mkdir</code>, <code>create</code> (without overwrite), <code>getFileInfo</code> (without locations), <code>getListing</code> (without locations), <code>setOwner</code>, <code>setPermission</code>, <code>getStoragePolicy</code>, <code>setStoragePolicy</code>, <code>rename</code>, <code>isFileClosed</code>, <code>getFileLinkInfo</code>, <code>setTimes</code>, <code>modifyAclEntries</code>, <code>removeAclEntries</code>, <code>setAcl</code>, <code>getAcl</code>, <code>setXAttr</code>, <code>getXAttrs</code>, <code>listXAttrs</code>, <code>removeXAttr</code>, <code>checkAccess</code>, <code>getErasureCodingPolicy</code>, <code>unsetErasureCodingPolicy</code>, <code>getQuotaUsage</code>, <code>getPreferredBlockSize</code> </td></tr>
<tr class="a">
<td> Involving only blocks                  </td>
<td> <code>reportBadBlocks</code>, <code>updateBlockForPipeline</code>, <code>updatePipeline</code>                                                                                                                                                                                                                                                                                                                                                                                                             </td></tr>
<tr class="b">
<td> Involving only DNs                     </td>
<td> <code>registerDatanode</code>, <code>setBalancerBandwidth</code>, <code>sendHeartbeat</code>                                                                                                                                                                                                                                                                                                                                                                                                               </td></tr>
<tr class="a">
<td> Involving both namespace tree &amp; blocks </td>
<td> <code>getBlockLocation</code>, <code>create</code> (with overwrite), <code>append</code>, <code>setReplication</code>, <code>abandonBlock</code>, <code>addBlock</code>, <code>getAdditionalDatanode</code>, <code>complete</code>, <code>concat</code>, <code>truncate</code>, <code>delete</code>, <code>getListing</code> (with locations), <code>getFileInfo</code> (with locations), <code>recoverLease</code>, <code>listCorruptFileBlocks</code>, <code>fsync</code>, <code>commitBlockSynchronization</code>, <code>RedundancyMonitor</code>, <code>processMisReplicatedBlocks</code>                                                                                              </td></tr>
<tr class="b">
<td> Involving both DNs &amp; blocks            </td>
<td> <code>getBlocks</code>, <code>errorReport</code>                                                                                                                                                                                                                                                                                                                                                                                                                                                </td></tr>
<tr class="a">
<td> Involving namespace tree, DNs &amp; blocks </td>
<td> <code>blockReport</code>, <code>blockReceivedAndDeleted</code>, <code>HeartbeatManager</code>, <code>Decommission</code>                                                                                                                                                                                                                                                                                                                                                                                              </td></tr>
<tr class="b">
<td> Requiring locking the entire namespace </td>
<td> <code>rollEditLog</code>, <code>startCommonService</code>, <code>startActiveService</code>, <code>saveNamespace</code>, <code>rollEdits</code>, <code>EditLogTailer</code>, <code>rollingUpgrade</code>                                                                                                                                                                                                                                                                                                                                                </td></tr>
<tr class="a">
<td> Requiring no locking                   </td>
<td> <code>getServerDefaults</code>, <code>getStats</code>                                                                                                                                                                                                                                                                                                                                                                                                                                           </td></tr>
</tbody>
</table>
<p>For operations involving namespace tree, fully independent operations can be handled by NN concurrently. Almost all of them use the full path as a parameter, e.g. <code>create</code>, <code>mkdirs</code>, <code>getFileInfo</code>, etc. So we can use a full path lock to make them thread-safe.</p>
<p>For operations involving blocks, one block belongs to one and only one <code>INodeFile</code>, so NN can use the namespace tree to make these operations thread-safe.</p>
<p>For operations involving DNs, NN needs a separate DN lock because DNs operate separately from the namespace tree.</p>
<p>For operations requiring the entire namespace locked, the global lock can be used to make these operations thread-safe. In general, these operations have low frequency and thus low impact despite the global locking.</p></section><section>
<h3><a name="Full_Path_Lock"></a>Full Path Lock</h3>
<p>Used to protect operations involving namespace tree.</p>
<p>All of these operations receive a path or INodeID as a parameter and can further be divided into 3 main subcategories: 1. Parameters contain only one path (<code>create</code>, <code>mkdir</code>) 2. Parameters contain multiple paths (<code>rename</code>, <code>concat</code>) 3. Parameters contain INodeID (<code>addBlock</code>, <code>complete</code>)</p>
<p>For type 1, NN acquires a full path lock according to its semantics. Take <code>setPermission(&quot;/a/b/c/f.txt&quot;)</code> for example, the set of locks to acquire are ReadLock(&#x201c;/&#x201d;), ReadLock(&#x201c;a&#x201d;), ReadLock(&#x201c;b&#x201d;), ReadLock(&#x201c;c&#x201d;) and WriteLock(&#x201c;f.txt&#x201d;). Different lock patterns are explained in a later section.</p>
<p>For type 2, NN acquires full path locks in a predefined order, such as the lexicographic order, to avoid deadlocks.</p>
<p>For type 3, NN acquires a full path lock by in the following fashion: - Unsafely obtains full path recursively - Acquires the full path lock according to the lock mode - Rechecks whether the last node of the full path is equal to the INodeID given - If not, that means that the <code>INodeFile</code> might have been renamed or concatenated, need to retry - If the max retry attempts have been reached, throw a <code>RetryException</code> to client to let client retry</p></section><section>
<h3><a name="INodeFile_Lock"></a><code>INodeFile</code> Lock</h3>
<p>Used to protect operations involving blocks.</p>
<p>One block belongs to one and only one <code>INodeFile</code>, so NN can use the INodeFile lock to make operations thread-safe. Normally, there is no need to acquire the full path lock since changing the namespace tree structure does not affect the block.</p>
<p><code>concat</code> might change the <code>INodeFile</code> a block belongs to. Since both block related operations and <code>concat</code> need to acquire the <code>INodeFile</code> write lock, only one of them can be processed at a time.</p></section><section>
<h3><a name="DN_Lock"></a>DN Lock</h3>
<p>Used to protect operations involving DNs.</p>
<p>NN uses a <code>DatanodeDescriptor</code> object to store the information of one DN and uses <code>DatanodeManager</code> to manage all DNs in the memory. <code>DatanodeDescriptor</code> uses <code>DatanodeStorageInfo</code> to store the information of one storage device on one DN.</p>
<p>DNs have nothing to do with the namespace tree, so NN uses a separate DN lock for these operations. Since DNs are independent of one another, NN can assign a lock to each DN.</p></section><section>
<h3><a name="Global_Lock"></a>Global Lock</h3>
<p>Used for operations requiring the entire namespace locked.</p>
<p>There are some operations that need to lock the entire namespace, e.g. safe mode related operations, HA service related, etc. NN uses the global lock to make these operations thread-safe. Outside of these infrequent operations that require the global write lock, all other operations have to acquire the global read lock. The only exception to this rule is JMX operations being allowed to bypass locking entirely to ensure that metrics can be collected regardless of long write lock holding.</p></section><section>
<h3><a name="Lock_Order"></a>Lock Order</h3>
<p>As mentioned above, there are the global lock, DN lock, and full path lock. NN acquires locks in this specific order to avoid deadlocks.</p>
<p>Locks are to be acquired in this order: - Global &gt; DN &gt; Full path - Global &gt; DN &gt; Last <code>INodeFile</code></p>
<p>Possible lock combinations are as follows: - Global write lock - Global read lock &gt; Full path lock - Global read lock &gt; DN read/write lock - Global read lock &gt; DN read/write lock &gt; Read/Write lock of last <code>INodeFile</code> - Global read lock &gt; DN read/write lock &gt; Full path lock</p></section><section>
<h3><a name="Lock_Pools"></a>Lock Pools</h3>
<p>NN allocates locks as needed to the INodes used by active threads, and deletes them after the locks are no longer in use. Locks for commonly accessed <code>INode</code>s like the root are cached.</p>
<p>NN uses an <code>INodeLockPool</code> to manage these locks. The lock pool: - Returns a closeable lock for an INode based on the lock type, - Removes this lock if it is no longer used by any threads.</p>
<p>Similar to <code>INodeLockPool</code>, a <code>DNLockPool</code> is used to manage the locks for DNs. Unlike <code>INodeLockPool</code>, <code>DNLockPool</code> keeps all locks in memory due to the comparatively lower number of locks.</p></section><section>
<h3><a name="Lock_Modes"></a>Lock Modes</h3>
<p>Operations related to namespace tree have different semantics and may involve the modification or access of different INodes, for example: <code>getBlockLocation</code> only accesses the last iNodeFile, <code>delete</code> modifies both the parent and the last INode, <code>mkdir</code> may modify multiple ancestor INodes.</p>
<p>Four lock modes (plus no locking): - LOCK_READ - This lock mode acquires the read locks for all INodes down the path. - Example operations: <code>getBlockLocation</code>, <code>getFileInfo</code>. - LOCK_WRITE - This lock mode acquires the write lock for the last INode and the read locks for all ancestor INodes in the full path. - Example operations: <code>setPermission</code>, <code>setReplication</code>. - LOCK_PARENT - This lock mode acquires the write lock for the last two INodes and the read locks for all remaining ancestor INodes in the full path. - Example operations: <code>rename</code>, <code>delete</code>, <code>create</code> (when the parent directory exists). - LOCK_ANCESTOR - This lock mode acquires the write lock for the last existing INode and the read locks for all remaining ancestor INodes in the full path. - Example operations: <code>mkdir</code>, <code>create</code> (when the parent directory doesn&#x2019;t exist). - NONE - This lock mode does not acquire any locks for the given path.</p></section></section><section>
<h2><a name="Roadmap"></a>Roadmap</h2><section><section>
<h4><a name="Stage_1:_Split_the_global_lock_into_FSLock_and_BMLock"></a>Stage 1: Split the global lock into FSLock and BMLock</h4>
<p>Split the global lock into two global locks, FSLock and BMLock. - FSLock for operations that relate to namespace tree. - BMLock for operations related to blocks and/or operations related to DNs. - Both FSLock and BMLock for HA related operations. After this step, FGL contains global FSLock and global BMLock.</p>
<p>No big logic changes in this step. The original logic with the global lock retains. This step aims to make the lock mode configurable.</p>
<p>JIRA: <a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-17384">HDFS-17384</a> [Progress: Done]</p></section><section>
<h4><a name="Stage_2:_Split_the_global_FSLock"></a>Stage 2: Split the global FSLock</h4>
<p>After splitting the global lock into FSLock and BMLock, this step aims to split the global FSLock into full path locks so that fully independent operations that only involve namespace tree can be processed concurrently. In this step, NN still uses the global BMLock to protect block related operations and DN related operations. After this step, FGL contains global FSLock, full path lock, and global BMLock.</p>
<p>JIRA: <a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-17385">HDFS-17385</a> [Progress: Ongoing]</p></section><section>
<h4><a name="Stage_3:_Split_the_global_BMLock"></a>Stage 3: Split the global BMLock</h4>
<p>This step aims to split the global BMLock into full path locks and DN locks. After this step, FGL contains global FSLock, DN lock, and full path lock.</p>
<p>JIRA: <a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-17386">HDFS-17386</a> [Progress: Ongoing]</p></section></section></section><section>
<h2><a name="Configuration"></a>Configuration</h2>
<p>NN FGL implementation can be used by adding this configuration to <code>hdfs-site.xml</code>.</p>

<div class="source">
<div class="source">
<pre>&lt;property&gt;
  &lt;name&gt;dfs.namenode.lock.model.provider.class&lt;/name&gt;
  &lt;value&gt;org.apache.hadoop.hdfs.server.namenode.fgl.FineGrainedFSNamesystemLock&lt;/value&gt;
  &lt;description&gt;
    An implementation class of FSNamesystem lock.
    Defaults to GlobalFSNamesystemLock.class
  &lt;/description&gt;
&lt;/property&gt;
</pre></div></div>

<p>The lock manager class must implement the interface defined by <code>org.apache.hadoop.hdfs.server.namenode.fgl.FSNLockManager</code>. Currently, there are two implementations: * <code>org.apache.hadoop.hdfs.server.namenode.fgl.GlobalFSNamesystemLock</code>: the original lock mode that utilizes one global FS lock, also the default value for this config; * <code>org.apache.hadoop.hdfs.server.namenode.fgl.FineGrainedFSNamesystemLock</code>: FGL implementation.</p></section><section>
<h2><a name="Adding_RPC"></a>Adding RPC</h2>
<p>For developers adding a new RPC operation, the operation should follow FGL locking schematic to ensure data integrity: * Global FSLock should be acquired in read mode, unless it is an administrative operation (related to HA, edit logs, etc.) * If the operation requires access/modification of <code>DatanodeDescriptor</code> and/or <code>DatanodeStorageInfo</code>, DN lock should be acquired in read/write mode accordingly. * Only applicable in stage 3 once DN lock is implemented. During stage 1 and stage 2, global BMLock is to be used instead. * If the operation deals with one or more paths/blocks, the full path lock(s) should be acquired based on the implementation details described above. It is best to check an existing RPC operation that has a similar method of access to the new operation to consult the lock implementation.</p></section>
      </div>
    </div>
    <div class="clear">
      <hr/>
    </div>
    <div id="footer">
      <div class="xright">
        &#169;            2008-2025
              Apache Software Foundation
            
                          - <a href="http://maven.apache.org/privacy-policy.html">Privacy Policy</a>.
        Apache Maven, Maven, Apache, the Apache feather logo, and the Apache Maven project logos are trademarks of The Apache Software Foundation.
      </div>
      <div class="clear">
        <hr/>
      </div>
    </div>
  </body>
</html>