hadoop/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

Hadoop HDFS Change Log

Trunk (Unreleased)

  INCOMPATIBLE CHANGES

    HDFS-4053. Increase the default block size. (eli)

    HDFS-3034. Remove the deprecated DFSOutputStream.sync() method.  (szetszwo)

  NEW FEATURES

    HDFS-3125. Add JournalService to enable Journal Daemon. (suresh)

    HDFS-3601. Add BlockPlacementPolicyWithNodeGroup to support block placement
    with 4-layer network topology.  (Junping Du via szetszwo)

    HDFS-3495. Update Balancer to support new NetworkTopology with NodeGroup.
    (Junping Du via szetszwo)

    HADOOP-8562. Enhancements to support Hadoop on Windows Server and Windows
    Azure environments. (See breakdown of tasks below for subtasks and
    contributors)

    HDFS-4659 Support setting execution bit for regular files (Brandon Li via sanjay)

  IMPROVEMENTS

    HDFS-4665. Move TestNetworkTopologyWithNodeGroup to common.
    (Junping Du via llu)

    HDFS-1620. Rename HdfsConstants -> HdfsServerConstants, FSConstants ->
               HdfsConstants. (Harsh J Chouraria via atm)

    HDFS-2197. Refactor RPC call implementations out of NameNode class (todd)

    HDFS-2572. Remove unnecessary double-check in DN#getHostName. (harsh)

    HDFS-2857. Cleanup BlockInfo class. (suresh)

    HDFS-2786. Fix host-based token incompatibilities in DFSUtil. (Kihwal Lee
    via jitendra)

    HDFS-2486. Remove unnecessary priority level checks in
    UnderReplicatedBlocks.  (Uma Maheswara Rao G via szetszwo)

    HDFS-2878. Fix TestBlockRecovery and move it back into main test directory.
    (todd)

    HDFS-2655. BlockReaderLocal#skip performs unnecessary IO.
    (Brandon Li via jitendra) 

    HDFS-3009. Remove duplicate code in DFSClient#isLocalAddress by using 
    NetUtils. (Hari Mankude via suresh)

    HDFS-3002. TestNameNodeMetrics need not wait for metrics update.
    (suresh)

    HDFS-3016. Security in unit tests. (Jaimin Jetly via jitendra)

    HDFS-3030. Remove getProtocolVersion and getProtocolSignature from translators.
    (jitendra)

    HDFS-2976. Remove unnecessary method (tokenRefetchNeeded) in DFSClient.

    HDFS-3111. Missing license headers in trunk. (umamahesh)

    HDFS-3091. Update the usage limitations of ReplaceDatanodeOnFailure policy in
    the config description for the smaller clusters. (szetszwo via umamahesh)

    HDFS-309. FSEditLog should log progress during replay. (Sho Shimauchi
    via todd)

    HDFS-3131. Improve TestStorageRestore. (Brandon Li via atm)

    HDFS-3178. Add states and state handler for journal synchronization in
    JournalService.  (szetszwo)

    HDFS-3273. Refactor BackupImage and FSEditLog, and rename
    JournalListener.rollLogs(..) to startLogSegment(..).  (szetszwo)

    HDFS-3292. Remove the deprecated DiskStatus, getDiskStatus(), getRawUsed()
    and getRawCapacity() from DistributedFileSystem.  (Arpit Gupta via szetszwo)

    HADOOP-8285. HDFS changes for Use ProtoBuf for RpcPayLoadHeader. (sanjay
    radia)

    HDFS-2743. Streamline usage of bookkeeper journal manager. 
    (Ivan Kelly via umamahesh)

    HDFS-3293. Add toString(), equals(..) and hashCode() to JournalInfo.
    (Hari Mankude via szetszwo)

    HDFS-3197. Incorrect class comments in a few tests. (Andy Isaacson via eli)

    HDFS-3476. Correct the default used in TestDFSClientRetries.busyTest()
    after HDFS-3462 (harsh)

    HDFS-3478. Test quotas with Long.Max_Value. (Sujay Rau via eli)

    HDFS-3498. Support replica removal in BlockPlacementPolicy and make
    BlockPlacementPolicyDefault extensible for reusing code in subclasses.
    (Junping Du via szetszwo)

    HDFS-3630 Modify TestPersistBlocks to use both flush and hflush  (sanjay)

    HDFS-3768. Exception in TestJettyHelper is incorrect. 
    (Eli Reisman via jghoman)

    HDFS-3851. DFSOutputStream class code cleanup. (Jing Zhao via suresh)

    HDFS-2580. NameNode#main(...) can make use of GenericOptionsParser. (harsh)

    HDFS-3880. Use Builder to build RPC server in HDFS.
    (Brandon Li vias suresh)

    HDFS-2127. Add a test that ensure AccessControlExceptions contain
    a full path. (Stephen Chu via eli)

    HDFS-3995. Use DFSTestUtil.createFile() for file creation and 
    writing in test cases. (Jing Zhao via suresh)

    HDFS-3735. NameNode WebUI should allow sorting live datanode list by fields
    Block Pool Used, Block Pool Used(%) and Failed Volumes.
    (Brahma Reddy Battula via suresh)

    HDFS-4052. BlockManager#invalidateWork should print log outside the lock.
    (Jing Zhao via suresh)

    HDFS-4200. Reduce the size of synchronized sections in PacketResponder.
    (suresh)

    HDFS-3358. Specify explicitly that the NN UI status total is talking
    of persistent objects on heap. (harsh)

    HDFS-4234. Use generic code for choosing datanode in Balancer.  (szetszwo)

    HDFS-4633 TestDFSClientExcludedNodes fails sporadically if excluded nodes
    cache expires too quickly  (Chris Nauroth via Sanjay)

    HDFS-347. DFS read performance suboptimal when client co-located on nodes
    with data. (Colin Patrick McCabe via todd and atm)

  OPTIMIZATIONS

  BUG FIXES

    HDFS-2299. TestOfflineEditsViewer is failing on trunk. (Uma Maheswara Rao G
               via atm)
    HDFS-2310. TestBackupNode fails since HADOOP-7524 went in.
               (Ivan Kelly via todd)

    HDFS-2313. Rat excludes has a typo for excluding editsStored files. (atm)

    HDFS-2314. MRV1 test compilation broken after HDFS-2197 (todd)

    HDFS-46.   Change default namespace quota of root directory from
    Integer.MAX_VALUE to Long.MAX_VALUE. (Uma Maheswara Rao G via szetszwo)

    HDFS-2373. Commands using WebHDFS and hftp print unnecessary debug 
    info on the console with security enabled. (Arpit Gupta via suresh)

    HDFS-2776. Missing interface annotation on JournalSet. 
    (Brandon Li via jitendra)

    HDFS-3037. TestMulitipleNNDataBlockScanner#testBlockScannerAfterRestart is
    racy. (atm)

    HDFS-3116. Typo in fetchdt error message. (AOE Takashi via atm)

    HDFS-3126. Journal stream from Namenode to BackupNode needs to have 
    timeout. (Hari Mankude via suresh)

    HDFS-3121. Add HDFS tests for HADOOP-8014 change. (John George via
    suresh)

    HDFS-3119. Overreplicated block is not deleted even after the replication 
    factor is reduced after sync follwed by closing that file. (Ashish Singhi 
    via umamahesh)

    HDFS-3265. PowerPc Build error. (Kumar Ravi via mattf)

    HDFS-2312. FSNamesystem javadoc incorrectly says its for DNs. (harsh)

    HDFS-3163. TestHDFSCLI.testAll fails if the user name is not all lowercase.
    (Brandon Li via atm)

    HDFS-3462. TestDFSClientRetries.busyTest() should restore default
    xceiver count in the config. (Madhukara Phatak via harsh)

    HDFS-3550. Fix raid javadoc warnings. (Jason Lowe via daryn)

    HDFS-3549. Fix dist tar build fails in hadoop-hdfs-raid project. (Jason Lowe via daryn)

    HDFS-3482. hdfs balancer throws ArrayIndexOutOfBoundsException 
    if option is specified without values. ( Madhukara Phatak via umamahesh) 

    HDFS-3614. Revert unused MiniDFSCluster constructor from HDFS-3049.
    (acmurthy via eli)

    HDFS-3792. Fix two findbugs introduced by HDFS-3695 (todd)

    HDFS-3827. TestHASafeMode#assertSafemode method should be made static.
    (Jing Zhao via suresh)

    HDFS-3834. Remove unused static fields NAME, DESCRIPTION and Usage from
    Command. (Jing Zhao via suresh)

    HADOOP-8158. Interrupting hadoop fs -put from the command line
    causes a LeaseExpiredException. (daryn via harsh)

    HDFS-2434. TestNameNodeMetrics.testCorruptBlock fails intermittently.
    (Jing Zhao via suresh)

    HDFS-4067. TestUnderReplicatedBlocks intermittently fails due to 
    ReplicaAlreadyExistsException. (Jing Zhao via suresh)

    HDFS-4115. TestHDFSCLI.testAll fails one test due to number format.
    (Trevor Robinson via suresh)

    HDFS-4165. Faulty sanity check in FsDirectory.unprotectedSetQuota.
    (Binglin Chang via suresh)

    HDFS-4105. The SPNEGO user for secondary namenode should use the web 
    keytab. (Arpit Gupta via jitendra)

    HDFS-4003. test-patch should build the common native libs before
    running hdfs tests. (Colin Patrick McCabe via eli)

    HDFS-4240. For nodegroup-aware block placement, when a node is excluded,
    the nodes in the same nodegroup should also be excluded.  (Junping Du
    via szetszwo)

    HDFS-4260 Fix HDFS tests to set test dir to a valid HDFS path as opposed
    to the local build path (Chri Nauroth via Sanjay)

    HADOOP-8957 HDFS tests for AbstractFileSystem#IsValidName should be overridden for
    embedded file systems like ViewFs (Chris Nauroth via Sanjay Radia)

    HDFS-4310. fix test org.apache.hadoop.hdfs.server.datanode.
    TestStartSecureDataNode (Ivan A. Veselovsky via atm)

    HDFS-4275. MiniDFSCluster-based tests fail on Windows due to failure
    to delete test namenode directory. (Chris Nauroth via suresh)

    HDFS-4338. TestNameNodeMetrics#testCorruptBlock is flaky. (Andrew Wang via
    atm)

    HDFS-4261. Fix bugs in Balaner causing infinite loop and
    TestBalancerWithNodeGroup timeing out.  (Junping Du via szetszwo)

    HDFS-4382. Fix typo MAX_NOT_CHANGED_INTERATIONS. (Ted Yu via suresh)

    HDFS-4391. TestDataTransferKeepalive fails when tests are executed in a
    certain order. (Andrew Wang via atm)

    HDFS-4586. TestDataDirs.testGetDataDirsFromURIs fails with all directories
    in dfs.datanode.data.dir are invalid. (Ivan Mitic via atm)

    HDFS-4732. Fix TestDFSUpgradeFromImage which fails on Windows due to
    failure to unpack old image tarball that contains hard links.
    (Chris Nauroth via szetszwo)

    HDFS-4757. Update FSDirectory#inodeMap when replacing an INodeDirectory
    while setting quota.  (Jing Zhao via szetszwo)

    HDFS-4761. When resetting FSDirectory, the inodeMap should also be reset.
    (Jing Zhao via szetszwo)

    HDFS-4687. TestDelegationTokenForProxyUser#testWebHdfsDoAs is flaky with
    JDK7. (Andrew Wang via atm)

  BREAKDOWN OF HADOOP-8562 and HDFS-3602 SUBTASKS AND RELATED JIRAS

    HDFS-4145. Merge hdfs cmd line scripts from branch-1-win. (David Lao,
    Bikas Saha, Lauren Yang, Chuan Liu, Thejas M Nair and Ivan Mitic via suresh)

    HDFS-4163. HDFS distribution build fails on Windows. (Chris Nauroth via
    suresh)

    HDFS-4316. branch-trunk-win contains test code accidentally added during 
    work on fixing tests on Windows. (Chris Nauroth via suresh)

    HDFS-4297. Fix issues related to datanode concurrent reading and writing on
    Windows. (Arpit Agarwal, Chuan Liu via suresh)

    HDFS-4573. Fix TestINodeFile on Windows. (Arpit Agarwal via suresh)

    HDFS-4572. Fix TestJournal failures on Windows. (Arpit Agarwal via suresh)

    HDFS-4287. HTTPFS tests fail on Windows. (Chris Nauroth via suresh)

    HDFS-4593. TestSaveNamespace fails on Windows. (Arpit Agarwal via suresh)

    HDFS-4582. TestHostsFiles fails on Windows. (Ivan Mitic via suresh)

    HDFS-4603. TestMiniDFSCluster fails on Windows. (Ivan Mitic via suresh)

    HDFS-4604. TestJournalNode fails on Windows. (Ivan Mitic via suresh)

    HDFS-4625. Make TestNNWithQJM#testNewNamenodeTakesOverWriter work on
    Windows. (Ivan Mitic via suresh)

    HDFS-4674. TestBPOfferService fails on Windows due to failure parsing 
    datanode data directory as URI. (Chris Nauroth via suresh)

    HDFS-4725. Fix HDFS file handle leaks in FSEditLog, NameNode,
    OfflineEditsBinaryLoader and some tests.  (Chris Nauroth via szetszwo)

    HDFS-4615. Fix TestDFSShell failures on Windows.  (Arpit Agarwal
    via szetszwo)

    HDFS-4584. Skip TestNNWithQJM.testNewNamenodeTakesOverWriter() on Windows.
    (Arpit Agarwal via szetszwo)

    HDFS-4741. TestStorageRestore#testStorageRestoreFailure fails on Windows.
    (Arpit Agarwal via suresh)

    HDFS-4743. TestNNStorageRetentionManager fails on Windows.
    (Chris Nauroth via suresh)

    HDFS-4740. Fixes for a few test failures on Windows.
    (Arpit Agarwal via suresh)

    HDFS-4722. TestGetConf#testFederation times out on Windows.
    (Ivan Mitic via suresh)

    HDFS-4705. Address HDFS test failures on Windows because of invalid
    dfs.namenode.name.dir. (Ivan Mitic via suresh)

    HDFS-4734. HDFS Tests that use ShellCommandFencer are broken on Windows.
    (Arpit Agarwal via suresh)

    HDFS-4610. Use common utils FileUtil#setReadable/Writable/Executable and 
    FileUtil#canRead/Write/Execute. (Ivan Mitic via suresh)

  BREAKDOWN OF HDFS-347 SUBTASKS AND RELATED JIRAS

    HDFS-4353. Encapsulate connections to peers in Peer and PeerServer classes.
    (Colin Patrick McCabe via todd)
    
    HDFS-4354. Create DomainSocket and DomainPeer and associated unit tests.
    (Colin Patrick McCabe via todd)
    
    HDFS-4356. BlockReaderLocal should use passed file descriptors rather than paths.
    (Colin Patrick McCabe via todd)
    
    HDFS-4388. DomainSocket should throw AsynchronousCloseException when appropriate.
    (Colin Patrick McCabe via todd)
    
    HDFS-4390. Bypass UNIX domain socket unit tests when they cannot be run.
    (Colin Patrick McCabe via todd)
    
    HDFS-4400. DFSInputStream#getBlockReader: last retries should ignore the cache
    (Colin Patrick McCabe via todd)
    
    HDFS-4401. Fix bug in DomainSocket path validation
    (Colin Patrick McCabe via todd)
    
    HDFS-4402. Some small DomainSocket fixes: avoid findbugs warning, change
    log level, etc. (Colin Patrick McCabe via todd)
    
    HDFS-4418. increase default FileInputStreamCache size (todd)
    
    HDFS-4416. Rename dfs.datanode.domain.socket.path to dfs.domain.socket.path
    (Colin Patrick McCabe via todd)
    
    HDFS-4417. Fix case where local reads get disabled incorrectly
    (Colin Patrick McCabe and todd via todd)
    
    HDFS-4433. Make TestPeerCache not flaky (Colin Patrick McCabe via todd)
    
    HDFS-4438. TestDomainSocket fails when system umask is set to 0002. (Colin
    Patrick McCabe via atm)
    
    HDFS-4440. Avoid annoying log message when dfs.domain.socket.path is not
    set. (Colin Patrick McCabe via atm)
    
    HDFS-4473. Don't create domain socket unless we need it. (Colin Patrick McCabe via atm)
    
    HDFS-4485. DN should chmod socket path a+w. (Colin Patrick McCabe via atm)
    
    HDFS-4453. Make a simple doc to describe the usage and design of the
    shortcircuit read feature. (Colin Patrick McCabe via atm)
    
    HDFS-4496. DFSClient: don't create a domain socket unless we need it (Colin
    Patrick McCabe via todd)
    
    HDFS-347: style cleanups (Colin Patrick McCabe via atm)
    
    HDFS-4538. Allow use of legacy blockreader (Colin Patrick McCabe via todd)

Release 2.0.5-beta - UNRELEASED

  INCOMPATIBLE CHANGES

    HDFS-4434. Provide a mapping from INodeId to INode. (suresh)

    HDFS-4305. Add a configurable limit on number of blocks per file, and min
    block size. (Andrew Wang via atm)

  NEW FEATURES

    HDFS-1804. Add a new block-volume device choosing policy that looks at
    free space. (atm)

    HDFS-4296. Reserve layout version for release 1.2.0. (suresh)

    HDFS-4334. Add a unique id to INode.  (Brandon Li via szetszwo)

    HDFS-4339. Persist inode id in fsimage and editlog. (Brandon Li via
    suresh)

    HDFS-4340. Update addBlock() to inculde inode id as additional argument.
    (Brandon Li via suresh)

    HDFS-4502. JsonUtil.toFileStatus(..) should check if the fileId property
    exists.  (Brandon Li via suresh)

    HDFS-2576. Enhances the DistributedFileSystem's create API so that clients
    can specify favored datanodes for a file's blocks. (ddas)

  IMPROVEMENTS

    HDFS-4222. NN is unresponsive and loses heartbeats from DNs when 
    configured to use LDAP and LDAP has issues. (Xiaobo Peng, suresh)

    HDFS-4304. Make FSEditLogOp.MAX_OP_SIZE configurable. (Colin Patrick
    McCabe via atm)

    HDFS-4518. Finer grained metrics for HDFS capacity.
    (Arpit Agarwal via suresh)

    HDFS-4519. Support overriding jsvc binary and log file locations
    when launching secure datanode. (Chris Nauroth via suresh)

    HDFS-4569. Small image transfer related cleanups.
    (Andrew Wang via suresh)

    HDFS-4521. Invalid network toploogies should not be cached. (Colin Patrick
    McCabe via atm)

    HDFS-4246. The exclude node list should be more forgiving, for each output
    stream. (harsh via atm)

    HDFS-4635. Move BlockManager#computeCapacity to LightWeightGSet. (suresh)

    HDFS-4621. Additional logging to help diagnose slow QJM syncs. (todd)

    HDFS-4618. Default transaction interval for checkpoints is too low. (todd)

    HDFS-4525. Provide an API for knowing that whether file is closed or not. 
    (SreeHari via umamahesh)

    HDFS-3940. Add Gset#clear method and clear the block map when namenode is
    shutdown. (suresh)

    HDFS-4679. Namenode operation checks should be done in a consistent
    manner. (suresh)

    HDFS-4693. Some test cases in TestCheckpoint do not clean up after 
    themselves. (Arpit Agarwal, suresh via suresh)

    HDFS-3817. Avoid printing SafeModeException stack trace.
    (Brandon Li via suresh)

    HDFS-4124. Refactor INodeDirectory#getExistingPathINodes() to enable 
    returning more than INode array. (Jing Zhao via suresh)

    HDFS-4151. Change the methods in FSDirectory to pass INodesInPath instead
    of INode[] as a parameter. (szetszwo)

    HDFS-4129. Add utility methods to dump NameNode in memory tree for 
    testing. (szetszwo via suresh)

    HDFS-4152. Add a new class BlocksMapUpdateInfo for the parameter in
    INode.collectSubtreeBlocksAndClear(..). (Jing Zhao via szetszwo)

    HDFS-4206. Change the fields in INode and its subclasses to private.
    (szetszwo)

    HDFS-4215. Remove locking from addToParent(..) since it is used in image
    loading, and add INode.isFile().  (szetszwo)

    HDFS-4243. When replacing an INodeDirectory, the parent pointers of the
    children of the child have to be updated to the new child.  (Jing Zhao
    via szetszwo)

    HDFS-4209. Clean up the addNode/addChild/addChildNoQuotaCheck methods in
    FSDirectory and INodeDirectory. (szetszwo)

    HDFS-4346. Add SequentialNumber as a base class for INodeId and
    GenerationStamp.  (szetszwo)

    HDFS-4721. Speed up lease recovery by avoiding stale datanodes and choosing
    the datanode with the most recent heartbeat as the primary.  (Varun Sharma
    via szetszwo)

  OPTIMIZATIONS

  BUG FIXES
    
    HDFS-4470. Several HDFS tests attempt file operations on invalid HDFS
    paths when running on Windows. (Chris Nauroth via suresh)

    HDFS-4471. Namenode WebUI file browsing does not work with wildcard
    addresses configured. (Andrew Wang via atm)

    HDFS-4342. Directories configured in dfs.namenode.edits.dir.required
    but not in dfs.namenode.edits.dir are silently ignored.  (Arpit Agarwal
    via szetszwo)

    HDFS-4482. ReplicationMonitor thread can exit with NPE due to the race 
    between delete and replication of same file. (umamahesh)

    HDFS-4269. Datanode rejects all datanode registrations from localhost
    in single-node developer setup on Windows. (Chris Nauroth via suresh)

    HDFS-4235. When outputting XML, OfflineEditsViewer can't handle some edits
    containing non-ASCII strings. (Colin Patrick McCabe via atm)

    HDFS-4541. Set hadoop.log.dir and hadoop.id.str when starting secure
    datanode to write the logs to right dir by default. (Arpit Gupta via
    suresh)

    HDFS-4540. Namenode http server should use the web authentication 
    keytab for spnego principal. (Arpit Gupta via suresh)

    HDFS-4544. Error in deleting blocks should not do check disk, for
    all types of errors. (Arpit Agarwal via suresh)

    HDFS-4565. Use DFSUtil.getSpnegoKeytabKey() to get the spnego keytab key
    in secondary namenode and namenode http server. (Arpit Gupta via suresh)

    HDFS-4571. WebHDFS should not set the service hostname on the server side. 
    (tucu)

    HDFS-4013. TestHftpURLTimeouts throws NPE. (Chao Shi via suresh)

    HDFS-4592. Default values for access time precision are out of sync between
    hdfs-default.xml and the code. (atm)

    HDFS-4522. LightWeightGSet expects incrementing a volatile to be atomic.
    (Colin Patrick McCabe via atm)

    HDFS-4484. libwebhdfs compilation broken with gcc 4.6.2. (Colin Patrick
    McCabe via atm)

    HDFS-4595. When short circuit read is fails, DFSClient does not fallback
    to regular reads. (suresh)

    HDFS-4583. TestNodeCount fails. (Ivan Mitic via suresh)

    HDFS-4591. HA clients can fail to fail over while Standby NN is performing
    long checkpoint. (atm)

    HDFS-3277. fail over to loading a different FSImage if the first one we
    try to load is corrupt. (Colin Patrick McCabe and Andrew Wang via atm)

    HDFS-4596. Shutting down namenode during checkpointing can lead to md5sum
    error. (Andrew Wang via atm)

    HDFS-4614. FSNamesystem#getContentSummary should use getPermissionChecker
    helper method. (atm)

    HDFS-4620. Documentation for dfs.namenode.rpc-address specifies wrong
    format. (Sandy Ryza via atm)

    HDFS-4607. In TestGetConf.testGetSpecificKey(), use a platform-specific
    line separator; otherwise, it fails on Windows.  (Ivan Mitic via szetszwo)

    HDFS-4609. TestAuditLogs should release log handles between tests. 
    (Ivan Mitic via szetszwo)

    HDFS-4598. Fix the default value of ConcatSourcesParam and the WebHDFS doc.
    (szetszwo)

    HDFS-4655. DNA_FINALIZE is logged as being an unknown command by the DN
    when received from the standby NN. (atm)

    HDFS-4656. DN heartbeat loop can be briefly tight. (atm)

    HDFS-4658. Standby NN will log that it has received a block report "after
    becoming active" (atm)

    HDFS-4646. createNNProxyWithClientProtocol ignores configured timeout
    value (Jagane Sundar via cos)

    HDFS-3981. Fix handling of FSN lock in getBlockLocations. (Xiaobo Peng
    and todd via todd)

    HDFS-4676. TestHDFSFileSystemContract should set MiniDFSCluster variable
    to null to free up memory. (suresh)

    HDFS-4669. TestBlockPoolManager fails using IBM java. (Tian Hong Wang via
    suresh)

    HDFS-4643. Fix flakiness in TestQuorumJournalManager. (todd)

    HDFS-4639. startFileInternal() should not increment generation stamp.
    (Plamen Jeliazkov via shv)

    HDFS-4695. TestEditLog leaks open file handles between tests.
    (Ivan Mitic via suresh)

    HDFS-4737. JVM path embedded in fuse binaries. (Sean Mackrory via atm)

    HDFS-4739. NN can miscalculate the number of extra edit log segments to
    retain. (atm)

    HDFS-4745. TestDataTransferKeepalive#testSlowReader has race condition that
    causes sporadic failure. (Chris Nauroth via suresh)

    HDFS-4768. File handle leak in datanode when a block pool is removed.
    (Chris Nauroth via suresh)

    HDFS-4748. MiniJournalCluster#restartJournalNode leaks resources, which 
    causes sporadic test failures. (Chris Nauroth via suresh)

    HDFS-4733. Make HttpFS username pattern configurable. (tucu via atm)

    HDFS-4778. Fixes some issues that the first patch on HDFS-2576 missed.
    (ddas)

    HDFS-4785. Concat operation does not remove concatenated files from
    InodeMap. (suresh)

    HDFS-4784. NPE in FSDirectory.resolvePath(). (Brandon Li via suresh)

Release 2.0.4-alpha - UNRELEASED

  INCOMPATIBLE CHANGES

  NEW FEATURES

  IMPROVEMENTS

  OPTIMIZATIONS

  BUG FIXES

Release 2.0.3-alpha - 2013-02-06

  INCOMPATIBLE CHANGES

    HDFS-4122. Cleanup HDFS logs and reduce the size of logged messages.
    (suresh)

    HDFS-4362. GetDelegationTokenResponseProto does not handle null token.
    (suresh)

    HDFS-4367. GetDataEncryptionKeyResponseProto does not handle null
    response. (suresh)

    HDFS-4364. GetLinkTargetResponseProto does not handle null path. (suresh)

    HDFS-4369. GetBlockKeysResponseProto does not handle null response.
    (suresh)

    HDFS-4451. hdfs balancer command returns exit code 1 on success instead
    of 0. (Joshua Blatt via suresh)

    HDFS-4350. Make enabling of stale marking on read and write paths
    independent. (Andrew Wang via suresh)


  NEW FEATURES

    HDFS-2656. Add libwebhdfs, a pure C client based on WebHDFS.
    (Jaimin D Jetly and Jing Zhao via szetszwo)

    HDFS-3912. Detect and avoid stale datanodes for writes.
    (Jing Zhao via suresh)

    HDFS-4059. Add number of stale DataNodes to metrics. (Jing Zhao via suresh)

    HDFS-4155. libhdfs implementation of hsync API (Liang Xie via todd)

    HDFS-4213. Add an API to hsync for updating the last block length at the
    namenode. (Jing Zhao via szetszwo)

    HDFS-3077. Implement QuorumJournalManager, a distributed mechanism for
    reliably storing HDFS edit logs. See dedicated section below for breakdown
    of subtasks.

  IMPROVEMENTS
  
    HDFS-3925. Prettify PipelineAck#toString() for printing to a log
    (Andrew Wang via todd)

    HDFS-3939. NN RPC address cleanup. (eli)

    HDFS-3373. Change DFSClient input stream socket cache to global static and
    add a thread to cleanup expired cache entries. (John George via szetszwo)

    HDFS-3896. Add descriptions for dfs.namenode.rpc-address and
    dfs.namenode.servicerpc-address to hdfs-default.xml. (Jeff Lord via atm)

    HDFS-3996. Add debug log removed in HDFS-3873 back. (eli)

    HDFS-3916. libwebhdfs (C client) code cleanups.
    (Colin Patrick McCabe via eli)

    HDFS-3813. Log error message if security and WebHDFS are enabled but
    principal/keytab are not configured. (Stephen Chu via atm)

    HDFS-3483. Better error message when hdfs fsck is run against a ViewFS
    config. (Stephen Fritz via atm)

    HDFS-3682. MiniDFSCluster#init should provide more info when it fails.
    (todd via eli)

    HDFS-4008. TestBalancerWithEncryptedTransfer needs a timeout. (eli)

    HDFS-4007. Rehabilitate bit-rotted unit tests under
    hadoop-hdfs-project/hadoop-hdfs/src/test/unit/ 
    (Colin Patrick McCabe via todd)

    HDFS-4041. Hadoop HDFS Maven protoc calls must not depend on external
    sh script. (Chris Nauroth via suresh)

    HADOOP-8911. CRLF characters in source and text files.
    (Raja Aluri via suresh)

    HDFS-4037. Rename the getReplication() method in BlockCollection to
    getBlockReplication(). (szetszwo)

    HDFS-4036. Remove "throws UnresolvedLinkException" from
    FSDirectory.unprotectedAddFile(..). (Jing Zhao via szetszwo)

    HDFS-2946. HA: Put a cap on the number of completed edits files retained
    by the NN. (atm)

    HDFS-4029. GenerationStamp should use an AtomicLong. (eli)

    HDFS-4068. DatanodeID and DatanodeInfo member should be private. (eli)

    HDFS-4073. Two minor improvements to FSDirectory.  (Jing Zhao via szetszwo)

    HDFS-4074. Remove the unused default constructor from INode.  (Brandon Li
    via szetszwo)

    HDFS-4088. Remove "throws QuotaExceededException" from an
    INodeDirectoryWithQuota constructor. (szetszwo)

    HDFS-4099. Clean up replication code and add more javadoc. (szetszwo)

    HDFS-4107. Add utility methods for casting INode to INodeFile and
    INodeFileUnderConstruction. (szetszwo)

    HDFS-4112. A few improvements on INodeDirectory include adding a utility
    method for casting; avoiding creation of new empty lists; cleaning up 
    some code and rewriting some javadoc. (szetszwo)

    HDFS-4121. Add namespace declarations in hdfs .proto files for languages 
    other than java. (Binglin Chang via suresh)

    HDFS-3573. Supply NamespaceInfo when instantiating JournalManagers. 
    (todd and ivank via umamahesh)

    HDFS-3695. Genericize format() to non-file JournalManagers. 
    (todd via umamahesh)

    HDFS-3789. JournalManager#format() should be able to throw IOException. 
    (Ivan Kelly via umamahesh)

    HDFS-3809. Make BKJM use protobufs for all serialization with ZK. 
    (Ivan Kelly via umamhesh)

    HDFS-3916. libwebhdfs testing code cleanup. (Jing Zhao via suresh)

    HDFS-4143. Change blocks to private in INodeFile and renames isLink() to
    isSymlink() in INode. (szetszwo)

    HDFS-4046. Rename ChecksumTypeProto enum NULL since it is illegal in
    C/C++. (Binglin Chang via suresh)

    HDFS-4048. Use ERROR instead of INFO for volume failure logs.
    (Stephen Chu via eli)

    HDFS-1322. Document umask in DistributedFileSystem#mkdirs javadocs.
    (Colin Patrick McCabe via eli)

    HDFS-4038. Override toString() for BookKeeperEditLogInputStream.
    (Vinay via umamahesh)

    HDFS-4214. OfflineEditsViewer should print out the offset at which it
    encountered an error. (Colin Patrick McCabe via atm)

    HDFS-4199. Provide test for HdfsVolumeId. (Ivan A. Veselovsky via atm)

    HDFS-3049. During the normal NN startup process, fall back on a different
    edit log if we see one that is corrupt (Colin Patrick McCabe via todd)

    HDFS-3571. Allow EditLogFileInputStream to read from a remote URL (todd)

    HDFS-4110. Refine a log printed in JNStorage. (Liang Xie via suresh)

    HDFS-4153. Add START_MSG/SHUTDOWN_MSG for JournalNode. (liang xie via atm)

    HDFS-3935. Add JournalNode to the start/stop scripts (Andy Isaacson via todd)

    HDFS-4268. Remove redundant enum NNHAStatusHeartbeat.State. (shv)

    HDFS-3680. Allow customized audit logging in HDFS FSNamesystem. (Marcelo
    Vanzin via atm)

    HDFS-4130. BKJM: The reading for editlog at NN starting using bkjm is not efficient.
    (Han Xiao via umamahesh)

    HDFS-4326. bump up Tomcat version for HttpFS to 6.0.36. (tucu via acmurthy)

    HDFS-4270. Introduce soft and hard limits for max replication so that
    replications of the highest priority are allowed to choose a source datanode
    that has reached its soft limit but not the hard limit.  (Derek Dagit via
    szetszwo)

    HADOOP-9173. Add security token protobuf definition to common and
    use it in hdfs. (suresh)

    HDFS-4030. BlockManager excessBlocksCount and
    postponedMisreplicatedBlocksCount should be AtomicLongs. (eli)

    HDFS-4031. Update findbugsExcludeFile.xml to include findbugs 2
    exclusions. (eli)

    HDFS-4033. Miscellaneous findbugs 2 fixes. (eli)

    HDFS-4034. Remove redundant null checks. (eli)

    HDFS-4035. LightWeightGSet and LightWeightHashSet increment a
    volatile without synchronization. (eli)
    
    HDFS-4032. Specify the charset explicitly rather than rely on the
    default. (eli)

    HDFS-4363. Combine PBHelper and HdfsProtoUtil and remove redundant
    methods. (suresh)

    HDFS-4377. Some trivial DN comment cleanup. (eli)

    HDFS-4381. Document fsimage format details in FSImageFormat class javadoc.
    (Jing Zhao via suresh)

    HDFS-4375. Use token request messages defined in hadoop common.
    (suresh)

    HDFS-4392. Use NetUtils#getFreeSocketPort in MiniDFSCluster.
    (Andrew Purtell via suresh)

    HDFS-4393. Make empty request and responses in protocol translators can be
    static final members. (Brandon Li via suresh)

    HDFS-4403. DFSClient can infer checksum type when not provided by reading
    first byte (todd)

    HDFS-4259. Improve pipeline DN replacement failure message (harsh)

    HDFS-3598. WebHDFS support for file concat. (Plamen Jeliazkov via shv)

    HDFS-4456. Add concat to HttpFS and WebHDFS REST API docs. (plamenj2003 via tucu)

  OPTIMIZATIONS

    HDFS-3429. DataNode reads checksums even if client does not need them (todd)

  BUG FIXES

    HDFS-3919. MiniDFSCluster:waitClusterUp can hang forever.
    (Andy Isaacson via eli)

    HDFS-3924. Multi-byte id in HdfsVolumeId. (Andrew Wang via atm)

    HDFS-3936. MiniDFSCluster shutdown races with BlocksMap usage. (eli)

    HDFS-3951. datanode web ui does not work over HTTPS when datanode is started in secure mode. (tucu)

    HDFS-3949. NameNodeRpcServer#join should join on both client and
    server RPC servers. (eli)

    HDFS-3932. NameNode Web UI broken if the rpc-address is set to the wildcard.
    (Colin Patrick McCabe via eli)

    HDFS-3931. TestDatanodeBlockScanner#testBlockCorruptionPolicy2 is broken.
    (Andy Isaacson via eli)

    HDFS-3964. Make NN log of fs.defaultFS debug rather than info. (eli)

    HDFS-3992. Method org.apache.hadoop.hdfs.TestHftpFileSystem.tearDown()
    sometimes throws NPEs. (Ivan A. Veselovsky via atm)

    HDFS-3753. Tests don't run with native libraries.
    (Colin Patrick McCabe via eli)

    HDFS-4000. TestParallelLocalRead fails with "input ByteBuffers
    must be direct buffers". (Colin Patrick McCabe via eli)

    HDFS-3999. HttpFS OPEN operation expects len parameter, it should be length. (tucu)

    HDFS-4006. TestCheckpoint#testSecondaryHasVeryOutOfDateImage
    occasionally fails due to unexpected exit. (todd via eli)

    HDFS-4018. testMiniDFSClusterWithMultipleNN is missing some
    cluster cleanup. (eli)

    HDFS-4020. TestRBWBlockInvalidation may time out. (eli)

    HDFS-4021. Misleading error message when resources are low on the NameNode.
    (Christopher Conner via atm)

    HDFS-4044. Duplicate ChecksumType definition in HDFS .proto files.
    (Binglin Chang via suresh)

    HDFS-4049. Fix hflush performance regression due to nagling delays
    (todd)

    HDFS-3678. Edit log files are never being purged from 2NN. (atm)

    HDFS-4058. DirectoryScanner may fail with IOOB if the directory
    scanning threads return out of volume order. (eli)

    HDFS-3985. Add timeouts to TestMulitipleNNDataBlockScanner. (todd via eli)

    HDFS-4061. TestBalancer and TestUnderReplicatedBlocks need timeouts. (eli)

    HDFS-3997. OfflineImageViewer incorrectly passes value of imageVersion when
    visiting IS_COMPRESSED element. (Mithun Radhakrishnan via atm)

    HDFS-4055. TestAuditLogs is flaky. (Binglin Chang via eli)

    HDFS-4072. On file deletion remove corresponding blocks pending
    replications. (Jing Zhao via suresh)

    HDFS-4022. Replication not happening for appended block.
    (Vinay via umamahesh)

    HDFS-3948. Do not use hflush in TestWebHDFS.testNamenodeRestart() since the
    out stream returned by WebHdfsFileSystem does not support it. (Jing Zhao
    via szetszwo)

    HDFS-3616. Fix a ConcurrentModificationException bug that BP actor threads
    may not be shutdown properly in DataNode.  (Jing Zhao via szetszwo)

    HDFS-4127. Log message is not correct in case of short of replica.
    (Junping Du via suresh)

    HADOOP-8994. TestDFSShell creates file named "noFileHere", making further
    tests hard to understand (Andy Isaacson via daryn)

    HDFS-3804.  TestHftpFileSystem fails intermittently with JDK7
    (Trevor Robinson via daryn)

    HDFS-4132. When libwebhdfs is not enabled, nativeMiniDfsClient frees
    uninitialized memory (Colin Patrick McCabe via todd)

    HDFS-1331. dfs -test should work like /bin/test (Andy Isaacson via daryn)

    HDFS-3979. For hsync, datanode should wait for the local sync to complete
    before sending ack. (Lars Hofhansl via szetszwo)

    HDFS-3810. Implement format() for BKJM (Ivan Kelly via umamahesh)

    HDFS-3625. Fix TestBackupNode by properly initializing edit log during
    startup. (Junping Du via todd)

    HDFS-4138. BackupNode startup fails due to uninitialized edit log.
    (Kihwal Lee via shv)

    HDFS-4162. Some malformed and unquoted HTML strings are returned from 
    datanode web ui. (Darek Dagit via suresh)

    HDFS-4164. fuse_dfs: add -lrt to the compiler command line on Linux.
    (Colin Patrick McCabe via eli)

    HDFS-3921. NN will prematurely consider blocks missing when entering active
    state while still in safe mode. (atm)

    HDFS-4106. BPServiceActor#lastHeartbeat, lastBlockReport and
    lastDeletedReport should be volatile. (Jing Zhao via suresh)

    HDFS-4139. fuse-dfs RO mode still allows file truncation.
    (Colin Patrick McCabe via eli)    

    HDFS-4104. dfs -test -d prints inappropriate error on nonexistent directory
    (Andy Isaacson via daryn)

    HDFS-3623. BKJM: zkLatchWaitTimeout hard coded to 6000. Make use of ZKSessionTimeout instead.
    (umamahesh)

    HDFS-4100. Fix all findbug security warings. (Liang Xie via eli)

    HDFS-3507. DFS#isInSafeMode needs to execute only on Active NameNode.
    (Vinay via atm)

    HDFS-4105. The SPNEGO user for secondary namenode should use the web
    keytab. (Arpit Gupta via jitendra)

    HDFS-4156. Seeking to a negative position should throw an IOE.
    (Eli Reisman via eli)

    HDFS-4171. WebHDFS and HttpFs should accept only valid Unix user 
    names. (tucu)

    HDFS-4178. Shell scripts should not close stderr (Andy Isaacson via daryn)

    HDFS-4179. BackupNode: allow reads, fix checkpointing, safeMode. (shv)

    HDFS-4216. Do not ignore QuotaExceededException when adding symlinks.
    (szetszwo)

    HDFS-4242. Map.Entry is incorrectly used in LeaseManager since the behavior
    of it is undefined after the iteration or modifications of the map.
    (szetszwo)

    HDFS-4231. BackupNode: Introduce BackupState. (shv)

    HDFS-4238. Standby namenode should not do purging of shared
    storage edits. (todd)

    HDFS-4282. TestEditLog.testFuzzSequences FAILED in all pre-commit test
    (todd)

    HDFS-4236. Remove artificial limit on username length introduced in
    HDFS-4171. (tucu via suresh)

    HDFS-4279. NameNode does not initialize generic conf keys when started
    with -recover. (Colin Patrick McCabe via atm)

    HDFS-4291. edit log unit tests leave stray test_edit_log_file around
    (Colin Patrick McCabe via todd)

    HDFS-4292. Sanity check not correct in RemoteBlockReader2.newBlockReader
    (Binglin Chang via todd)

    HDFS-4295. Using port 1023 should be valid when starting Secure DataNode
    (Stephen Chu via todd)

    HDFS-4294. Backwards compatibility is not maintained for TestVolumeId.
    (Ivan A. Veselovsky and Robert Parker via atm)

    HDFS-2264. NamenodeProtocol has the wrong value for clientPrincipal in
    KerberosInfo annotation. (atm)

    HDFS-4307. SocketCache should use monotonic time. (Colin Patrick McCabe
    via atm)

    HDFS-4315. DNs with multiple BPs can have BPOfferServices fail to start
    due to unsynchronized map access. (atm)

    HDFS-4140. fuse-dfs handles open(O_TRUNC) poorly. (Colin Patrick McCabe
    via atm)

    HDFS-4308. addBlock() should persist file blocks once.
    (Plamen Jeliazkov via shv)

    HDFS-4347. Avoid infinite waiting checkpoint to complete in TestBackupNode.
    (Plamen Jeliazkov via shv)

    HDFS-4349. Add test for reading files from BackupNode. (shv)

    HDFS-4302. Fix fatal exception when starting NameNode with DEBUG logs
    (Eugene Koontz via todd)

    HDFS-3970. Fix bug causing rollback of HDFS upgrade to result in bad
    VERSION file. (Vinay and Andrew Wang via atm)

    HDFS-4306. PBHelper.convertLocatedBlock miss convert BlockToken. (Binglin
    Chang via atm)

    HDFS-4384. test_libhdfs_threaded gets SEGV if JNIEnv cannot be
    initialized. (Colin Patrick McCabe via eli)

    HDFS-4328. TestLargeBlock#testLargeBlockSize is timing out. (Chris Nauroth
    via atm)

    HDFS-4274. BlockPoolSliceScanner does not close verification log during
    shutdown. (Chris Nauroth via suresh)

    HDFS-1245. Pluggable block id generation. (shv)

    HDFS-4415. HostnameFilter should handle hostname resolution failures and
    continue processing. (Robert Kanter via atm)

    HDFS-4359. Slow RPC responses from NN can prevent metrics collection on
    DNs. (liang xie via atm)

    HDFS-4444. Add space between total transaction time and number of
    transactions in FSEditLog#printStatistics. (Stephen Chu via suresh)

    HDFS-4428. FsDatasetImpl should disclose what the error is when a rename
    fails. (Colin Patrick McCabe via atm)

    HDFS-4452. getAdditionalBlock() can create multiple blocks if the client
    times out and retries. (shv)

    HDFS-4445. All BKJM ledgers are not checked while tailing, So failover will fail.
    (Vinay via umamahesh)

    HDFS-4462. 2NN will fail to checkpoint after an HDFS upgrade from a
    pre-federation version of HDFS. (atm)

    HDFS-4404. Create file failure when the machine of first attempted NameNode
    is down. (Todd Lipcon via atm)

    HDFS-4344. dfshealth.jsp throws NumberFormatException when
    dfs.hosts/dfs.hosts.exclude includes port number. (Andy Isaacson via atm)

    HDFS-4468.  Use the new StringUtils methods added by HADOOP-9252 and fix
    TestHDFSCLI and TestQuota. (szetszwo)

    HDFS-4458. In DFSUtil.getNameServiceUris(..), convert default fs URI using
    NetUtils.createSocketAddr(..) for being consistent with other addresses.
    (Binglin Chang via szetszwo)

  BREAKDOWN OF HDFS-3077 SUBTASKS

    HDFS-3077. Quorum-based protocol for reading and writing edit logs.
    (todd, Brandon Li, and Hari Mankude via todd)
    
    HDFS-3694. Fix getEditLogManifest to fetch httpPort if necessary (todd)
    
    HDFS-3692. Support purgeEditLogs() call to remotely purge logs on JNs
    (todd)
    
    HDFS-3693. JNStorage should read its storage info even before a writer
    becomes active (todd)
    
    HDFS-3725. Fix QJM startup when individual JNs have gaps (todd)
    
    HDFS-3741. Exhaustive failure injection test for skipped RPCs (todd)
    
    HDFS-3773. TestNNWithQJM fails after HDFS-3741. (atm)
    
    HDFS-3793. Implement genericized format() in QJM (todd)
    
    HDFS-3795. QJM: validate journal dir at startup (todd)
    
    HDFS-3798. Avoid throwing NPE when finalizeSegment() is called on invalid
    segment (todd)
    
    HDFS-3799. QJM: handle empty log segments during recovery (todd)
    
    HDFS-3797. QJM: add segment txid as a parameter to journal() RPC (todd)
    
    HDFS-3800. improvements to QJM fault testing (todd)
    
    HDFS-3823. QJM: TestQJMWithFaults fails occasionally because of missed
    setting of HTTP port. (todd and atm)
    
    HDFS-3826. QJM: Some trivial logging / exception text improvements. (todd
    and atm)
    
    HDFS-3839. QJM: hadoop-daemon.sh should be updated to accept "journalnode"
    (eli)
    
    HDFS-3845. Fixes for edge cases in QJM recovery protocol (todd)
    
    HDFS-3877. QJM: Provide defaults for dfs.journalnode.*address (eli)
    
    HDFS-3863. Track last "committed" txid in QJM (todd)
    
    HDFS-3869. Expose non-file journal manager details in web UI (todd)
    
    HDFS-3884. Journal format() should reset cached values (todd)
    
    HDFS-3870. Add metrics to JournalNode (todd)
    
    HDFS-3891. Make selectInputStreams throw IOE instead of RTE (todd)
    
    HDFS-3726. If a logger misses an RPC, don't retry that logger until next
    segment (todd)
    
    HDFS-3893. QJM: Make QJM work with security enabled. (atm)
    
    HDFS-3897. QJM: TestBlockToken fails after HDFS-3893. (atm)
    
    HDFS-3898. QJM: enable TCP_NODELAY for IPC (todd)
    
    HDFS-3885. QJM: optimize log sync when JN is lagging behind (todd)
    
    HDFS-3900. QJM: avoid validating log segments on log rolls (todd)
    
    HDFS-3901. QJM: send 'heartbeat' messages to JNs even when they are
    out-of-sync (todd)
    
    HDFS-3899. QJM: Add client-side metrics (todd)
    
    HDFS-3914. QJM: acceptRecovery should abort current segment (todd)
    
    HDFS-3915. QJM: Failover fails with auth error in secure cluster (todd)
    
    HDFS-3906. QJM: quorum timeout on failover with large log segment (todd)
    
    HDFS-3840. JournalNodes log JournalNotFormattedException backtrace error
    before being formatted (todd)
    
    HDFS-3894. QJM: testRecoverAfterDoubleFailures can be flaky due to IPC
    client caching (todd)
    
    HDFS-3926. QJM: Add user documentation for QJM. (atm)
    
    HDFS-3943. QJM: remove currently-unused md5sum field (todd)
    
    HDFS-3950. QJM: misc TODO cleanup, improved log messages, etc. (todd)
    
    HDFS-3955. QJM: Make acceptRecovery() atomic. (todd)
    
    HDFS-3956. QJM: purge temporary files when no longer within retention
    period (todd)
    
    HDFS-4004. TestJournalNode#testJournal fails because of test case execution
    order (Chao Shi via todd)
    
    HDFS-4017. Unclosed FileInputStream in GetJournalEditServlet
    (Chao Shi via todd)

    HDFS-4351. In BlockPlacementPolicyDefault.chooseTarget(..), numOfReplicas
    needs to be updated when avoiding stale nodes.  (Andrew Wang via szetszwo)

    HDFS-2908. Add apache license header for StorageReport.java. (Brandon Li
    via tgraves)

    HDFS-4399. Fix RAT warnings by excluding images sub-dir in docs. (Thomas
    Graves via acmurthy) 

Release 2.0.2-alpha - 2012-09-07 

  INCOMPATIBLE CHANGES

    HDFS-3446. HostsFileReader silently ignores bad includes/excludes
    (Matthew Jacobs via todd)

    HDFS-3755. Creating an already-open-for-write file with overwrite=true fails
    (todd)

  NEW FEATURES

    HDFS-744. Support hsync in HDFS. (Lars Hofhansl via szetszwo)

    HDFS-3042. Automatic failover support for NameNode HA (todd)
    (see dedicated section below for breakdown of subtasks)

    HDFS-3518. Add a utility method HdfsUtils.isHealthy(uri) for checking if
    the given HDFS is healthy. (szetszwo)

    HDFS-3113. httpfs does not support delegation tokens. (tucu)

    HDFS-3513. HttpFS should cache filesystems. (tucu)

    HDFS-3637. Add support for encrypting the DataTransferProtocol. (atm)

    HDFS-3150. Add option for clients to contact DNs via hostname. (eli)

    HDFS-2793. Add an admin command to trigger an edit log roll. (todd)

    HDFS-3703. Datanodes are marked stale if heartbeat is not received in
    configured timeout and are selected as the last location to read from.
    (Jing Zhao via suresh)
    
  IMPROVEMENTS

    HDFS-3040. TestMulitipleNNDataBlockScanner is misspelled. (Madhukara Phatak
    via atm)

    HDFS-3390. DFSAdmin should print full stack traces of errors when DEBUG
    logging is enabled. (atm)

    HDFS-3341. Change minimum RPC versions to respective SNAPSHOTs instead of
    final releases. (todd)

    HDFS-3369. Rename {get|set|add}INode(..) methods in BlockManager and
    BlocksMap to {get|set|add}BlockCollection(..).  (John George via szetszwo)

    HDFS-3134. harden edit log loader against malformed or malicious input.
    (Colin Patrick McCabe via eli)

    HDFS-3230. Cleanup DatanodeID creation in the tests. (eli)

    HDFS-3401. Cleanup DatanodeDescriptor creation in the tests. (eli)

    HDFS-3400. DNs should be able start with jsvc even if security is disabled.
    (atm via eli)

    HDFS-3404. Make putImage in GetImageServlet infer remote address to fetch
    from request. (atm)

    HDFS-3335. check for edit log corruption at the end of the log
    (Colin Patrick McCabe via todd)

    HDFS-3417. Rename BalancerDatanode#getName to getDisplayName to be
    consistent with Datanode. (eli)

    HDFS-3416. Cleanup DatanodeID and DatanodeRegistration
    constructors used by testing. (eli)

    HDFS-3419. Cleanup LocatedBlock. (eli)

    HDFS-3440. More effectively limit stream memory consumption when reading
    corrupt edit logs (Colin Patrick McCabe via todd)

    HDFS-3438. BootstrapStandby should not require a rollEdits on active node
    (todd)

    HDFS-2885. Remove "federation" from the nameservice config options.
    (Tsz Wo (Nicholas) Sze via eli)

    HDFS-3394. Do not use generic in INodeFile.getLastBlock(): the run-time
    ClassCastException check is useless since generic type information is only
    available in compile-time.  (szetszwo)

    HDFS-3454. Balancer unconditionally logs InterruptedException at
    INFO level on shutdown if security is enabled. (eli)

    HDFS-1013. Miscellaneous improvements to HTML markup for web UIs
    (Eugene Koontz via todd)

    HDFS-3052. Change INodeFile and INodeFileUnderConstruction to package
    private.  (szetszwo)

    HDFS-3520. Add transfer rate logging to TransferFsImage. (eli)

    HDFS-3504. Support configurable retry policy in DFSClient for RPC
    connections and RPC calls, and add MultipleLinearRandomRetry, a new retry
    policy.  (szetszwo)

    HDFS-3372. offlineEditsViewer should be able to read a binary
    edits file with recovery mode. (Colin Patrick McCabe via eli)

    HDFS-3516. Check content-type in WebHdfsFileSystem.  (szetszwo)

    HDFS-3535. Audit logging should log denied accesses. (Andy Isaacson via eli)

    HDFS-3481. Refactor HttpFS handling of JAX-RS query string parameters (tucu)

    HDFS-3572. Cleanup code which inits SPNEGO in HttpServer (todd)

    HDFS-3475. Make the replication monitor multipliers configurable.
    (harsh via eli)

    HDFS-3343. Improve metrics for DN read latency (Andrew Wang via todd)

    HDFS-3170. Add more useful metrics for write latency (Matthew Jacobs via
    todd)

    HDFS-3604. Add dfs.webhdfs.enabled to hdfs-default.xml. (eli)

    HDFS-2988. Improve error message when storage directory lock fails
    (Miomir Boljanovic via harsh)

    HDFS-2391. Newly set BalancerBandwidth value is not displayed anywhere.
    (harsh)

    HDFS-3067. NPE in DFSInputStream.readBuffer if read is repeated on
    corrupted block. (Henry Robinson via atm)

    HDFS-3555. idle client socket triggers DN ERROR log
    (should be INFO or DEBUG). (Andy Isaacson via harsh)

    HDFS-3568. fuse_dfs: add support for security. (Colin McCabe via atm)

    HDFS-3629. Fix the typo in the error message about inconsistent
    storage layout version. (Brandon Li via harsh)

    HDFS-3613. GSet prints some INFO level values, which aren't
    really very useful to all (Andrew Wang via harsh)

    HDFS-3611. NameNode prints unnecessary WARNs about edit log normally skipping 
    a few bytes. (Colin Patrick McCabe via harsh)

    HDFS-3582. Hook daemon process exit for testing. (eli)

    HDFS-3641. Move server Util time methods to common and use now
    instead of System#currentTimeMillis. (eli)

    HDFS-3633. libhdfs: hdfsDelete should pass JNI_FALSE or JNI_TRUE.
    (Colin Patrick McCabe via eli)

    HDFS-799. libhdfs must call DetachCurrentThread when a thread is destroyed.
    (Colin Patrick McCabe via eli)

    HDFS-3306. fuse_dfs: don't lock release operations.
    (Colin Patrick McCabe via eli)

    HDFS-3612. Single namenode image directory config warning can
    be improved. (Andy Isaacson via harsh)

    HDFS-3606. libhdfs: create self-contained unit test.
    (Colin Patrick McCabe via eli)

    HDFS-3539. libhdfs code cleanups. (Colin Patrick McCabe via eli)

    HDFS-3610. fuse_dfs: Provide a way to use the default (configured) NN URI.
    (Colin Patrick McCabe via eli)

    HDFS-3663. MiniDFSCluster should capture the code path that led to
    the first ExitException. (eli)

    HDFS-3659. Add missing @Override to methods across the hadoop-hdfs
    project. (Brandon Li via harsh)

    HDFS-3537. Move libhdfs and fuse-dfs source to native subdirectories.
    (Colin Patrick McCabe via eli)

    HDFS-3665. Add a test for renaming across file systems via a symlink. (eli)

    HDFS-3666. Plumb more exception messages to terminate. (eli)

    HDFS-3673. libhdfs: fix some compiler warnings. (Colin Patrick McCabe via eli)

    HDFS-3675. libhdfs: follow documented return codes. (Colin Patrick McCabe via eli)

    HDFS-1249. With fuse-dfs, chown which only has owner (or only group)
    argument fails with Input/output error. (Colin Patrick McCabe via eli)

    HDFS-3583. Convert remaining tests to Junit4. (Andrew Wang via atm)

    HDFS-3711. Manually convert remaining tests to JUnit4. (Andrew Wang via atm)

    HDFS-3650. Use MutableQuantiles to provide latency histograms for various
    operations. (Andrew Wang via atm)

    HDFS-3667.  Add retry support to WebHdfsFileSystem.  (szetszwo)

    HDFS-3291. add test that covers HttpFS working w/ a non-HDFS Hadoop
    filesystem (tucu)

    HDFS-3634. Add self-contained, mavenized fuse_dfs test. (Colin Patrick
    McCabe via atm)

    HDFS-3190. Simple refactors in existing NN code to assist
    QuorumJournalManager extension. (todd)

    HDFS-3276. initializeSharedEdits should have a -nonInteractive flag (todd)

    HDFS-3765. namenode -initializeSharedEdits should be able to initialize
    all shared storages. (Vinay and todd via todd)

    HDFS-3723. Add support -h, -help to all the commands. (Jing Zhao via
    suresh)

    HDFS-3803. Change BlockPoolSliceScanner chatty INFO log to DEBUG.
    (Andrew Purtell via suresh)

    HDFS-3802. StartupOption.name in HdfsServerConstants should be final.
    (Jing Zhao via szetszwo)

    HDFS-3796. Speed up edit log tests by avoiding fsync() (todd)

    HDFS-2963. Console Output is confusing while executing metasave
    (dfsadmin command). (Andrew Wang via eli)

    HDFS-3672. Expose disk-location information for blocks to enable better
    scheduling. (Andrew Wang via atm)

    HDFS-2727. libhdfs should get the default block size from the server.
    (Colin Patrick McCabe via eli)

    HDFS-3832. Remove protocol methods related to DistributedUpgrade. (suresh)

    HDFS-3819. Should check whether invalidate work percentage default value is
    not greater than 1.0f. (Jing Zhao via jitendra)

    HDFS-3177. Update DFSClient and DataXceiver to handle different checkum
    types in file checksum computation.  (Kihwal Lee via szetszwo)

    HDFS-3844. Add @Override and remove {@inheritdoc} and unnecessary
    imports. (Jing Zhao via suresh)

    HDFS-3853. Port MiniDFSCluster enableManagedDfsDirsRedundancy
    option to branch-2. (Colin Patrick McCabe via eli)

    HDFS-3871. Change NameNodeProxies to use RetryUtils.  (Arun C Murthy
    via szetszwo)

    HDFS-3887. Remove redundant chooseTarget methods in BlockPlacementPolicy.
    (Jing Zhao via szetszwo)

    HDFS-3888. Clean up BlockPlacementPolicyDefault.  (Jing Zhao via szetszwo)

    HDFS-3907. Allow multiple users for local block readers. (eli)

    HDFS-3510. Editlog pre-allocation is performed prior to writing edits
    to avoid partial edits case disk out of space. (Colin McCabe via todd)

    HDFS-3910. DFSTestUtil#waitReplication should timeout. (eli)
    
    HDFS-3920. libwebdhfs string processing and using strerror consistently
    to handle all errors. (Jing Zhao via suresh)

  OPTIMIZATIONS

    HDFS-2982. Startup performance suffers when there are many edit log
    segments. (Colin Patrick McCabe via todd)

    HDFS-2834. Add a ByteBuffer-based read API to DFSInputStream.
    (Henry Robinson via todd)

    HDFS-3110. Use directRead API to reduce the number of buffer copies in
    libhdfs (Henry Robinson via todd)

    HDFS-3697. Enable fadvise readahead by default. (todd)

    HDFS-2421. Improve the concurrency of SerialNumberMap in NameNode.
    (Jing Zhao and Weiyan Wang via szetszwo)

    HDFS-3866. HttpFS POM should have property where to download tomcat from (zero45 via tucu)

  BUG FIXES

    HDFS-3385. The last block of INodeFileUnderConstruction is not
    necessarily a BlockInfoUnderConstruction, so do not cast it in
    FSNamesystem.recoverLeaseInternal(..).  (szetszwo)

    HDFS-3414. Balancer does not find NameNode if rpc-address or
    servicerpc-address are not set in client configs. (atm)

    HDFS-3031. Fix complete() and getAdditionalBlock() RPCs to be idempotent
    (todd)

    HDFS-2759. Pre-allocate HDFS edit log files after writing version number.
    (atm)

    HDFS-3413. TestFailureToReadEdits timing out. (atm)

    HDFS-3422. TestStandbyIsHot timeouts too aggressive (todd)

    HDFS-3433. GetImageServlet should allow administrative requestors when
    security is enabled. (atm)

    HDFS-1153. dfsnodelist.jsp should handle invalid input parameters.
    (Ravi Phulari via eli)

    HDFS-3434. InvalidProtocolBufferException when visiting DN
    browseDirectory.jsp (eli)

    HDFS-2800. Fix cancellation of checkpoints in the standby node to be more
    reliable. (todd)

    HDFS-3391. Fix InvalidateBlocks to compare blocks including their
    generation stamps. (todd)

    HDFS-3444. hdfs groups command doesn't work with security enabled. (atm)

    HDFS-2717. BookKeeper Journal output stream doesn't check addComplete rc.
    (Ivan Kelly via umamahesh)

    HDFS-3415. Make sure all layout versions are the same for all storage
    directories in the Namenode.  (Brandon Li via szetszwo)

    HDFS-3436. In DataNode.transferReplicaForPipelineRecovery(..), it should
    use the stored generation stamp to check if the block is valid.  (Vinay
    via szetszwo)

    HDFS-3460. HttpFS proxyuser validation with Kerberos ON uses full 
    principal name. (tucu)

    HDFS-3058. HA: Bring BookKeeperJournalManager up to date with HA changes.
    (Ivan Kelly via umamahesh)

    HDFS-3368. Missing blocks due to bad DataNodes coming up and down. (shv)

    HDFS-3452. BKJM:Switch from standby to active fails and NN gets shut down
    due to delay in clearing of lock. (umamahesh)

    HDFS-3398. Client will not retry when primaryDN is down once it's just got pipeline.
    (Amith D K via umamahesh)

    HDFS-3474. Cleanup Exception handling in BookKeeper journal manager.
    (Ivan Kelly via umamahesh)

    HDFS-3468. Make BKJM-ZK session timeout configurable. (umamahesh)

    HDFS-3423. BKJM: NN startup is failing, when tries to recoverUnfinalizedSegments()
    a bad inProgress_ ZNodes. (Ivan Kelly and Uma via umamahesh)

    HDFS-3441. Race condition between rolling logs at active NN and purging at standby.
    (Rakesh R via umamahesh)

    HDFS-3484. hdfs fsck doesn't work if NN HTTP address is set to
    0.0.0.0 even if NN RPC address is configured. (atm via eli)

    HDFS-3486. offlineimageviewer can't read fsimage files that contain
    persistent delegation tokens. (Colin Patrick McCabe via eli)

    HDFS-3487. offlineimageviewer should give byte offset information
    when it encounters an exception. (Colin Patrick McCabe via eli)

    HDFS-3442. Incorrect count for Missing Replicas in FSCK report. (Andrew
    Wang via atm)

    HDFS-2025. Go Back to File View link is not working in tail.jsp.
    (Ashish and Sravan via umamahesh)

    HDFS-3501. Checkpointing with security enabled will stop working
    after ticket lifetime expires. (atm via eli)

    HDFS-3266. DFSTestUtil#waitCorruptReplicas doesn't sleep between checks.
    (Madhukara Phatak via atm)

    HDFS-3505. DirectoryScanner does not join all threads in shutdown.
    (Colin Patrick McCabe via eli)

    HDFS-3485. DataTransferThrottler will over-throttle when currentTimeMillis
    jumps (Andy Isaacson via todd)

    HDFS-2914. HA: Standby should not enter safemode when resources are low.
    (Vinay via atm)

    HDFS-3235. MiniDFSClusterManager doesn't correctly support -format option.
    (Henry Robinson via atm)

    HDFS-3514. Add missing TestParallelLocalRead. (Henry Robinson via atm)

    HDFS-3243. TestParallelRead timing out on jenkins. (Henry Robinson via todd)

    HDFS-3490. DatanodeWebHdfsMethods throws NullPointerException if
    NamenodeRpcAddressParam is not set.  (szetszwo)

    HDFS-2797. Fix misuses of InputStream#skip in the edit log code.
    (Colin Patrick McCabe via eli)

    HDFS-3517. TestStartup should bind ephemeral ports. (eli)

    HDFS-3522. If a namenode is in safemode, it should throw SafeModeException
    when getBlockLocations has zero locations.  (Brandon Li via szetszwo)

    HDFS-3408. BKJM : Namenode format fails, if there is no BK root. (Rakesh R via umamahesh)

    HDFS-3389. Document the BKJM usage in Namenode HA. (umamahesh and Ivan Kelly via umamahesh)

    HDFS-3531. EditLogFileOutputStream#preallocate should check for
    incomplete writes. (Colin Patrick McCabe via eli)

    HDFS-766. Error message not clear for set space quota out of boundary
    values. (Jon Zuanich via atm)

    HDFS-3480. Multiple SLF4J binding warning. (Vinay via eli)

    HDFS-3524. Update TestFileLengthOnClusterRestart for HDFS-3522.  (Brandon
    Li via szetszwo)

    HDFS-3559. DFSTestUtil: use Builder class to construct DFSTestUtil
    instances. (Colin Patrick McCabe via atm)

    HDFS-3551. WebHDFS CREATE should use client location for HTTP redirection.
    (szetszwo)

    HDFS-3157. Fix a bug in the case that the generation stamps of the stored
    block in a namenode and the reported block from a datanode do not match.
    (Ashish Singhi via szetszwo)

    HDFS-3575. HttpFS does not log Exception Stacktraces (brocknoland via tucu)

    HDFS-3574. Fix small race and do some cleanup in GetImageServlet (todd)

    HDFS-3581. FSPermissionChecker#checkPermission sticky bit check
    missing range check. (eli)

    HDFS-3541. Deadlock between recovery, xceiver and packet responder.
    (Vinay via umamahesh)

    HDFS-3428. Move DelegationTokenRenewer to common (tucu)

    HDFS-3491. HttpFs does not set permissions correctly (tucu)

    HDFS-3580. incompatible types; no instance(s) of type variable(s) V exist 
    so that V conforms to boolean compiling HttpFSServer.java with OpenJDK 
    (adi2 via tucu)

    HDFS-3603. Decouple TestHDFSTrash from TestTrash. (Jason Lowe via eli)

    HDFS-711. hdfsUtime does not handle atime = 0 or mtime = 0 correctly.
    (Colin Patrick McCabe via eli)

    HDFS-3548. NamenodeFsck.copyBlock fails to create a Block Reader.
    (Colin Patrick McCabe via eli)

    HDFS-3615. Two BlockTokenSecretManager findbugs warnings. (atm)

    HDFS-470. libhdfs should handle 0-length reads from FSInputStream
    correctly. (Colin Patrick McCabe via eli)

    HDFS-3492. fix some misuses of InputStream#skip.
    (Colin Patrick McCabe via eli)

    HDFS-3609. libhdfs: don't force the URI to look like hdfs://hostname:port.
    (Colin Patrick McCabe via eli)

    HDFS-2966 TestNameNodeMetrics tests can fail under load. (stevel)

    HDFS-3605. Block mistakenly marked corrupt during edit log catchup
    phase of failover. (todd and Brahma Reddy Battula via todd)

    HDFS-3690. BlockPlacementPolicyDefault incorrectly casts LOG. (eli)

    HDFS-3597. SNN fails to start after DFS upgrade. (Andy Isaacson via todd)

    HDFS-3608. fuse_dfs: detect changes in UID ticket cache. (Colin Patrick
    McCabe via atm)

    HDFS-3709. TestStartup tests still binding to the ephemeral port. (eli)

    HDFS-3720. hdfs.h must get packaged. (Colin Patrick McCabe via atm)

    HDFS-3626. Creating file with invalid path can corrupt edit log (todd)

    HDFS-3679. fuse_dfs notrash option sets usetrash. (Conrad Meyer via suresh)

    HDFS-3732. fuse_dfs: incorrect configuration value checked for connection
    expiry timer period. (Colin Patrick McCabe via atm)

    HDFS-3738. TestDFSClientRetries#testFailuresArePerOperation sets incorrect
    timeout config. (atm)

    HDFS-3756. DelegationTokenFetcher creates 2 HTTP connections, the second 
    one not properly configured. (tucu)

    HDFS-3579. libhdfs: fix exception handling. (Colin Patrick McCabe via atm)

    HDFS-3754. BlockSender doesn't shutdown ReadaheadPool threads. (eli)

    HDFS-3760. primitiveCreate is a write, not a read. (Andy Isaacson via atm)

    HDFS-3710. libhdfs misuses O_RDONLY/WRONLY/RDWR. (Andy Isaacson via atm)

    HDFS-3721. hsync support broke wire compatibility. (todd and atm)

    HDFS-3758. TestFuseDFS test failing. (Colin Patrick McCabe via eli)

    HDFS-2330. In NNStorage and FSImagePreTransactionalStorageInspector,
    IOExceptions of stream closures can mask root exceptions. (Uma Maheswara
    Rao G via szetszwo)

    HDFS-3790. test_fuse_dfs.c doesn't compile on centos 5. (Colin Patrick
    McCabe via atm)

    HDFS-3658. Fix bugs in TestDFSClientRetries and add more tests.  (szetszwo)

    HDFS-3794. WebHDFS OPEN returns the incorrect Content-Length in the HTTP
    header when offset is specified and length is omitted.
    (Ravi Prakash via szetszwo)

    HDFS-3048. Small race in BlockManager#close. (Andy Isaacson via eli)

    HDFS-3194. DataNode block scanner is running too frequently.
    (Andy Isaacson via eli)

    HDFS-3808. fuse_dfs: postpone libhdfs intialization until after fork.
    (Colin Patrick McCabe via atm)

    HDFS-3788. ByteRangeInputStream should not expect HTTP Content-Length header
    when chunked transfer-encoding is used.  (szetszwo)

    HDFS-3816. Invalidate work percentage default value should be 0.32f
    instead of 32. (Jing Zhao via suresh)

    HDFS-3707. TestFSInputChecker: improper use of skip.
    (Colin Patrick McCabe via eli)

    HDFS-3830. test_libhdfs_threaded: use forceNewInstance.
    (Colin Patrick McCabe via eli)

    HDFS-3835. Long-lived 2NN cannot perform a checkpoint if security is
    enabled and the NN restarts with outstanding delegation tokens. (atm)

    HDFS-3715. Fix TestFileCreation#testFileCreationNamenodeRestart.
    (Andrew Whang via eli)

    HDFS-3683. Edit log replay progress indicator shows >100% complete. (Plamen
    Jeliazkov via atm)

    HDFS-3731. Release upgrade must handle blocks being written from 1.0.
    (Colin Patrick McCabe via eli)

    HDFS-3856. TestHDFSServerPorts failure is causing surefire fork failure.
    (eli)

    HDFS-3860. HeartbeatManager#Monitor may wrongly hold the writelock of
    namesystem. (Jing Zhao via atm)

    HDFS-3849. When re-loading the FSImage, we should clear the existing
    genStamp and leases. (Colin Patrick McCabe via atm)

    HDFS-3864. NN does not update internal file mtime for OP_CLOSE when reading
    from the edit log. (atm)

    HDFS-3837. Fix DataNode.recoverBlock findbugs warning. (eli)

    HDFS-3733. Audit logs should include WebHDFS access. (Andy Isaacson via 
    eli)

    HDFS-2686. Remove DistributedUpgrade related code. (suresh)

    HDFS-3833. TestDFSShell fails on windows due to concurrent file 
    read/write. (Brandon Li via suresh)

    HDFS-3466. Get HTTP kerberos principal from the web authentication keytab.
    (omalley)

    HDFS-3879. Fix findbugs warning in TransferFsImage on branch-2. (eli)

    HDFS-3469. start-dfs.sh will start zkfc, but stop-dfs.sh will not stop zkfc similarly.
    (Vinay via umamahesh)

    HDFS-1490. TransferFSImage should timeout (Dmytro Molkov and Vinay via todd)

    HDFS-3828. Block Scanner rescans blocks too frequently.
    (Andy Isaacson via eli)

    HDFS-3895. hadoop-client must include commons-cli (tucu)

    HDFS-2757. Cannot read a local block that's being written to when
    using the local read short circuit. (Jean-Daniel Cryans via eli)

    HDFS-3664. BlockManager race when stopping active services.
    (Colin Patrick McCabe via eli)

    HDFS-3928. MiniDFSCluster should reset the first ExitException on shutdown. (eli)
   
    HDFS-3938. remove current limitations from HttpFS docs. (tucu)

    HDFS-3944. Httpfs resolveAuthority() is not resolving host correctly. (tucu)

    HDFS-3972. Trash emptier fails in secure HA cluster. (todd via eli)

    HDFS-4443. Remove a trailing '`' character from the HTML code generated by
    NamenodeJspHelper.generateNodeData(..).  (Christian Rohling via szetszwo)
 
  BREAKDOWN OF HDFS-3042 SUBTASKS

    HDFS-2185. HDFS portion of ZK-based FailoverController (todd)
    
    HDFS-3200. Scope all ZKFC configurations by nameservice (todd)
    
    HDFS-3223. add zkfc to hadoop-daemon.sh script (todd)
    
    HDFS-3261. TestHASafeMode fails on HDFS-3042 branch (todd)
    
    HDFS-3159. Document NN auto-failover setup and configuration (todd)
    
    HDFS-3412. Fix findbugs warnings in auto-HA branch (todd)
    
    HDFS-3432. TestDFSZKFailoverController tries to fail over too early (todd)

    HDFS-3902. TestDatanodeBlockScanner#testBlockCorruptionPolicy is broken.
    (Andy Isaacson via eli)

Release 2.0.0-alpha - 05-23-2012

  INCOMPATIBLE CHANGES

    HDFS-2676. Remove Avro RPC. (suresh)

    HDFS-2303. Unbundle jsvc. (Roman Shaposhnik and Mingjie Lai via eli)

    HDFS-3137. Bump LAST_UPGRADABLE_LAYOUT_VERSION to -16. (eli)
    
    HDFS-3138. Move DatanodeInfo#ipcPort to DatanodeID. (eli)

    HDFS-3164. Move DatanodeInfo#hostName to DatanodeID. (eli)

  NEW FEATURES

    HDFS-2978. The NameNode should expose name dir statuses via JMX. (atm)

    HDFS-395.  DFS Scalability: Incremental block reports. (Tomasz Nykiel
    via hairong)

    HDFS-2517. Add protobuf service for JounralProtocol. (suresh)

    HDFS-2518. Add protobuf service for NamenodeProtocol. (suresh)

    HDFS-2520. Add protobuf service for InterDatanodeProtocol. (suresh)

    HDFS-2519. Add protobuf service for DatanodeProtocol. (suresh)

    HDFS-2581. Implement protobuf service for JournalProtocol. (suresh)

    HDFS-2618. Implement protobuf service for NamenodeProtocol. (suresh)

    HDFS-2629. Implement protobuf service for InterDatanodeProtocol. (suresh)

    HDFS-2636. Implement protobuf service for ClientDatanodeProtocol. (suresh)

    HDFS-2642. Protobuf translators for DatanodeProtocol. (jitendra)

    HDFS-2647. Used protobuf based RPC for InterDatanodeProtocol, 
    ClientDatanodeProtocol, JournalProtocol, NamenodeProtocol. (suresh)

    HDFS-2661. Enable protobuf RPC for DatanodeProtocol. (jitendra)

    HDFS-2697. Move RefreshAuthPolicy, RefreshUserMappings, GetUserMappings 
    protocol to protocol buffers. (jitendra)

    HDFS-2880. Protobuf changes in DatanodeProtocol to add multiple storages.
    (suresh)

    HDFS-2899. Service protocol changes in DatanodeProtocol to add multiple 
    storages. (suresh)

    HDFS-2430. The number of failed or low-resource volumes the NN can tolerate
    should be configurable. (atm)

    HDFS-1623. High Availability Framework for HDFS NN. Contributed by Todd
    Lipcon, Aaron T. Myers, Eli Collins, Uma Maheswara Rao G, Bikas Saha,
    Suresh Srinivas, Jitendra Nath Pandey, Hari Mankude, Brandon Li, Sanjay
    Radia, Mingjie Lai, and Gregory Chanan

    HDFS-2941. Add an administrative command to download a copy of the fsimage
    from the NN. (atm)

    HDFS-2413. Add an API DistributedFileSystem.isInSafeMode() and change
    DistributedFileSystem to @InterfaceAudience.LimitedPrivate.
    (harsh via szetszwo)

    HDFS-3167. CLI-based driver for MiniDFSCluster. (Henry Robinson via atm)

    HDFS-3148. The client should be able to use multiple local interfaces
    for data transfer. (eli)

    HDFS-3000. Add a public API for setting quotas. (atm)

    HDFS-3102. Add CLI tool to initialize the shared-edits dir. (atm)

    HDFS-3004. Implement Recovery Mode. (Colin Patrick McCabe via eli)

    HDFS-3282. Add HdfsDataInputStream as a public API. (umamahesh)

    HDFS-3298. Add HdfsDataOutputStream as a public API.  (szetszwo)

    HDFS-234. Integration with BookKeeper logging system. (Ivan Kelly 
    via jitendra)

  IMPROVEMENTS

    HDFS-2018. Move all journal stream management code into one place.
    (Ivan Kelly via jitendra)

    HDFS-2223. Untangle depencencies between NN components (todd)

    HDFS-2351. Change Namenode and Datanode to register each of their protocols
    seperately (sanjay)

    HDFS-2337. DFSClient shouldn't keep multiple RPC proxy references (atm)
 
    HDFS-2181. Separate HDFS Client wire protocol data types (sanjay)

    HDFS-2459. Separate datatypes for Journal Protocol. (suresh)

    HDFS-2480. Separate datatypes for NamenodeProtocol. (suresh)

    HDFS-2489. Move Finalize and Register to separate file out of
    DatanodeCommand.java. (suresh)

    HDFS-2488. Separate datatypes for InterDatanodeProtocol. (suresh)

    HDFS-2496. Separate datatypes for DatanodeProtocol. (suresh)

    HDFS-2479. HDFS Client Data Types in Protocol Buffers (sanjay)

    HADOOP-7862. Hdfs changes to work with HADOOP-7862: Move the support for
    multiple protocols to lower layer so that Writable, PB and Avro can all
    use it. (sanjay)

    HDFS-2597. ClientNameNodeProtocol in Protocol Buffers. (sanjay)

    HDFS-2651. ClientNameNodeProtocol Translators for Protocol Buffers. (sanjay)

    HDFS-2650. Replace @inheritDoc with @Override. (Hari Mankude via suresh).

    HDFS-2669. Enable protobuf rpc for ClientNamenodeProtocol. (sanjay)

    HDFS-2801. Provide a method in client side translators to check for a 
    methods supported in underlying protocol. (jitendra)

    HDFS-2895. Remove Writable wire protocol types and translators to
    complete transition to protocol buffers. (suresh)

    HDFS-2992. Edit log failure trace should include transaction ID of
    error.  (Colin Patrick McCabe via eli)

    HDFS-2507. Allow saveNamespace operations to be canceled. (todd)

    HDFS-2410. Further cleanup of hardcoded configuration keys and values.
    (suresh)

    HDFS-208. name node should warn if only one dir is listed in dfs.name.dir.
    (Uma Maheswara Rao G via eli)

    HDFS-3003. Remove getHostPortString() from NameNode, replace it with
    NetUtils.getHostPortString(). (Brandon Li via atm)

    HDFS-3014. FSEditLogOp and its subclasses should have toString() method.
    (Sho Shimauchi via atm)

    HDFS-3036. Remove unused method DFSUtil#isDefaultNamenodeAddress. (atm)

    HDFS-2158. Add JournalSet to manage the set of journals. (jitendra)

    HDFS-2334. Add Closeable to JournalManager. (Ivan Kelly via jitendra)

    HDFS-1580. Add interface for generic Write Ahead Logging mechanisms.
    (Ivan Kelly via jitendra)

    HDFS-3060. Bump TestDistributedUpgrade#testDistributedUpgrade timeout (eli)

    HDFS-3021. Use generic type to declare FSDatasetInterface.  (szetszwo)

    HDFS-3056.  Add a new interface RollingLogs for DataBlockScanner logging.
    (szetszwo)

    HDFS-2731. Add command to bootstrap the Standby Node's name directories
    from the Active NameNode. (todd)

    HDFS-3082. Clean up FSDatasetInterface and change DataNode.data to package
    private.  (szetszwo)

    HDFS-3057. httpfs and hdfs launcher scripts should honor CATALINA_HOME 
    and HADOOP_LIBEXEC_DIR (rvs via tucu)

    HDFS-3088. Move FSDatasetInterface inner classes to a package.  (szetszwo)

    HDFS-3105. Add DatanodeStorage information to block recovery.  (szetszwo)

    HDFS-3086. Change Datanode not to send storage list in registration.
    (szetszwo)

    HDFS-3044. fsck move should be non-destructive by default.
    (Colin Patrick McCabe via eli)

    HDFS-3071. haadmin failover command does not provide enough detail when
    target NN is not ready to be active. (todd)

    HDFS-3089. Move FSDatasetInterface and the related classes to a package.
    (szetszwo)

    HDFS-3129. NetworkTopology: add test that getLeaf should check for
    invalid topologies (Colin Patrick McCabe via eli)

    HDFS-3155. Clean up FSDataset implemenation related code.  (szetszwo)

    HDFS-3158. LiveNodes member of NameNodeMXBean should list non-DFS used
    space and capacity per DN. (atm)

    HDFS-3172. dfs.upgrade.permission is dead code. (eli)

    HDFS-3171. The DatanodeID "name" field is overloaded. (eli)

    HDFS-3144. Refactor DatanodeID#getName by use. (eli)

    HDFS-3130. Move fsdataset implementation to a package.  (szetszwo)

    HDFS-3120. Enable hsync and hflush by default. (eli)

    HDFS-3187. Upgrade guava to 11.0.2 (todd)

    HDFS-3168. Remove unnecessary "throw IOException" and change fields to
    final in FSNamesystem and BlockManager.  (szetszwo)

    HDFS-2564. Cleanup unnecessary exceptions thrown and unnecessary casts.
    (Hari Mankude via eli)

    HDFS-3084. FenceMethod.tryFence() and ShellCommandFencer should pass
    namenodeId as well as host:port (todd)

    HDFS-3050. rework OEV to share more code with the NameNode.
    (Colin Patrick McCabe via eli)

    HDFS-3226. Allow GetConf tool to print arbitrary keys (todd)

    HDFS-3204. Minor modification to JournalProtocol.proto to make
    it generic. (suresh)

    HDFS-2505. Add a test to verify getFileChecksum(..) with ViewFS.  (Ravi
    Prakash via szetszwo)

    HDFS-3240. Drop log level of "heartbeat: ..." in BPServiceActor to DEBUG
    (todd)

    HDFS-3238. ServerCommand and friends don't need to be writables. (eli)

    HDFS-3094. add -nonInteractive and -force option to namenode -format
    command (Arpit Gupta via todd)

    HDFS-3244. Remove dead writable code from hdfs/protocol. (eli)

    HDFS-3247. Improve bootstrapStandby behavior when original NN is not active
    (todd)

    HDFS-3249. Use ToolRunner.confirmPrompt in NameNode (todd)

    HDFS-3179.  Improve the exception message thrown by DataStreamer when 
    it failed to add a datanode.  (szetszwo)

    HDFS-2983. Relax the build version check to permit rolling upgrades within
    a release. (atm)

    HDFS-3259. NameNode#initializeSharedEdits should populate shared edits dir
    with edit log segments. (atm)

    HDFS-2708. Stats for the # of blocks per DN. (atm)

    HDFS-3279. Move the FSEditLog constructor with @VisibleForTesting to
    TestEditLog.  (Arpit Gupta via szetszwo)

    HDFS-3294. Fix code indentation in NamenodeWebHdfsMethods and
    DatanodeWebHdfsMethods.  (szetszwo)

    HDFS-3263. HttpFS should read HDFS config from Hadoop site.xml files (tucu)

    HDFS-3206. Miscellaneous xml cleanups for OEV.
    (Colin Patrick McCabe via eli)

    HDFS-3169. TestFsck should test multiple -move operations in a row.
    (Colin Patrick McCabe via eli)

    HDFS-3258. Test for HADOOP-8144 (pseudoSortByDistance in
    NetworkTopology for first rack local node). (Junping Du via eli)

    HDFS-3322. Use HdfsDataInputStream and HdfsDataOutputStream in Hdfs.
    (szetszwo)

    HDFS-3339. Change INode to package private.  (John George via szetszwo)

    HDFS-3303. Remove Writable implementation from RemoteEditLogManifest.
    (Brandon Li via szetszwo)

    HDFS-2617. Replaced Kerberized SSL for image transfer and fsck
    with SPNEGO-based solution. (jghoman, omalley, tucu, and atm via eli)

    HDFS-3365. Enable users to disable socket caching in DFS client
    configuration (todd)

    HDFS-3375. Put client name in DataXceiver thread name for readBlock
    and keepalive (todd)

    HDFS-3363. Define BlockCollection and MutableBlockCollection interfaces
    so that INodeFile and INodeFileUnderConstruction do not have to be used in
    block management.  (John George via szetszwo)

    HDFS-3211. Add fence(..) and replace NamenodeRegistration with JournalInfo
    and epoch in JournalProtocol. (suresh via szetszwo)

    HDFS-3418. Rename BlockWithLocationsProto datanodeIDs field to storageIDs.
    (eli)

  OPTIMIZATIONS

    HDFS-3024. Improve performance of stringification in addStoredBlock (todd)

    HDFS-2477. Optimize computing the diff between a block report and the
    namenode state. (Tomasz Nykiel via hairong)

    HDFS-2495. Increase granularity of write operations in ReplicationMonitor
    thus reducing contention for write lock. (Tomasz Nykiel via hairong)

    HDFS-2476. More CPU efficient data structure for under-replicated,
    over-replicated, and invalidated blocks. (Tomasz Nykiel via todd)

    HDFS-3378. Remove DFS_NAMENODE_SECONDARY_HTTPS_PORT_KEY and DEFAULT. (eli)

  BUG FIXES

    HDFS-2481. Unknown protocol: org.apache.hadoop.hdfs.protocol.ClientProtocol.
    (sanjay)

    HDFS-2497. Fix TestBackupNode failure. (suresh)

    HDFS-2499. RPC client is created incorrectly introduced in HDFS-2459.
    (suresh)

    HDFS-2526. (Client)NamenodeProtocolTranslatorR23 do not need to keep a
    reference to rpcProxyWithoutRetry (atm)

    HDFS-2532. TestDfsOverAvroRpc timing out in trunk (Uma Maheswara Rao G
    via todd)

    HDFS-2666. Fix TestBackupNode failure. (suresh)

    HDFS-2663. Optional protobuf parameters are not handled correctly. (suresh)

    HDFS-2694. Removal of Avro broke non-PB NN services. (atm)

    HDFS-2687. Tests failing with ClassCastException post protobuf RPC
    changes. (suresh)

    HDFS-2700. Fix failing TestDataNodeMultipleRegistrations in trunk
    (Uma Maheswara Rao G via todd)

    HDFS-2739. SecondaryNameNode doesn't start up. (jitendra)

    HDFS-2768. BackupNode stop can not close proxy connections because
    it is not a proxy instance. (Uma Maheswara Rao G via eli)

    HDFS-2968. Protocol translator for BlockRecoveryCommand broken when
    multiple blocks need recovery. (todd)

    HDFS-3020. Fix editlog to automatically sync when buffer is full. (todd)

    HDFS-3038. Add FSEditLog.metrics to findbugs exclude list. (todd via atm)

    HDFS-2285. BackupNode should reject requests to modify namespace.
    (shv and Uma Maheswara Rao)

    HDFS-2764. TestBackupNode is racy. (atm)

    HDFS-2188. Make FSEditLog create its journals from a list of URIs rather 
    than NNStorage. (Ivan Kelly via jitendra)

    HDFS-1765. Block Replication should respect under-replication
    block priority. (Uma Maheswara Rao G via eli)

    HDFS-3093. Fix bug where namenode -format interpreted the -force flag in
    reverse. (todd)

    HDFS-3005. FSVolume.decDfsUsed(..) should be synchronized.  (szetszwo)

    HDFS-3099. SecondaryNameNode does not properly initialize metrics system.
    (atm)

    HDFS-3062. Fix bug which prevented MR job submission from creating
    delegation tokens on an HA cluster. (Mingjie Lai via todd)

    HDFS-3083. Cannot run an MR job with HA and security enabled when
    second-listed NN active. (atm)

    HDFS-3100. In BlockSender, throw an exception when it needs to verify
    checksum but the meta data does not exist.  (Brandon Li via szetszwo)

    HDFS-3132. Fix findbugs warning on HDFS trunk. (todd)

    HDFS-3156. TestDFSHAAdmin is failing post HADOOP-8202. (atm)

    HDFS-3143. TestGetBlocks.testGetBlocks is failing. (Arpit Gupta via atm)

    HDFS-3142. TestHDFSCLI.testAll is failing. (Brandon Li via atm)

    HDFS-3070. HDFS balancer doesn't ensure that hdfs-site.xml is loaded. (atm)

    HDFS-2995. start-dfs.sh should only start the 2NN for namenodes
    with dfs.namenode.secondary.http-address configured. (eli)

    HDFS-3174. Fix assert in TestPendingDataNodeMessages. (eli)

    HDFS-3199. TestValidateConfigurationSettings is failing. (todd via eli)

    HDFS-3202. NamespaceInfo PB translation drops build version. (atm)

    HDFS-3109. Remove hsqldf exclusions from pom.xml. (Ravi Prakash
    via suresh)

    HDFS-3210. JsonUtil#toJsonMap for for a DatanodeInfo should use
    "ipAddr" instead of "name". (eli)

    HDFS-3208. Bogus entries in hosts files are incorrectly displayed
    in the report. (eli)

    HDFS-3136. Remove SLF4J dependency as HDFS does not need it to fix
    unnecessary warnings. (Jason Lowe via suresh)

    HDFS-3214. InterDatanodeProtocolServerSideTranslatorPB doesn't handle
    null response from initReplicaRecovery (todd)

    HDFS-3234. Accidentally left log message in GetConf after HDFS-3226 (todd)

    HDFS-3236. NameNode does not initialize generic conf keys when started
    with -initializeSharedEditsDir (atm)

    HDFS-3248. bootstrapStandby repeated twice in hdfs namenode usage message
    (Colin Patrick McCabe via todd)

    HDFS-2696. Fix the fuse-fds build. (Bruno Mahé via eli)

    HDFS-3260. TestDatanodeRegistration should set minimum DN version in
    addition to minimum NN version. (atm)

    HDFS-3255. HA DFS returns wrong token service (Daryn Sharp via todd)

    HDFS-3256. HDFS considers blocks under-replicated if topology script is
    configured with only 1 rack. (atm)

    HDFS-2799. Trim fs.checkpoint.dir values. (Amith D K via eli)

    HDFS-2765. TestNameEditsConfigs is incorrectly swallowing IOE. (atm)

    HDFS-3268. FileContext API mishandles token service and incompatible with
    HA (Daryn Sharp via todd)

    HDFS-3284. bootstrapStandby fails in secure cluster (todd)

    HDFS-3165. HDFS Balancer scripts are refering to wrong path of
    hadoop-daemon.sh (Amith D K via eli)

    HDFS-891. DataNode no longer needs to check for dfs.network.script.
    (harsh via eli)

    HDFS-3305. GetImageServlet should consider SBN a valid requestor in a
    secure HA setup. (atm)

    HDFS-3314. HttpFS operation for getHomeDirectory is incorrect. (tucu)

    HDFS-3319. Change DFSOutputStream to not to start a thread in constructors.
    (szetszwo)

    HDFS-3181. Fix a test case in TestLeaseRecovery2.  (szetszwo)

    HDFS-3309. HttpFS (Hoop) chmod not supporting octal and sticky bit 
    permissions. (tucu)

    HDFS-3326. Append enabled log message uses the wrong variable.
    (Matthew Jacobs via eli)

    HDFS-3336. hdfs launcher script will be better off not special casing 
    namenode command with regards to hadoop.security.logger (rvs via tucu)

    HDFS-3330. If GetImageServlet throws an Error or RTE, response should not
    have HTTP "OK" status. (todd)

    HDFS-3351. NameNode#initializeGenericKeys should always set fs.defaultFS
    regardless of whether HA or Federation is enabled. (atm)

    HDFS-3359. DFSClient.close should close cached sockets. (todd)

    HDFS-3350. In INode, add final to compareTo(..), equals(..) and hashCode(),
    and remove synchronized from updatePermissionStatus(..).  (szetszwo)

    HDFS-3357. DataXceiver reads from client socket with incorrect/no timeout
    (todd)

    HDFS-3376. DFSClient fails to make connection to DN if there are many
    unusable cached sockets (todd)

    HDFS-3328. NPE in DataNode.getIpcPort. (eli)

    HDFS-3396. FUSE build fails on Ubuntu 12.04. (Colin Patrick McCabe via eli)

    HDFS-3395. NN doesn't start with HA+security enabled and HTTP address
    set to 0.0.0.0. (atm)

    HDFS-3026. HA: Handle failure during HA state transition. (atm)

    HDFS-860. fuse-dfs truncate behavior causes issues with scp.
    (Brian Bockelman via eli)

  BREAKDOWN OF HDFS-1623 SUBTASKS

    HDFS-2179. Add fencing framework and mechanisms for NameNode HA. (todd)
    
    HDFS-1974. Introduce active and standy states to the namenode. (suresh)
    
    HDFS-2407. getServerDefaults and getStats don't check operation category (atm)
    
    HDFS-1973. HA: HDFS clients must handle namenode failover and switch over to
    the new active namenode. (atm)
    
    HDFS-2301. Start/stop appropriate namenode services when transition to active
    and standby states. (suresh)
    
    HDFS-2231. Configuration changes for HA namenode. (suresh)
    
    HDFS-2418. Change ConfiguredFailoverProxyProvider to take advantage of
    HDFS-2231. (atm)
    
    HDFS-2393. Mark appropriate methods of ClientProtocol with the idempotent
    annotation. (atm)
    
    HDFS-2523. Small NN fixes to include HAServiceProtocol and prevent NPE on
    shutdown. (todd)
    
    HDFS-2577. NN fails to start since it tries to start secret manager in
    safemode. (todd)
    
    HDFS-2582. Scope dfs.ha.namenodes config by nameservice (todd)
    
    HDFS-2591. MiniDFSCluster support to mix and match federation with HA (todd)
    
    HDFS-1975. Support for sharing the namenode state from active to standby.
    (jitendra, atm, todd)
    
    HDFS-1971. Send block report from datanode to both active and standby
    namenodes. (sanjay, todd via suresh)
    
    HDFS-2616. Change DatanodeProtocol#sendHeartbeat() to return HeartbeatResponse.
    (suresh)
    
    HDFS-2622. Fix TestDFSUpgrade in HA branch. (todd)
    
    HDFS-2612. Handle refreshNameNodes in federated HA clusters (todd)
    
    HDFS-2623. Add test case for hot standby capability (todd)
    
    HDFS-2626. BPOfferService.verifyAndSetNamespaceInfo needs to be synchronized
    (todd)
    
    HDFS-2624. ConfiguredFailoverProxyProvider doesn't correctly stop
    ProtocolTranslators (todd)
    
    HDFS-2625. TestDfsOverAvroRpc failing after introduction of HeartbeatResponse
    type (todd)
    
    HDFS-2627. Determine DN's view of which NN is active based on heartbeat
    responses (todd)
    
    HDFS-2634. Standby needs to ingest latest edit logs before transitioning to
    active (todd)
    
    HDFS-2671. NN should throw StandbyException in response to RPCs in STANDBY
    state (todd)
    
    HDFS-2680. DFSClient should construct failover proxy with exponential backoff
    (todd)
    
    HDFS-2683. Authority-based lookup of proxy provider fails if path becomes
    canonicalized (todd)
    
    HDFS-2689. HA: BookKeeperEditLogInputStream doesn't implement isInProgress()
    (atm)
    
    HDFS-2602. NN should log newly-allocated blocks without losing BlockInfo (atm)
    
    HDFS-2667. Fix transition from active to standby (todd)
    
    HDFS-2684. Fix up some failing unit tests on HA branch (todd)
    
    HDFS-2679. Add interface to query current state to HAServiceProtocol (eli via
    todd)
    
    HDFS-2677. Web UI should indicate the NN state. (eli via todd)
    
    HDFS-2678. When a FailoverProxyProvider is used, DFSClient should not retry
    connection ten times before failing over (atm via todd)
    
    HDFS-2682. When a FailoverProxyProvider is used, Client should not retry for 45
    times if it is timing out to connect to server. (Uma Maheswara Rao G via todd)
    
    HDFS-2693. Fix synchronization issues around state transition (todd)
    
    HDFS-1972. Fencing mechanism for block invalidations and replications (todd)
    
    HDFS-2714. Fix test cases which use standalone FSNamesystems (todd)
    
    HDFS-2692. Fix bugs related to failover from/into safe mode. (todd)
    
    HDFS-2716. Configuration needs to allow different dfs.http.addresses for each
    HA NN (todd)
    
    HDFS-2720. Fix MiniDFSCluster HA support to work properly on Windows. (Uma
    Maheswara Rao G via todd)
    
    HDFS-2291. Allow the StandbyNode to make checkpoints in an HA setup. (todd)
    
    HDFS-2709. Appropriately handle error conditions in EditLogTailer (atm via
    todd)
    
    HDFS-2730. Refactor shared HA-related test code into HATestUtil class (todd)
    
    HDFS-2762. Fix TestCheckpoint timing out on HA branch. (Uma Maheswara Rao G via
    todd)
    
    HDFS-2724. NN web UI can throw NPE after startup, before standby state is
    entered. (todd)
    
    HDFS-2753. Fix standby getting stuck in safemode when blocks are written while
    SBN is down. (Hari Mankude and todd via todd)
    
    HDFS-2773. Reading edit logs from an earlier version should not leave blocks in
    under-construction state. (todd)
    
    HDFS-2775. Fix TestStandbyCheckpoints.testBothNodesInStandbyState failing
    intermittently. (todd)
    
    HDFS-2766. Test for case where standby partially reads log and then performs
    checkpoint. (atm)
    
    HDFS-2738. FSEditLog.selectinputStreams is reading through in-progress streams
    even when non-in-progress are requested. (atm)
    
    HDFS-2789. TestHAAdmin.testFailover is failing (eli)
    
    HDFS-2747. Entering safe mode after starting SBN can NPE. (Uma Maheswara Rao G
    via todd)
    
    HDFS-2772. On transition to active, standby should not swallow ELIE. (atm)
    
    HDFS-2767. ConfiguredFailoverProxyProvider should support NameNodeProtocol.
    (Uma Maheswara Rao G via todd)
    
    HDFS-2795. Standby NN takes a long time to recover from a dead DN starting up.
    (todd)
    
    HDFS-2592. Balancer support for HA namenodes. (Uma Maheswara Rao G via todd)
    
    HDFS-2367. Enable the configuration of multiple HA cluster addresses. (atm)
    
    HDFS-2812. When becoming active, the NN should treat all leases as freshly
    renewed. (todd)
    
    HDFS-2737. Automatically trigger log rolls periodically on the active NN. (todd
    and atm)
    
    HDFS-2820. Add a simple sanity check for HA config (todd)
    
    HDFS-2688. Add tests for quota tracking in an HA cluster. (todd)
    
    HDFS-2804. Should not mark blocks under-replicated when exiting safemode (todd)
    
    HDFS-2807. Service level authorizartion for HAServiceProtocol. (jitendra)
    
    HDFS-2809. Add test to verify that delegation tokens are honored after
    failover. (jitendra and atm)
    
    HDFS-2838. NPE in FSNamesystem when in safe mode. (Gregory Chanan via eli)
    
    HDFS-2805. Add a test for a federated cluster with HA NNs. (Brandon Li via
    jitendra)
    
    HDFS-2841. HAAdmin does not work if security is enabled. (atm)
    
    HDFS-2691. Fixes for pipeline recovery in an HA cluster: report RBW replicas
    immediately upon pipeline creation. (todd)
    
    HDFS-2824. Fix failover when prior NN died just after creating an edit log
    segment. (atm via todd)
    
    HDFS-2853. HA: NN fails to start if the shared edits dir is marked required
    (atm via eli)
    
    HDFS-2845. SBN should not allow browsing of the file system via web UI. (Bikas
    Saha via atm)
    
    HDFS-2742. HA: observed dataloss in replication stress test. (todd via eli)
    
    HDFS-2870. Fix log level for block debug info in processMisReplicatedBlocks
    (todd)
    
    HDFS-2859. LOCAL_ADDRESS_MATCHER.match has NPE when called from
    DFSUtil.getSuffixIDs when the host is incorrect (Bikas Saha via todd)
    
    HDFS-2861. checkpointing should verify that the dfs.http.address has been
    configured to a non-loopback for peer NN (todd)
    
    HDFS-2860. TestDFSRollback#testRollback is failing. (atm)
    
    HDFS-2769. HA: When HA is enabled with a shared edits dir, that dir should be
    marked required. (atm via eli)
    
    HDFS-2863. Failures observed if dfs.edits.dir and shared.edits.dir have same
    directories. (Bikas Saha via atm)
    
    HDFS-2874. Edit log should log to shared dirs before local dirs. (todd)
    
    HDFS-2890. DFSUtil#getSuffixIDs should skip unset configurations. (atm)
    
    HDFS-2792. Make fsck work. (atm)
    
    HDFS-2808. HA: haadmin should use namenode ids. (eli)
    
    HDFS-2819. Document new HA-related configs in hdfs-default.xml. (eli)
    
    HDFS-2752. HA: exit if multiple shared dirs are configured. (eli)
    
    HDFS-2894. HA: automatically determine the nameservice Id if only one
    nameservice is configured. (eli)
    
    HDFS-2733. Document HA configuration and CLI. (atm)
    
    HDFS-2794. Active NN may purge edit log files before standby NN has a chance to
    read them (todd)
    
    HDFS-2901. Improvements for SBN web UI - not show under-replicated/missing
    blocks. (Brandon Li via jitendra)
    
    HDFS-2905. HA: Standby NN NPE when shared edits dir is deleted. (Bikas Saha via
    jitendra)
    
    HDFS-2579. Starting delegation token manager during safemode fails. (todd)
    
    HDFS-2510. Add HA-related metrics. (atm)
    
    HDFS-2924. Standby checkpointing fails to authenticate in secure cluster.
    (todd)
    
    HDFS-2915. HA: TestFailureOfSharedDir.testFailureOfSharedDir() has race
    condition. (Bikas Saha via jitendra)
    
    HDFS-2912. Namenode not shutting down when shared edits dir is inaccessible.
    (Bikas Saha via atm)
    
    HDFS-2917. HA: haadmin should not work if run by regular user (eli)
    
    HDFS-2939. TestHAStateTransitions fails on Windows. (Uma Maheswara Rao G via
    atm)
    
    HDFS-2947. On startup NN throws an NPE in the metrics system. (atm)
    
    HDFS-2942. TestActiveStandbyElectorRealZK fails if build dir does not exist.
    (atm)
    
    HDFS-2948. NN throws NPE during shutdown if it fails to startup (todd)
    
    HDFS-2909. HA: Inaccessible shared edits dir not getting removed from FSImage
    storage dirs upon error. (Bikas Saha via jitendra)
    
    HDFS-2934. Allow configs to be scoped to all NNs in the nameservice. (todd)
    
    HDFS-2935. Shared edits dir property should be suffixed with nameservice and
    namenodeID (todd)
    
    HDFS-2928. ConfiguredFailoverProxyProvider should not create a NameNode proxy
    with an underlying retry proxy. (Uma Maheswara Rao G via atm)
    
    HDFS-2955. IllegalStateException during standby startup in getCurSegmentTxId.
    (Hari Mankude via atm)
    
    HDFS-2937. TestDFSHAAdmin needs tests with MiniDFSCluster. (Brandon Li via
    suresh)
    
    HDFS-2586. Add protobuf service and implementation for HAServiceProtocol.
    (suresh via atm)
    
    HDFS-2952. NN should not start with upgrade option or with a pending an
    unfinalized upgrade. (atm)
    
    HDFS-2974. MiniDFSCluster does not delete standby NN name dirs during format.
    (atm)
    
    HDFS-2929. Stress test and fixes for block synchronization (todd)
    
    HDFS-2972. Small optimization building incremental block report (todd)
    
    HDFS-2973. Re-enable NO_ACK optimization for block deletion. (todd)
    
    HDFS-2922. HA: close out operation categories (eli)
    
    HDFS-2993. HA: BackupNode#checkOperation should permit CHECKPOINT operations
    (eli)
    
    HDFS-2904. Client support for getting delegation tokens. (todd)
    
    HDFS-3013. HA: NameNode format doesn't pick up
    dfs.namenode.name.dir.NameServiceId configuration (Mingjie Lai via todd)
    
    HDFS-3019. Fix silent failure of TestEditLogJournalFailures (todd)
    
    HDFS-2958. Sweep for remaining proxy construction which doesn't go through
    failover path. (atm)
    
    HDFS-2920. fix remaining TODO items. (atm and todd)
    
    HDFS-3027. Implement a simple NN health check. (atm)
    
    HDFS-3023. Optimize entries in edits log for persistBlocks call. (todd)
    
    HDFS-2979. Balancer should use logical uri for creating failover proxy with HA
    enabled. (atm)
    
    HDFS-3035. Fix failure of TestFileAppendRestart due to OP_UPDATE_BLOCKS (todd)
    
    HDFS-3039. Address findbugs and javadoc warnings on branch. (todd via atm)

Release 0.23.8 - UNRELEASED

  INCOMPATIBLE CHANGES

  NEW FEATURES

  IMPROVEMENTS

  OPTIMIZATIONS

  BUG FIXES

    HDFS-4477. Secondary namenode may retain old tokens (daryn via kihwal)

    HDFS-4699. TestPipelinesFailover#testPipelineRecoveryStress fails
    sporadically (Chris Nauroth via kihwal)

Release 0.23.7 - UNRELEASED

  INCOMPATIBLE CHANGES

  NEW FEATURES

  IMPROVEMENTS

  OPTIMIZATIONS

    HDFS-4532. RPC call queue may fill due to current user lookup (daryn)

  BUG FIXES

    HDFS-4288. NN accepts incremental BR as IBR in safemode (daryn via kihwal)

    HDFS-4495. Allow client-side lease renewal to be retried beyond soft-limit
    (kihwal)

    HDFS-4128. 2NN gets stuck in inconsistent state if edit log replay fails
    in the middle (kihwal via daryn)

    HDFS-4542. Webhdfs doesn't support secure proxy users (Daryn Sharp via
    kihwal)

    HDFS-4560. Webhdfs cannot use tokens obtained by another user (daryn)

    HDFS-4566. Webdhfs token cancelation should use authentication (daryn)

    HDFS-4567. Webhdfs does not need a token for token operations (daryn via
    kihwal)

    HDFS-4577. Webhdfs operations should declare if authentication is required
    (daryn via kihwal)

    HDFS-3344. Unreliable corrupt blocks counting in TestProcessCorruptBlocks
    (kihwal)

    HDFS-3367. WebHDFS doesn't use the logged in user when opening
    connections (daryn)

    HDFS-4581. checkDiskError should not be called on network errors (Rohit
    Kochar via kihwal)

    HDFS-4649. Webhdfs cannot list large directories (daryn via kihwal)

    HDFS-4548. Webhdfs doesn't renegotiate SPNEGO token (daryn via kihwal)

Release 0.23.6 - UNRELEASED

  INCOMPATIBLE CHANGES

  NEW FEATURES

  IMPROVEMENTS

  OPTIMIZATIONS

  BUG FIXES

    HDFS-4247. saveNamespace should be tolerant of dangling lease (daryn)

    HDFS-4248. Renaming directories may incorrectly remove the paths in leases
    under the tree.  (daryn via szetszwo)

    HDFS-4385. Maven RAT plugin is not checking all source files (tgraves)

    HDFS-4426. Secondary namenode shuts down immediately after startup.
    (Arpit Agarwal via suresh)

Release 0.23.5 - 2012-11-28

  INCOMPATIBLE CHANGES

    HDFS-4080. Add a separate logger for block state change logs to enable turning
    off those logs. (Kihwal Lee via suresh)

  NEW FEATURES

  IMPROVEMENTS

  OPTIMIZATIONS

    HDFS-4075. Reduce recommissioning overhead (Kihwal Lee via daryn)

    HDFS-3990.  NN's health report has severe performance problems (daryn)

    HDFS-4181.  LeaseManager tries to double remove and prints extra messages
    (Kihwal Lee via daryn)

  BUG FIXES

    HDFS-3829. TestHftpURLTimeouts fails intermittently with JDK7  (Trevor
    Robinson via tgraves)

    HDFS-3824. TestHftpDelegationToken fails intermittently with JDK7 (Trevor
    Robinson via tgraves)

    HDFS-3224. Bug in check for DN re-registration with different storage ID
    (jlowe)

    HDFS-4090. getFileChecksum() result incompatible when called against
    zero-byte files. (Kihwal Lee via daryn)

    HDFS-4172. namenode does not URI-encode parameters when building URI for
    datanode request (Derek Dagit via bobby)

    HDFS-4182. SecondaryNameNode leaks NameCache entries (bobby)

    HDFS-4186. logSync() is called with the write lock held while releasing
    lease (Kihwal Lee via daryn)

Release 0.23.4

  INCOMPATIBLE CHANGES

  NEW FEATURES

  IMPROVEMENTS

  OPTIMIZATIONS

  BUG FIXES

    HDFS-3831. Failure to renew tokens due to test-sources left in classpath
    (jlowe via bobby)

Release 0.23.3

  INCOMPATIBLE CHANGES

  NEW FEATURES

  IMPROVEMENTS

  OPTIMIZATIONS

  BUG FIXES

    HDFS-3166. Add timeout to Hftp connections.  (Daryn Sharp via szetszwo)

    HDFS-3176. Use MD5MD5CRC32FileChecksum.readFields() in JsonUtil .  (Kihwal
    Lee via szetszwo)

    HDFS-2652. Add support for host-based delegation tokens.  (Daryn Sharp via
    szetszwo)

    HDFS-3308. Uses canonical URI to select delegation tokens in HftpFileSystem
    and WebHdfsFileSystem.  (Daryn Sharp via szetszwo)

    HDFS-3312. In HftpFileSystem, the namenode URI is non-secure but the
    delegation tokens have to use secure URI.  (Daryn Sharp via szetszwo)

    HDFS-3318. Use BoundedInputStream in ByteRangeInputStream, otherwise, it
    hangs on transfers >2 GB.  (Daryn Sharp via szetszwo)

    HDFS-3321. Fix safe mode turn off tip message.  (Ravi Prakash via szetszwo)

    HDFS-3334. Fix ByteRangeInputStream stream leakage.  (Daryn Sharp via
    szetszwo)

    HDFS-3331. In namenode, check superuser privilege for setBalancerBandwidth
    and acquire the write lock for finalizeUpgrade.  (szetszwo)

    HDFS-3577. In DatanodeWebHdfsMethods, use MessageBodyWriter instead of
    StreamingOutput, otherwise, it will fail to transfer large files.
    (szetszwo)

    HDFS-3646. LeaseRenewer can hold reference to inactive DFSClient
    instances forever. (Kihwal Lee via daryn)

    HDFS-3696. Set chunked streaming mode in WebHdfsFileSystem write operations
    to get around a Java library bug causing OutOfMemoryError.  (szetszwo)

    HDFS-3553. Hftp proxy tokens are broken (daryn)

    HDFS-3718. Datanode won't shutdown because of runaway DataBlockScanner
    thread (Kihwal Lee via daryn)

    HDFS-3861. Deadlock in DFSClient (Kihwal Lee via daryn)

    HDFS-3873. Hftp assumes security is disabled if token fetch fails (daryn)

    HDFS-3852. TestHftpDelegationToken is broken after HADOOP-8225 (daryn)

    HDFS-3890. filecontext mkdirs doesn't apply umask as expected
    (Tom Graves via daryn)

Release 0.23.2 - UNRELEASED

  INCOMPATIBLE CHANGES

    HDFS-2887. FSVolume, is a part of FSDatasetInterface implementation, should
    not be referred outside FSDataset.  A new FSVolumeInterface is defined.
    The BlockVolumeChoosingPolicy.chooseVolume(..) method signature is also
    updated.  (szetszwo)

  NEW FEATURES

    HDFS-2943. Expose last checkpoint time and transaction stats as JMX
    metrics. (atm)

  IMPROVEMENTS

    HDFS-2931. Switch DataNode's BlockVolumeChoosingPolicy to private-audience.
    (harsh via szetszwo)

    HDFS-2725. hdfs script usage information is missing the information 
    about "dfs" command (Prashant Sharma via stevel)

    HDFS-2907.  Add a conf property dfs.datanode.fsdataset.factory to make
    FSDataset in Datanode pluggable.  (szetszwo)

    HDFS-2985. Improve logging when replicas are marked as corrupt. (todd)

    HDFS-3098. Update and add tests for HADOOP-8173. (Daryn Sharp via szetszwo)

    HDFS-3104. Add tests for HADOOP-8175. (Daryn Sharp via szetszwo)

    HDFS-3066. Cap space usage of default log4j rolling policy.
    (Patrick Hunt via eli)

  OPTIMIZATIONS

  BUG FIXES
    HDFS-2923. Namenode IPC handler count uses the wrong configuration key
    (todd)

    HDFS-2869. Fix an error in the webhdfs docs for the mkdir op (harsh)

    HDFS-776.  Fix exception handling in Balancer.  (Uma Maheswara Rao G
    via szetszwo)

    HDFS-2815. Namenode sometimes oes not come out of safemode during 
    NN crash + restart. (Uma Maheswara Rao via suresh)

    HDFS-2950. Secondary NN HTTPS address should be listed as a
    NAMESERVICE_SPECIFIC_KEY. (todd)

    HDFS-2525. Race between BlockPoolSliceScanner and append. (Brandon Li
    via jitendra)

    HDFS-2938. Recursive delete of a large directory make namenode
    unresponsive. (Hari Mankude via suresh)

    HDFS-2969. ExtendedBlock.equals is incorrectly implemented (todd)

    HDFS-2944. Typo in hdfs-default.xml causes
    dfs.client.block.write.replace-datanode-on-failure.enable to be mistakenly
    disabled. (atm)

    HDFS-2981. In hdfs-default.xml, the default value of
    dfs.client.block.write.replace-datanode-on-failure.enable should be true.
    (szetszwo)

    HDFS-3008. Negative caching of local addrs doesn't work. (eli)

    HDFS-3006. In WebHDFS, when the return body is empty, set the Content-Type
    to application/octet-stream instead of application/json.  (szetszwo)

    HDFS-2991. Fix case where OP_ADD would not be logged in append(). (todd)

    HDFS-3012. Exception while renewing delegation token. (Bobby Evans via
    jitendra)

    HDFS-3032. Change DFSClient.renewLease() so that it only retries up to the
    lease soft-limit.  (Kihwal Lee via szetszwo)

    HDFS-2038. Update TestHDFSCLI to handle relative paths with globs.
    (Kihwal Lee via szetszwo)

    HDFS-3101. Cannot read empty file using WebHDFS.  (szetszwo)

    HDFS-3160. httpfs should exec catalina instead of forking it.
    (Roman Shaposhnik via eli)

Release 0.23.1 - 2012-02-17 

  INCOMPATIBLE CHANGES

  NEW FEATURES

    HDFS-2316. [umbrella] WebHDFS: a complete FileSystem implementation for 
    accessing HDFS over HTTP (szetszwo)

    HDFS-2594. Support getDelegationTokens and createSymlink in WebHDFS.
    (szetszwo)

    HDFS-2545. Change WebHDFS to support multiple namenodes in federation.
    (szetszwo)

    HDFS-2178. Contributing Hoop to HDFS, replacement for HDFS proxy with 
    read/write capabilities. (tucu)

  IMPROVEMENTS
    HDFS-2560. Refactor BPOfferService to be a static inner class (todd)

    HDFS-2544. Hadoop scripts unconditionally source
    "$bin"/../libexec/hadoop-config.sh. (Bruno Mahé via tomwhite)

    HDFS-2543. HADOOP_PREFIX cannot be overridden. (Bruno Mahé via tomwhite)

    HDFS-2562. Refactor DN configuration variables out of DataNode class
    (todd)

    HDFS-2563. Some cleanup in BPOfferService. (todd)

    HDFS-2568. Use a set to manage child sockets in XceiverServer.
    (harsh via eli)

    HDFS-2454. Move maxXceiverCount check to before starting the
    thread in dataXceiver. (harsh via eli)

    HDFS-2570. Add descriptions for dfs.*.https.address in hdfs-default.xml.
    (eli)

    HDFS-2536. Remove unused imports. (harsh via eli)

    HDFS-2566. Move BPOfferService to be a non-inner class. (todd)

    HDFS-2552. Add Forrest doc for WebHDFS REST API.  (szetszwo)

    HDFS-2587. Add apt doc for WebHDFS REST API.  (szetszwo)

    HDFS-2604. Add a log message to show if WebHDFS is enabled and a
    configuration section in the forrest doc.  (szetszwo)

    HDFS-2511. Add dev script to generate HDFS protobufs. (tucu)

    HDFS-2654. Make BlockReaderLocal not extend RemoteBlockReader2. (eli)

    HDFS-2675. Reduce warning verbosity when double-closing edit logs
    (todd)

    HDFS-2335. DataNodeCluster and NNStorage always pull fresh entropy.
    (Uma Maheswara Rao G via eli)

    HDFS-2574. Remove references to some deprecated properties in conf 
    templates and defaults files. (Joe Crobak via harsh)

    HDFS-2722. HttpFs should not be using an int for block size. (harsh)

    HDFS-2710. Add HDFS tests related to HADOOP-7933. (sid via suresh)

    HDFS-2349. Corruption detected during block transfers between DNs
    should log a WARN instead of INFO. (harsh)

    HDFS-2729. Update BlockManager's comments regarding the invalid block
    set (harsh)

    HDFS-2726. Fix a logging issue under DFSClient's createBlockOutputStream
    method (harsh)

    HDFS-554. Use System.arraycopy in BlockInfo.ensureCapacity. (harsh)

    HDFS-1314. Make dfs.blocksize accept size-indicating prefixes.
    (Sho Shimauchi via harsh)

    HDFS-69. Improve the 'dfsadmin' commandline help. (harsh)

    HDFS-2788. HdfsServerConstants#DN_KEEPALIVE_TIMEOUT is dead code. (eli)

    HDFS-362.  FSEditLog should not writes long and short as UTF8, and should
    not use ArrayWritable for writing non-array items.  (Uma Maheswara Rao G
    via szetszwo)

    HDFS-2803. Add logging to LeaseRenewer for better lease expiration debugging.
    (Jimmy Xiang via todd)

    HDFS-2817. Combine the two TestSafeMode test suites. (todd)

    HDFS-2818. Fix a missing space issue in HDFS webapps' title tags.
    (Devaraj K via harsh)

    HDFS-2397. Undeprecate SecondaryNameNode. (eli)

    HDFS-2814 NamenodeMXBean does not account for svn revision in the version 
    information. (Hitesh Shah via jitendra)

    HDFS-2784. Update hftp and hdfs for host-based token support. 
    (Kihwal Lee via jitendra)

    HDFS-2785. Update webhdfs and httpfs for host-based token support.
    (Robert Joseph Evans via jitendra)

    HDFS-2868. Expose xceiver counts via the DataNode MXBean. (harsh)

    HDFS-3139. Minor Datanode logging improvement. (eli)

  OPTIMIZATIONS

    HDFS-2130. Switch default checksum to CRC32C. (todd)

    HDFS-2533. Remove needless synchronization on some FSDataSet methods.
    (todd)

    HDFS-2129. Simplify BlockReader to not inherit from FSInputChecker.
    (todd)

    HDFS-2246. Enable reading a block directly from local file system
    for a client on the same node as the block file.  (Andrew Purtell,
    Suresh Srinivas and Jitendra Nath Pandey via szetszwo)

    HDFS-2825. Add test hook to turn off the writer preferring its local
    DN. (todd)

    HDFS-2826. Add test case for HDFS-1476 (safemode can initialize
    replication queues before exiting) (todd)

    HDFS-2864. Remove some redundant methods and the constant METADATA_VERSION
    from FSDataset.  (szetszwo)

    HDFS-2879. Change FSDataset to package private.  (szetszwo)

  BUG FIXES

    HDFS-2541. For a sufficiently large value of blocks, the DN Scanner 
    may request a random number with a negative seed value. (harsh via eli)

    HDFS-2502. hdfs-default.xml should include dfs.name.dir.restore.
    (harsh via eli)

    HDFS-2567. When 0 DNs are available, show a proper error when
    trying to browse DFS via web UI. (harsh via eli)

    HDFS-2575. DFSTestUtil may create empty files (todd)

    HDFS-2588. hdfs jsp pages missing DOCTYPE. (Dave Vronay via mattf)

    HDFS-2590. Fix the missing links in the WebHDFS forrest doc.  (szetszwo)

    HDFS-2596. TestDirectoryScanner doesn't test parallel scans. (eli)

    HDFS-2606. webhdfs client filesystem impl must set the content-type 
    header for create/append. (tucu)

    HDFS-2614. hadoop dist tarball is missing hdfs headers. (tucu)
 
    HDFS-2653. DFSClient should cache whether addrs are non-local when
    short-circuiting is enabled. (eli)

    HDFS-2649. eclipse:eclipse build fails for hadoop-hdfs-httpfs.
    (Jason Lowe via eli)

    HDFS-2640. Javadoc generation hangs. (tomwhite)

    HDFS-2553. Fix BlockPoolSliceScanner spinning in a tight loop (Uma
    Maheswara Rao G via todd)

    HDFS-2658. HttpFS introduced 70 javadoc warnings. (tucu)

    HDFS-2706. Use configuration for blockInvalidateLimit if it is set.
    (szetszwo)

    HDFS-2646. Hadoop HttpFS introduced 4 findbug warnings. (tucu)

    HDFS-2657. TestHttpFSServer and TestServerWebApp are failing on trunk. 
    (tucu)

    HDFS-2705. HttpFS server should check that upload requests have correct 
    content-type. (tucu)

    HDFS-2707. HttpFS should read the hadoop-auth secret from a file 
    instead inline from the configuration. (tucu)

    HDFS-2790. FSNamesystem.setTimes throws exception with wrong
    configuration name in the message. (Arpit Gupta via eli)

    HDFS-2810. Leases not getting renewed properly by clients (todd)

    HDFS-2751. Datanode may incorrectly drop OS cache behind reads
    even for short reads. (todd)

    HDFS-2816. Fix missing license header in httpfs findbugsExcludeFile.xml.
    (hitesh via tucu)

    HDFS-2822. processMisReplicatedBlock incorrectly identifies
    under-construction blocks as under-replicated. (todd)

    HDFS-442. dfsthroughput in test jar throws NPE (harsh)

    HDFS-2836. HttpFSServer still has 2 javadoc warnings in trunk.
    (revans2 via tucu)

    HDFS-2837. mvn javadoc:javadoc not seeing LimitedPrivate class
    (revans2 via tucu)

    HDFS-2840. TestHostnameFilter should work with localhost or 
    localhost.localdomain (tucu)

    HDFS-2791. If block report races with closing of file, replica is
    incorrectly marked corrupt. (todd)

    HDFS-2827.  When the parent of a directory is the root, renaming the
    directory results in leases updated incorrectly.  (Uma Maheswara Rao G
    via szetszwo)

    HDFS-2835. Fix findbugs and javadoc issue with GetConf.java.
    (suresh)

    HDFS-2889. getNumCurrentReplicas is package private but should be public on
    0.23 (see HDFS-2408). (Gregory Chanan via atm)

    HDFS-2893. The start/stop scripts don't start/stop the 2NN when
    using the default configuration. (eli)

Release 0.23.0 - 2011-11-01 

  INCOMPATIBLE CHANGES

    HDFS-1526. Dfs client name for a map/reduce task should be unique
    among threads. (hairong)

    HDFS-1536. Improve HDFS WebUI. (hairong)

    HDFS-2210. Remove hdfsproxy. (eli)

    HDFS-1073. Redesign the NameNode's storage layout for image checkpoints
    and edit logs to introduce transaction IDs and be more robust.
    Please see HDFS-1073 section below for breakout of individual patches.

  NEW FEATURES

    HDFS-1359. Add BlockPoolID to Block. (suresh)

    HDFS-1365. Federation: propose ClusterID and BlockPoolID format 
    (Tanping via boryas)

    HDFS-1394. Federation: modify -format option for namenode to generated 
    new blockpool id and accept newcluster (boryas)

    HDFS-1400. Federation: DataTransferProtocol uses ExtendedBlockPool to 
    include BlockPoolID in the protocol. (suresh)

    HDFS-1428. Federation : add cluster ID and block pool ID into 
    Name node web UI(Tanping via boryas)

    HDFS-1450. Federation: Introduce block pool ID into FSDatasetInterface.
    (suresh)

    HDFS-1632. Federation: data node storage structure changes and
    introduce block pool storage. (Tanping via suresh)

    HDFS-1634. Federation: Convert single threaded DataNode into 
    per BlockPool thread model.(boryas)

    HDFS-1637. Federation: FSDataset in Datanode should be created after 
    initial handshake with namenode. (boryas and jitendra)

    HDFS-1653. Federation: Block received message from datanode sends invalid 
    DatanodeRegistration. (Tanping via suresh)

    HDFS-1645. Federation: DatanodeCommond.Finalize needs to include 
    BlockPoolId.  (suresh)

    HDFS-1638. Federation: DataNode.handleDiskError needs to inform 
    ALL namenodes if a disk failed (boryas)

    HDFS-1647. Federation: Multiple namenode configuration. (jitendra)

    HDFS-1639. Federation: Add block pool management to FSDataset. (suresh)

    HDFS-1648. Federation: Only DataStorage must be locked using in_use.lock 
    and no locks must be associated with BlockPoolStorage. (Tanping via suresh)

    HDFS-1641. Federation: Datanode fields that are no longer used should 
    be removed (boryas)

    HDFS-1642. Federation: add Datanode.getDNRegistration(String bpid) 
    method  (boryas)

    HDFS-1643. Federation: remove namenode argument from DataNode 
    constructor (boryas)

    HDFS-1657. Federation: Tests that corrupt block files fail due to changed 
    file path in federation. (suresh)

    HDFS-1661. Federation: Remove unnecessary TODO:FEDERATION comments.
    (jitendra)

    HDFS-1660. Federation: Datanode doesn't start with two namenodes (boryas)

    HDFS-1650. Federation: TestReplication fails. (Tanping via suresh)

    HDFS-1651. Federation: Tests fail due to null pointer exception in 
    Datnode#shutdown() method. (Tanping via suresh)

    HDFS-1649. Federation: Datanode command to refresh namenode list at 
    the datanode. (jitendra)

    HDFS-1646. Federation: MiniDFSClsuter#waitActive() waits for ever 
    with the introduction of BPOfferService in datanode. (suresh)

    HDFS-1659. Federation: BPOfferService exits after one iteration 
    incorrectly.  (Tanping via suresh)

    HDFS-1654. Federation: Fix TestDFSUpgrade and TestDFSRollback failures.
    (suresh)
    
    HDFS-1668. Federation: Datanodes sends block pool usage information 
    to the namenode in heartbeat. (suresh)

    HDFS-1669. Federation: Fix TestHftpFileSystem failure. (suresh)

    HDFS-1670. Federation: remove dnRegistration from Datanode (boryas)

    HDFS-1662. Federation: fix unit test case, TestCheckpoint 
    and TestDataNodeMXBean (tanping via boryas)

    HDFS-1671. Federation: shutdown in DataNode should be able to 
    shutdown individual BP threads as well as the whole DN (boryas).

    HDFS-1663. Federation: Rename getPoolId() everywhere to 
    getBlockPoolId() (tanping via boryas)

    HDFS-1652. FederationL Add support for multiple namenodes in 
    MiniDFSCluster. (suresh)

    HDFS-1672. Federation: refactor stopDatanode(name) to work 
    with multiple Block Pools (boryas)

    HDFS-1687. Federation: DirectoryScanner changes for 
    federation (Matt Foley via boryas)

    HDFS-1626. Make BLOCK_INVALIDATE_LIMIT configurable. (szetszwo)

    HDFS-1655. Federation: DatablockScanner should scan blocks for 
    all the block pools. (jitendra)

    HDFS-1664. Federation: Add block pool storage usage to Namenode WebUI.
    (Tanping via suresh)

    HDFS-1674. Federation: Rename BlockPool class to BlockPoolSlice. 
    (jghoman, Tanping via suresh)

    HDFS-1673. Federation: Datanode changes to track block token secret per 
    namenode. (suresh)

    HDFS-1677. Federation: Fix TestFsck and TestListCorruptFileBlocks 
    failures. (Tanping via suresh)

    HDFS-1678. Federation: Remove unnecessary #getBlockpool() 
    for NameNodeMXBean in FSNameSystem. (Tanping via Suresh)

    HDFS-1688. Federation: Fix failures in fault injection tests,
    TestDiskError, TestDatanodeRestart and TestDFSTartupVersions. (suresh)

    HDFS-1696. Federation: when build version doesn't match - 
    datanode should wait (keep connecting) untill NN comes up 
    with the right version (boryas)

    HDFS-1681. Balancer: support per pool and per node policies. (szetszwo)

    HDFS-1695. Federation: Fix testOIV and TestDatanodeUtils 
    (jhoman and tanping via boryas)

    HDFS:1699. Federation: Fix failure of TestBlockReport.
    (Matt Foley via suresh)

    HDFS-1698. Federation: TestOverReplicatedBlocks and TestWriteToReplica 
    failing. (jhoman and jitendra)

    HDFS-1701. Federation: Fix TestHeartbeathandling.
    (Erik Steffl and Tanping Wang via suresh)

    HDFS-1693. Federation: Fix TestDFSStorageStateRecovery failure. (suresh)

    HDFS-1694. Federation: SimulatedFSDataset changes to work with
    federation and multiple block pools. (suresh)

    HDFS-1689. Federation: Configuration for namenodes. (suresh and jitendra)

    HDFS-1682. Change Balancer CLI for multiple namenodes and balancing
    policy.  (szetszwo)

    HDFS-1697. Federation: fix TestBlockRecovery (boryas)

    HDFS-1702. Federation: fix TestBackupNode and TestRefreshNamendoes
    failures. (suresh)

    HDFS-1706. Federation: TestFileAppend2, TestFileAppend3 and 
    TestBlockTokenWithDFS failing. (jitendra)

    HDFS-1704. Federation: Add a tool that lists namenodes, secondary and
    backup from configuration file. (suresh)

    HDFS-1711. Federation: create method for updating machine name in 
    DataNode.java (boryas)

    HDFS-1712. Federation: when looking up datanode we should use machineNmae 
    (in testOverReplicatedBlocks) (boryas)

    HDFS-1709. Federation: Error "nnaddr url param is null" when clicking on a 
    node from NN Live Node Link. (jitendra)

    HDFS-1714. Federation: refactor upgrade object in DataNode (boryas) 

    HDFS-1715. Federation: warning/error not generated when datanode sees 
    inconsistent/different Cluster ID between namenodes (boryas)

    HDFS-1715. Federation: warning/error not generated when datanode sees 
    inconsistent/different Cluster ID between namenodes (boryas)

    HDFS-1716. Federation: Add decommission tests for federated namenodes.
    (suresh)

    HDFS-1713. Federation: Prevent DataBlockScanner from running in tight loop.
    (jitendra)

    HDFS-1721. Federation: Configuration for principal names should not be 
    namenode specific. (jitendra)

    HDFS-1717. Federation: FSDataset volumeMap access is not synchronized
    correctly. (suresh)

    HDFS-1722. Federation: Add flag to MiniDFSCluser to differentiate between
    federation and non-federation modes. (boryas via suresh)

    HDFS-1718. Federation: MiniDFSCluster#waitActive() bug causes some tests
    to fail. (suresh)

    HDFS-1719. Federation: Fix TestDFSRemove that fails intermittently.
    (suresh)

    HDFS-1720. Federation: FSVolumeSet volumes is not synchronized correctly.
    (suresh)

    HDFS-1700. Federation: fsck needs to work with federation changes.
    (Matt Foley via suresh)

    HDFS-1482. Add listCorruptFileBlocks to DistributedFileSystem.
    (Patrick Kling via hairong)

    HDFS-1448. Add a new tool Offline Edits Viewer (oev).  (Erik Steffl
    via szetszwo)

    HDFS-1735. Federation: merge FSImage change in federation to
    FSImage+NNStorage refactoring in trunk. (suresh)

    HDFS-1737. Federation: Update the layout version for federation
    changes. (suresh)

    HDFS-1744. Federation: Add new layout version to offline image viewer
    and edits viewer. (suresh)

    HDFS-1745. Federation: Fix fault injection test failures. (suresh)

    HDFS-1746. Federation: TestFileAppend3 fails intermittently. (jitendra)

    HDFS-1703. Improve start/stop scripts and add decommission tool for
    federation. (Tanping Wang, Erik Steffl via suresh)

    HDFS-1749. Federation: TestListCorruptFileBlocks failing in federation 
    branch. (jitendra)

    HDFS-1754. Federation: testFsck fails. (boryas)

    HDFS-1755. Federation: The BPOfferService must always connect to namenode
    as the login user. (jitendra)

    HDFS-1675. Support transferring RBW between datanodes. (szetszwo)

    HDFS-1791. Federation: Add command to delete block pool directories 
    from a datanode. (jitendra)

    HDFS-1761. Add a new DataTransferProtocol operation, Op.TRANSFER_BLOCK,
    for transferring RBW/Finalized with acknowledgement and without using RPC.
    (szetszwo)

    HDFS-1813. Federation: Authentication using BlockToken in RPC to datanode 
               fails. (jitendra)

    HDFS-1630. Support fsedits checksum. (hairong)

    HDFS-1606. Provide a stronger data guarantee in the write pipeline by
    adding a new datanode when an existing datanode failed.  (szetszwo)

    HDFS-1442. Api to get delegation token in Hdfs class. (jitendra)

    HDFS-1070. Speedup namenode image loading and saving by storing only
    local file names. (hairong)
    
    HDFS-1751. Intrinsic limits for HDFS files, directories (daryn via boryas).

    HDFS-1873. Federation: Add cluster management web console.
    (Tanping Wang via suresh)

    HDFS 1911 HDFS tests for the newly added viewfs

    HDFS-1814. Add "hdfs groups" command to query the server-side groups
    resolved for a user. (Aaron T. Myers via todd)

    HDFS-1914. Federation: namenode storage directories must be configurable
    specific to name service. (suresh)

    HDFS-1963. Create RPM and Debian packages for HDFS. Changes deployment
    layout to be consistent across the binary tgz, rpm, and deb.
    (Eric Yang via omalley)

    HDFS-2058. Change Data Transfer wire protocol to use protocol buffers.
    (todd)

    HDFS-2055. Add hflush support to libhdfs. (Travis Crawford via eli)
   
    HDFS-2083. Query JMX statistics over http via JMXJsonServlet. (tanping)

    HDFS-2156. Make hdfs and mapreduce rpm only depend on the same major 
    version for common and hdfs. (eyang via omalley)

    HDFS-2202. Add a new DFSAdmin command to set balancer bandwidth of
    datanodes without restarting.  (Eric Payne via szetszwo)

    HDFS-2284. Add a new FileSystem, webhdfs://, for supporting write Http
    access to HDFS.  (szetszwo)

    HDFS-2317. Support read access to HDFS in WebHDFS.  (szetszwo)

    HDFS-2338. Add configuration option to enable/disable WebHDFS.
    (jitendra via szetszwo)

    HDFS-2318. Provide authentication to WebHDFS using SPNEGO and delegation
    tokens.  (szetszwo)

    HDFS-2340. Support getFileBlockLocations and getDelegationToken in WebHDFS.
    (szetszwo)

    HDFS-2348. Support getContentSummary and getFileChecksum in WebHDFS.
    (szetszwo)

    HDFS-2385. Support renew and cancel delegation tokens in WebHDFS.
    (szetszwo)

    HDFS-2539. Support doAs and GETHOMEDIRECTORY in WebHDFS.
    (szetszwo)

  IMPROVEMENTS

    HDFS-1875. MiniDFSCluster hard-codes dfs.datanode.address to localhost
    (Eric Payne via mattf)

    HDFS-2019. Fix all the places where Java method File.list is used with
    FileUtil.list API (Bharath Mundlapudi via mattf)

    HDFS-1934. Fix NullPointerException when certain File APIs return null
    (Bharath Mundlapudi via mattf)

    HDFS-1510. Added test-patch.properties required by test-patch.sh (nigel)

    HDFS-1628. Display full path in AccessControlException.  (John George
    via szetszwo)

    HDFS-1707. Federation: Failure in browsing data on new namenodes. 
    (jitendra)

    HDFS-1683. Test Balancer with multiple NameNodes.  (szetszwo)

    HDFS-1547. Improve decommission mechanism. (suresh)

    HDFS-2143. Federation: In cluster web console, add link to namenode page
    that displays live and dead datanodes. (Ravi Prakash via suresh)

    HDFS-1588. Remove hardcoded strings for configuration keys, "dfs.hosts"
    and "dfs.hosts.exlude". (Erik Steffl via suresh)

    HDFS-1481. NameNode should validate fsimage before rolling. (hairong)

    HDFS-1506. Refactor fsimage loading code. (hairong)

    HDFS-1533. A more elegant FileSystem#listCorruptFileBlocks API
    (HDFS portion) (Patrick Kling via hairong)

    HDFS-1476. listCorruptFileBlocks should be functional while the
    name node is in safe mode. (Patrick Kling via hairong)

    HDFS-1534. Fix some incorrect logs in FSDirectory. (eli)
    
    HDFS-1539. A config option for the datanode to fsycn a block file
    when block is completely written. (dhruba)

    HDFS-1335. HDFS side change of HADDOP-6904: RPC compatibility. (hairong)

    HDFS-1557. Separate Storage from FSImage. (Ivan Kelly via jitendra)
    
    HDFS-560 Enhancements/tuning to hadoop-hdfs/build.xml

    HDFS-1629. Add a method to BlockPlacementPolicy for keeping the chosen
    nodes in the output array.  (szetszwo)

    HDFS-1731. Allow using a file to exclude certain tests from build (todd)

    HDFS-1736. Remove the dependency from DatanodeJspHelper to FsShell.
    (Daryn Sharp via szetszwo)
    
    HDFS-780. Revive TestFuseDFS. (eli)

    HDFS-1445. Batch the calls in DataStorage to FileUtil.createHardLink().
    (Matt Foley via jghoman)

    HDFS-1763. Replace hard-coded option strings with variables from
    DFSConfigKeys. (eli)

    HDFS-1541. Not marking datanodes dead when namenode in safemode.
    (hairong)

    HDFS-1120. Make DataNode's block-to-device placement policy pluggable
    (Harsh J Chouraria via todd)

    HDFS-1785. In BlockReceiver and DataXceiver, clientName.length() is used
    multiple times for determining whether the source is a client or a
    datanode.  (szetszwo)

    HDFS-1789. Refactor frequently used codes from DFSOutputStream and
    DataXceiver.  (szetszwo)

    HDFS-1767. Namenode ignores non-initial block report from datanodes
    when in safemode during startup. (Matt Foley via suresh)

    HDFS-1817. Move pipeline_Fi_[39-51] from TestFiDataTransferProtocol
    to TestFiPipelineClose.  (szetszwo)

    HDFS-1760. In FSDirectory.getFullPathName(..), it is better to return "/"
    for root directory instead of an empty string.  (Daryn Sharp via szetszwo)

    HDFS-1833. Reduce repeated string constructions and unnecessary fields,
    and fix comments in BlockReceiver.PacketResponder.  (szetszwo)

    HDFS-1486. Generalize CLITest structure and interfaces to faciliate
    upstream adoption (e.g. for web testing). (cos)

    HDFS-1844. Move "fs -help" shell command tests from HDFS to COMMOM; see
    also HADOOP-7230.  (Daryn Sharp via szetszwo)

    HDFS-1840. In DFSClient, terminate the lease renewing thread when all files
    being written are closed for a grace period, and start a new thread when
    new files are opened for write.  (szetszwo)

    HDFS-1854. make failure message more useful in
    DFSTestUtil.waitReplication(). (Matt Foley via eli)

    HDFS-1562. Add rack policy tests. (eli)
 
    HDFS-1856. TestDatanodeBlockScanner waits forever, errs without giving 
    information. (Matt Foley via eli)

    HDFS-1295. Improve namenode restart times by short-circuiting the
    first block reports from datanodes. (Matt Foley via suresh)
    Corrected merge error in DataNode.java. (Matt Foley)

    HDFS-1843. Discover file not found early for file append. 
    (Bharath Mundlapudi via jitendra)

    HDFS-1862. Improve test reliability of HDFS-1594. (Aaron T. Myers via eli)

    HDFS-1846. Preallocate edit log with OP_INVALID instead of zero bytes
    to ensure blocks are actually allocated. (Aaron T. Myers via todd)

    HDFS-1741. Provide a minimal pom file to allow integration of HDFS into Sonar
    analysis (cos)

    HDFS-1870. Move and rename DFSClient.LeaseChecker to a seperated class
    LeaseRenewer.  (szetszwo)

    HDFS-1866. Document dfs.datanode.max.transfer.threads in hdfs-default.xml
    (Harsh J Chouraria via todd)

    HDFS-1890. Improve the name, class and value type of the map
    LeaseRenewer.pendingCreates.  (szetszwo)

    HDFS-1865. Share LeaseRenewer among DFSClients so that there is only a
    LeaseRenewer thread per namenode per user.  (szetszwo)

    HDFS-1906. Remove logging exception stack trace in client logs when one of
    the datanode targets to read from is not reachable. (suresh)

    HDFS-1378. Edit log replay should track and report file offsets in case of
    errors. (Aaron T. Myers and Todd Lipcon via todd)

    HDFS-1917. Separate hdfs jars from common in ivy configuration.  (Eric Yang
    via szetszwo)

    HDFS-1899. GenericTestUtils.formatNamenode should be moved to DFSTestUtil
    (Ted Yu via todd)

    HDFS-1117. Metrics 2.0 HDFS instrumentation. (Luke Lu via suresh)

    HDFS-1946. HDFS part of HADOOP-7291. (eli)

    HDFS-1945. Removed the deprecated fields in DataTransferProtocol.
    (szetszwo)

    HDFS-1730. Use DaemonFactory from common and delete it from HDFS.
    (Tanping via suresh)

    HDFS-1573. Add useful tracing information to Lease Renewer thread names
    (todd)

    HDFS-1939. In ivy.xml, test conf should not extend common conf.
    (Eric Yang via szetszwo)

    HDFS-1332  Include more information in exceptions and debug messages
    when BlockPlacementPolicy cannot be satisfied.  (Ted Yu via szetszwo)

    HDFS-1958. Confirmation should be more lenient of user input when
    formatting the NameNode. (todd)

    HDFS-1905. Improve namenode -format command by not making -clusterId
    parameter mandatory. (Bharath Mundlapudi via suresh)

    HDFS-1877. Add a new test for concurrent read and write.  (CW Chung
    via szetszwo)

    HDFS-1959. Better error message for missing namenode directory. (eli)

    HDFS-1996. ivy: hdfs test jar should be independent to common test jar.
    (Eric Yang via szetszwo)

    HDFS-1812. TestHDFSCLI should clean up cluster in teardown method.
    (Uma Maheswara Rao G via todd)

    HDFS-1884. Improve TestDFSStorageStateRecovery to properly throw in the
    case of errors. (Aaron T. Myers via todd)

    HDFS-1727. fsck command should display command usage if user passes any
    illegal argument. (Sravan Kumar via todd)

    HDFS-1636. If dfs.name.dir points to an empty dir, namenode format
    shouldn't require confirmation. (Harsh J Chouraria via todd)

    HDFS-1966. Encapsulate individual DataTransferProtocol op headers.
    (szetszwo)

    HDFS-2024. Format TestWriteRead source codes.  (CW Chung via szetszwo)

    HDFS-1968. Enhance TestWriteRead to support position/sequential read,
    append, truncate and verbose options.  (CW Chung via szetszwo)

    HDFS-1986. Add option to get http/https address from 
    DFSUtil#getInfoServer(). (Tanping via suresh)

    HDFS-2029. In TestWriteRead, check visible length immediately after
    openning the file and fix code style.  (John George via szetszwo)

    HDFS-2040. Only build libhdfs if a flag is passed. (eli)

    HDFS-1586. Add InterfaceAudience and InterfaceStability annotations to 
    MiniDFSCluster. (suresh)

    HDFS-2003. Separate FSEditLog reading logic from edit log memory state
    building logic. (Ivan Kelly via todd)

    HDFS-2066. Create a package and individual class files for
    DataTransferProtocol.  (szetszwo)

    HADOOP-7106. Reorganize project SVN layout to "unsplit" the projects.
    (todd, nigel)

    HDFS-2046. Force entropy to come from non-true random for tests. (todd)

    HDFS-2073. Add @Override annotation to NameNode. (suresh)

    HDFS-420. Fuse-dfs should cache fs handles. (Brian Bockelman and eli)

    HDFS-1568. Improve the log messages in DataXceiver.  (Joey Echeverria via
    szetszwo)

    HDFS-2100. Improve TestStorageRestore. (atm)

    HDFS-2092. Remove some object references to Configuration in DFSClient.
    (Bharath Mundlapudi via szetszwo)

    HDFS-2087. Declare methods in DataTransferProtocol interface, and change
    Sender and Receiver to implement the interface.  (szetszwo)

    HDFS-1723. quota errors messages should use the same scale. (Jim Plush via
    atm)

    HDFS-2110. StreamFile and ByteRangeInputStream cleanup. (eli)

    HDFS-2107. Move block management code from o.a.h.h.s.namenode to a new
    package o.a.h.h.s.blockmanagement.  (szetszwo)

    HDFS-2109. Store uMask as member variable to DFSClient.Conf.  (Bharath
    Mundlapudi via szetszwo)

    HDFS-2111. Add tests for ensuring that the DN will start with a few bad
    data directories. (Harsh J Chouraria via todd)

    HDFS-2134. Move DecommissionManager to the blockmanagement package.
    (szetszwo)

    HDFS-1977. Stop using StringUtils.stringifyException(). 
    (Bharath Mundlapudi via jitendra)

    HDFS-2131. Add new tests for the -overwrite/-f option in put and
    copyFromLocal by HADOOP-7361.  (Uma Maheswara Rao G via szetszwo)

    HDFS-2140. Move Host2NodesMap to the blockmanagement package.  (szetszwo)

    HDFS-2154. In TestDFSShell, use TEST_ROOT_DIR and fix some deprecated
    warnings.  (szetszwo)

    HDFS-2153. Move DFSClientAdapter to test and fix some javac warnings in
    OfflineEditsViewerHelper.  (szetszwo)

    HDFS-2159. Deprecate DistributedFileSystem.getClient() and fixed the
    deprecated warnings in DFSAdmin.  (szetszwo)

    HDFS-2157. Improve header comment in o.a.h.hdfs.server.namenode.NameNode.
    (atm via eli)

    HDFS-2147. Move cluster network topology to block management and fix some
    javac warnings.  (szetszwo)

    HDFS-2141. Remove NameNode roles Active and Standby (they become
    states of the namenode). (suresh)

    HDFS-2161. Move createNamenode(..), createClientDatanodeProtocolProxy(..)
    and Random object creation to DFSUtil; move DFSClient.stringifyToken(..)
    to DelegationTokenIdentifier.  (szetszwo)

    HDFS-1774. Small optimization to FSDataset. (Uma Maheswara Rao G via eli)

    HDFS-2167.  Move dnsToSwitchMapping and hostsReader from FSNamesystem to
    DatanodeManager.  (szetszwo)

    HDFS-2116. Use Mokito in TestStreamFile and TestByteRangeInputStream.
    (Plamen Jeliazkov via shv)

    HDFS-2112.  Move ReplicationMonitor to block management.  (Uma Maheswara
    Rao G via szetszwo)

    HDFS-1739.  Add available volume size to the error message when datanode
    throws DiskOutOfSpaceException.  (Uma Maheswara Rao G via szetszwo)

    HDFS-2144. If SNN shuts down during initialization it does not log the
    cause. (Ravi Prakash via atm)

    HDFS-2180. Refactor NameNode HTTP server into new class. (todd)
    
    HDFS-2198. Remove hardcoded configuration keys. (suresh)

    HDFS-2149. Move EditLogOp serialization formats into FsEditLogOp
    implementations. (Ivan Kelly via todd)

    HDFS-2191. Move datanodeMap from FSNamesystem to DatanodeManager.
    (szetszwo)

    HDFS-2200. Change FSNamesystem.LOG to package private. (szetszwo)

    HDFS-2195. Refactor StorageDirectory to not be an non-static inner class.
    (todd via eli)

    HDFS-2212. Refactor double-buffering code out of EditLogOutputStreams.
    (todd via eli)

    HDFS-2199. Move blockTokenSecretManager from FSNamesystem to BlockManager.
    (Uma Maheswara Rao G via szetszwo)

    HDFS-2187. Make EditLogInputStream act like an iterator over FSEditLogOps
    (Ivan Kelly and todd via todd)

    HDFS-2225. Refactor edit log file management so it's not in classes
    which should be generic to the type of edit log storage. (Ivan Kelly
    via todd)

    HDFS-2108. Move datanode heartbeat handling from namenode package to
    blockmanagement package.  (szetszwo)

    HDFS-2226. Clean up counting of operations in FSEditLogLoader (todd)

    HDFS-2228. Move block and datanode code from FSNamesystem to
    BlockManager and DatanodeManager.  (szetszwo)

    HDFS-2238. In NamenodeFsck.toString(), uses StringBuilder.(..) instead of
    string concatenation.  (Uma Maheswara Rao G via szetszwo)
   
    HDFS-2230. ivy to resolve/retrieve latest common-tests jar published by 
    hadoop common maven build. (gkesavan)

    HDFS-2227. getRemoteEditLogManifest should pull its information from
    FileJournalManager during checkpoint process (Ivan Kelly and Todd Lipcon
    via todd)

    HDFS-2239. Reduce access levels of the fields and methods in FSNamesystem.
    (szetszwo)

    HDFS-2241. Remove implementing FSConstants interface to just get the
    constants from the interface. (suresh)

    HDFS-2237. Change UnderReplicatedBlocks from public to package private.
    (szetszwo)

    HDFS-2233. Add WebUI tests with URI reserved chars. (eli)

    HDFS-2265. Remove unnecessary BlockTokenSecretManager fields/methods from
    BlockManager.  (szetszwo)

    HDFS-2260. Refactor BlockReader into an interface and implementation.
    (todd)

    HDFS-2096. Mavenization of hadoop-hdfs (Alejandro Abdelnur via tomwhite)

    HDFS-2273.  Refactor BlockManager.recentInvalidateSets to a new class.
    (szetszwo)

    HDFS-2266.  Add Namesystem and SafeMode interfaces to avoid directly
    referring to FSNamesystem in BlockManager. (szetszwo)

    HDFS-1217.  Change some NameNode methods from public to package private.
    (Laxman via szetszwo)

    HDFS-2332. Add test for HADOOP-7629 (using an immutable FsPermission
    object as an RPC parameter fails). (todd)

    HDFS-2363. Move datanodes size printing from FSNamesystem.metasave(..)
    to BlockManager.  (Uma Maheswara Rao G via szetszwo)

    HDFS-2209. Make MiniDFS easier to embed in other apps. (stevel)

    HDFS-2205. Log message for failed connection to datanode is not
    followed by a success message. (Ravi Prakash via stevel)

    HDFS-2401. Running a set of methods in a Single Test Class.
    (Jonathan Eagles via mahadev)

    HDFS-2471. Add federation documentation. (suresh)

    HDFS-2485. Improve code layout and constants in UnderReplicatedBlocks
    (stevel)

    HDFS-2356.  Support case insensitive query parameter names in WebHDFS.
    (szetszwo)

    HDFS-2368.  Move SPNEGO conf properties from hdfs-default.xml to
    hdfs-site.xml.  (szetszwo)

    HDFS-2395. Add a root element in the JSON responses of WebHDFS.
    (szetszwo)

    HDFS-2427. Change the default permission in WebHDFS to 755 and add range
    check/validation for all parameters.  (szetszwo)

    HDFS-2501. Add version prefix and root methods to WebHDFS.  (szetszwo)

    HDFS-1869. mkdirs should use the supplied permission for all of the created
    directories. (Daryn Sharp via szetszwo)

    HDFS-2355. Federation: enable using the same configuration file across 
    all the nodes in the cluster. (suresh)

    HDFS-2371. Refactor BlockSender.java for better readability. (suresh)

    HDFS-2493. Remove reference to FSNamesystem in blockmanagement classes.
    (szetszwo)

    HDFS-2294. Download of commons-daemon TAR should not be under target (tucu)

    HDFS-2322. the build fails in Windows because commons-daemon TAR cannot be 
    fetched. (tucu)

    HDFS-2436. Change FSNamesystem.setTimes(..) for allowing setting times on
    directories.  (Uma Maheswara Rao G via szetszwo)

    HDFS-2512. Add textual error message to data transfer protocol responses
    (todd)

    HDFS-2521. Remove custom checksum headers from data transfer protocol
    (todd)

    HDFS-2308. NamenodeProtocol.endCheckpoint is vestigial and can be removed.
    (eli)

  OPTIMIZATIONS

    HDFS-1458. Improve checkpoint performance by avoiding unnecessary image
    downloads and loading. (hairong)

    HDFS-1601. Pipeline ACKs are sent as lots of tiny TCP packets (todd)

    HDFS-1826. NameNode should save image to name directories in parallel
    during upgrade. (Matt Foley via hairong)

    HDFS-2030. Improve usability of namenode -upgrade command.
    (Bharath Mundlapudi via suresh)

    HDFS-2056. Update fetchdt usage. (Tanping Wang via jitendra)

    HDFS-2118. Couple dfs data dir improvements. (eli)

    HDFS-2500. Avoid file system operations in BPOfferService thread while
    processing deletes. (todd)

    HDFS-2465. Add HDFS support for fadvise readahead and drop-behind. (todd)

  BUG FIXES

    HDFS-2347. Fix checkpointTxnCount's comment about editlog size. 
    (Uma Maheswara Rao G via mattf)

    HDFS-2011. Removal and restoration of storage directories on checkpointing
    failure doesn't work properly. (Ravi Prakash via mattf)

    HDFS-1955. FSImage.doUpgrade() was made too fault-tolerant by HDFS-1826.
    (mattf)

    HDFS-2061. Two minor bugs in BlockManager block report processing. (mattf)

    HDFS-1449. Fix test failures - ExtendedBlock must return 
    block file name in #getBlockName(). (suresh)

    HDFS-1680. Fix TestBalancer. (szetszwo)

    HDFS-1705. Balancer command throws NullPointerException. (suresh via
    szetszwo)

    HDFS-1559. Add missing UGM overrides to TestRefreshUserMappings
    (Todd Lipcon via eli)

    HDFS-1585. Fix build after HDFS-1547 (todd)

    HDFS-1684. Balancer cannot start with with multiple namenodes.  (szetszwo)

    HDFS-1516. mvn-install is broken after 0.22 branch creation. (cos)

    HDFS-1360. TestBlockRecovery should bind ephemeral ports.
    (Todd Lipcon via hairong)

    HDFS-1551. Fix pom templates dependency list (gkesavan)

    HDFS-1509. A savenamespace command writes the fsimage and edits into
    all configured directories. (dhruba)

    HDFS-1540. Make Datanode handle errors from RPC calls to namenode
    more elegantly. (dhruba)

    HDFS-1463. Accesstime of a file is not updated in safeMode. (dhruba)

    HDFS-863. Potential deadlock in TestOverReplicatedBlocks. 
    (Ken Goodhope via jghoman)

    HDFS-1607. Fix referenced to misspelled method name getProtocolSigature
    (todd)

    HDFS-1610. Fix TestClientProtocolWithDelegationToken and TestBlockToken
    on trunk after HADOOP-6904 (todd)

    HDFS-1600. Fix release audit warnings on trunk. (todd)

    HDFS-1691. Remove a duplicated static initializer for reading default
    configurations in DFSck.  (Alexey Diomin via szetszwo)

    HDFS-1748. Balancer utilization classification is incomplete.  (szetszwo)

    HDFS-1738. change hdfs jmxget to return an empty string instead of 
    null when an attribute value is not available (tanping vi boryas)

    HDFS-1757. Don't compile fuse-dfs by default. (eli)

    HDFS-1770. TestFiRename fails due to invalid block size. (eli)

    HDFS-1797. Fix new findbugs warning introduced by HDFS-1120 (todd)

    HDFS-1611. Fix up some log messages in DFSClient and MBean registration
    (Uma Maheswara Rao G via todd)

    HDFS-1543. Reduce dev. cycle time by moving system testing artifacts from
    default build and push to maven for HDFS (Luke Lu via cos)

    HDFS-1818. TestHDFSCLI is failing on trunk after HADOOP-7202.
    (Aaron T. Myers via todd)

    HDFS-1828. TestBlocksWithNotEnoughRacks intermittently fails assert.
    (Matt Foley via eli)

    HDFS-1824. delay instantiation of file system object until it is
     needed (linked to HADOOP-7207) (boryas)

    HDFS-1831. Fix append bug in FileContext and implement CreateFlag
    check (related to HADOOP-7223). (suresh)

    HDFS-1594. When the disk becomes full Namenode is getting shutdown and 
    not able to recover. (Aaron T. Myers via eli)

    HDFS-1822. Handle editlog opcode conflict with 0.20.203 during upgrade,
    by throwing an error to indicate the editlog needs to be empty.
    (suresh)

    HDFS-1808. TestBalancer waits forever, errs without giving information.
    (Matt Foley via eli)

    HDFS-1829. TestNodeCount waits forever, errs without giving information.
    (Matt Foley via eli)

    HDFS-1860. when renewing/canceling DelegationToken over http we need to
     pass exception information back to the caller.(boryas)

    HDFS-1871. Mapreduce build fails due to MiniDFSCluster change from
    HDFS-1052. (suresh)

    HDFS-1876. One MiniDFSCluster constructor ignores numDataNodes parameter
    (todd)

    HDFS-1773. Do not show decommissioned datanodes, which are not in both
    include and exclude lists, on web and JMX interfaces.
    (Tanping Wang via szetszwo)

    HDFS-1888. MiniDFSCluster#corruptBlockOnDatanodes() access must be
    public. (suresh)

    HDFS-1889. incorrect path in start/stop dfs script. (John George via eli)

    HDFS-1891. Disable IPV6 for junit tests to fix TestBackupNode failure.
    (suresh)

    HDFS-1898. Tests failing on trunk due to use of NameNode.format.
    (todd via eli)

    HDFS-1902. Fix setrep path display for TestHDFSCLI.  (Daryn Sharp
    via szetszwo)

    HDFS-1827. Fix timeout problem in TestBlockReplacement.  (Matt Foley
    via szetszwo)

    HDFS-1908. Fix a NullPointerException in fi.DataTransferTestUtil.
    (szetszwo)

    HDFS-1912. Update tests for FsShell standardized error messages.
    (Daryn Sharp via szetszwo)

    HDFS-1903. Fix path display for rm/rmr in TestHDFSCLI and TestDFSShell.
    (Daryn Sharp via szetszwo)

    HDFS-1627. Fix NullPointerException in Secondary NameNode. (hairong)

    HDFS-1928. Fix path display for touchz in TestHDFSCLI.
    (Daryn Sharp via todd)

    HDFS-1938. Fix ivy-retrieve-hdfs dependence in build.xml and aop.xml.
    (Eric Yang via szetszwo)

    HDFS-1929. TestEditLogFileOutputStream fails if running on same host as NN
    (Aaron T. Myers via todd)

    HDFS-1933. Update TestDFSShell for improved "test" shell command. (Daryn
    Sharp via todd)
    
    HDFS-1931. Update TestDFSShell for improved "du" shell command. (Daryn
    Sharp via todd)

    HDFS-1439. HDFS Federation: Fix compilation error in TestFiHftp. (suresh)

    HDFS-1881. Federation: after taking snapshot the current directory 
    of datanode is empty. (Tanping Wang via suresh)

    HDFS-1927. Fix a bug which causes ip=null in NameNode audit log.
    (John George via szetszwo)

    HDFS-1953. Federation: Change name node mxbean name in cluster web 
    console. (Tanping Wang via suresh)

    HDFS-1922. Fix recurring failure of TestJMXGet (Luke Lu via todd)

    HDFS-1371. One bad node can incorrectly flag many files as corrupt.
    (Tanping Wang via jitendra)

    HDFS-1943. Fail to start datanode while start-dfs.sh is executed by 
    root user. (Wei Yongjun via jghoman)

    HDFS-1983. Fix path display for copy and rm commands in TestHDFSCLI and
    TestDFSShell. (Daryn Sharp via todd)

    HDFS-1999. Tests use deprecated configs. (Aaron T. Myers via eli)

    HDFS-1592. Datanode startup doesn't honor volumes.tolerated. 
    (Bharath Mundlapudi via jitendra)

    HDFS-1920. libhdfs does not build for ARM processors.
    (Trevor Robinson via eli)

    HDFS-1936. Layout version change from HDFS-1822 causes upgrade failure.
    (suresh)

    HDFS-2021. Update numBytesAcked before sending the ack in PacketResponder.
    (John George via szetszwo)

    HDFS-2020. Fix TestDFSUpgradeFromImage by removing the use of DataNode
    as a singleton. (suresh via todd)

    HDFS-2022. ant binary should build libhdfs. (Eric Yang via eli)

    HDFS-2014. Change HDFS scripts to work in developer enviroment post
    RPM packaging changes. (Eric Yang via suresh)

    HDFS-1995. Federation: Minor bug fixes and modification cluster web UI.
    (Tanping Wang via suresh)

    HDFS-1907. Fix position read for reading still-being-written file in
    DFSInputStream.  (John George via szetszwo)

    HDFS-1923. In TestFiDataTransferProtocol2, reduce random sleep time period
    and increase the number of datanodes.  (szetszwo)

    HDFS-1149. Lease reassignment should be persisted to the edit log.
    (Aaron T. Myers via todd)

    HDFS-1998. Federation: Make refresh-namenodes.sh refresh all the
    namenode. (Tanping Wang via suresh)

    HDFS-2041. OP_CONCAT_DELETE doesn't properly restore modification time
    of the concatenated file when edit logs are replayed. (todd)

    HDFS-2063. libhdfs test is broken. (Eric Yang via eli)

    HDFS-2067. Bump DATA_TRANSFER_VERSION constant in trunk after introduction
    of protocol buffers in the protocol. (szetszwo via todd)

    HDFS-2069. Incorrect default trash interval value in the docs.
    (Harsh J Chouraria via eli)

    HDFS-1942. Datanode must exist when all the block pool service threads
    exit. (Bharath Mundlapudi via suresh)

    HDFS-1656. Fixes an issue to do with fetching of delegation tokens in
    HftpFileSystem. Contributed by Kan Zhang.

    HDFS-1692. In secure mode, Datanode process doesn't exit when disks 
    fail. (Bharath Mundlapudi via suresh)

    HDFS-1734. 'Chunk size to view' option is not working in Name Node UI.
    (Uma Maheswara Rao G via jitendra)

    HDFS-2086. If the include hosts list contains host names, after restarting
    namenode, data nodes registration is denied.  Contributed by Tanping Wang.

    HDFS-2082. SecondaryNameNode web interface doesn't show the right info. (atm)

    HDFS-1321. If service port and main port are the same, there is no clear
    log message explaining the issue. (Jim Plush via atm)

    HDFS-1381. HDFS javadocs hard-code references to dfs.namenode.name.dir and
    dfs.datanode.data.dir parameters (Jim Plush via atm)

    HDFS-2053. Bug in INodeDirectory#computeContentSummary warning.
    (Michael Noll via eli)

    HDFS-1990. Fix resource leaks in BlockReceiver.close().  (Uma Maheswara
    Rao G via szetszwo)

    HDFS-2034. Length in DFSInputStream.getBlockRange(..) becomes -ve when
    reading only from a currently being written block. (John George via
    szetszwo)

    HDFS-2132. Potential resource leak in EditLogFileOutputStream.close. (atm)

    HDFS-2120. on reconnect, DN can connect to NN even with different source
    versions. (John George via atm)

    HDFS-2152. TestWriteConfigurationToDFS causing the random failures. (Uma
    Maheswara Rao G via atm)

    HDFS-2114. re-commission of a decommissioned node does not delete 
    excess replicas. (John George via mattf)

    HDFS-1776. Bug in Concat code. (Bharath Mundlapudi via Dmytro Molkov)

    HDFS-2196. Make ant build system work with hadoop-common JAR generated
    by Maven. (Alejandro Abdelnur via tomwhite)

    HDFS-2245. Fix a NullPointerException in BlockManager.chooseTarget(..).
    (szetszwo)

    HDFS-2229. Fix a deadlock in namenode by enforcing lock acquisition
    ordering.  (szetszwo)

    HDFS-2235. Encode servlet paths. (eli)

    HDFS-2186. DN volume failures on startup are not counted. (eli)

    HDFS-2240. Fix a deadlock in LeaseRenewer by enforcing lock acquisition
    ordering.  (szetszwo)

    HDFS-73. DFSOutputStream does not close all the sockets.
    (Uma Maheswara Rao G via eli)

    HDFS-1257. Fix a race condition on BlockManager.recentInvalidateSets.
    (Eric Payne via szetszwo)

    HDFS-2267. DataXceiver thread name incorrect while waiting on op during
    keepalive. (todd)

    HDFS-1480. All replicas of a block can end up on the same rack when
    some datanodes are decommissioning. (todd)

    HDFS-2286. DataXceiverServer logs AsynchronousCloseException at shutdown
    (todd)

    HDFS-2289. Ensure jsvc is bundled with the HDFS distribution artifact.
    (Alejandro Abdelnur via acmurthy)

    HDFS-2323. start-dfs.sh script fails for tarball install (tomwhite)

    HDFS-2412. Add backwards-compatibility layer for renamed FSConstants
               class (todd)

    HDFS-2414. Fix TestDFSRollback to avoid spurious failures. (todd)

    HDFS-2422. The NN should tolerate the same number of low-resource volumes
               as failed volumes (atm)

    HDFS-2467. HftpFileSystem uses incorrect compare for finding delegation
    tokens. (omalley)

    HDFS-2331. Fix WebHdfsFileSystem compilation problems for a bug in JDK
    version < 1.6.0_26.  (Abhijit Suresh Shingate via szetszwo)

    HDFS-2333. Change DFSOutputStream back to package private, otherwise,
    there are two SC_START_IN_CTOR findbugs warnings.  (szetszwo)

    HDFS-2366. Initialize WebHdfsFileSystem.ugi in object construction.
    (szetszwo)

    HDFS-2361. hftp is broken, fixed username checks in JspHelper. (jitendra)

    HDFS-2403. NamenodeWebHdfsMethods.generateDelegationToken(..) does not use
    the renewer parameter.  (szetszwo)

    HDFS-2409. _HOST in dfs.web.authentication.kerberos.principal. (jitendra)

    HDFS-2404. WebHDFS liststatus json response is not correct. (suresh)

    HDFS-2441. Remove the Content-Type set by HttpServer.QuotingInputFilter in
    WebHDFS responses.  (szetszwo)

    HDFS-2428. Convert com.sun.jersey.api.ParamException$QueryParamException
    to IllegalArgumentException and response it as http BAD_REQUEST in WebHDFS.
    (szetszwo)

    HDFS-2424. Added a root element "HdfsFileStatuses" for the response
    of WebHDFS listStatus.  (szetszwo)

    MAPREDUCE-2764. Fix renewal of dfs delegation tokens. (Owen via jitendra)

    HDFS-2439. Fix NullPointerException in WebHDFS when opening a non-existing
    file or creating a file without specifying the replication parameter.
    (szetszwo)

    HDFS-2453. Fix http response code for partial content in WebHDFS, added
    getDefaultBlockSize() and getDefaultReplication() in WebHdfsFileSystem
    and cleared content type in ExceptionHandler.  (szetszwo)

    HDFS-2411. The the auth to local mappings are not being respected, with 
    WebHDFS enabled. (jitendra)

    HDFS-2494. Close the streams and DFSClient in DatanodeWebHdfsMethods.
    (Uma Maheswara Rao G via szetszwo)

    HDFS-2298. Fix TestDfsOverAvroRpc by changing ClientProtocol to
    not include multiple methods of the same name. (cutting)

    HDFS-2432. WebHDFS: response FORBIDDEN when setReplication on non-files;
    clear umask before creating a flie; throw IllegalArgumentException if
    setOwner with both owner and group empty; throw FileNotFoundException if
    getFileStatus on non-existing files; fix bugs in getBlockLocations; and
    changed getFileChecksum json response root to "FileChecksum".  (szetszwo)

    HDFS-2065. Add null checks in DFSClient.getFileChecksum(..).  (Uma
    Maheswara Rao G via szetszwo)

    HDFS-2416. distcp with a WebHDFS uri on a secure cluster fails. (jitendra)

    HDFS-2527. WebHDFS: remove the use of "Range" header in Open; use ugi
    username if renewer parameter is null in GetDelegationToken; response OK
    when setting replication for non-files; rename GETFILEBLOCKLOCATIONS to
    GET_BLOCK_LOCATIONS and state that it is a private unstable API; replace
    isDirectory and isSymlink with enum {FILE, DIRECTORY, SYMLINK} in
    HdfsFileStatus JSON object.  (szetszwo)

    HDFS-2528. WebHDFS: set delegation kind to WEBHDFS and add a HDFS token
    when http requests are redirected to datanode.  (szetszwo)

    HDFS-2540. WebHDFS: change "Expect: 100-continue" to two-step write; change 
    "HdfsFileStatus" and "localName" respectively to "FileStatus" and
    "pathSuffix" in JSON response.  (szetszwo)

  BREAKDOWN OF HDFS-1073 SUBTASKS

    HDFS-1521. Persist transaction ID on disk between NN restarts.
               (Ivan Kelly and Todd Lipcon via todd)
    HDFS-1538. Refactor more startup and image loading code out of FSImage.
               (todd)
    HDFS-1729. Add code to detect valid length of an edits file. (todd)
    HDFS-1793. Add code to inspect a storage directory with txid-based
               filenames (todd)
    HDFS-1794. Add code to list which edit logs are available on a remote NN
               (todd)
    HDFS-1858. Add state management variables to FSEditLog (Ivan Kelly and Todd
               Lipcon via todd)
    HDFS-1859. Add some convenience functions to iterate over edit log streams
               (Ivan Kelly and Todd Lipcon via todd)
    HDFS-1894. Add constants for LAYOUT_VERSIONs in edits log branch (todd)
    HDFS-1892. Fix EditLogFileInputStream.getValidLength to be aware of
               OP_INVALID filler (todd)
    HDFS-1799. Refactor log rolling and filename management out of FSEditLog
               (Ivan Kelly and Todd Lipcon via todd)
    HDFS-1801. Remove use of timestamps to identify checkpoints and logs (todd)
    HDFS-1930. TestDFSUpgrade failing in HDFS-1073 branch (todd)
    HDFS-1800. Extend image checksumming to function with multiple fsimage
               files per directory. (todd)
    HDFS-1725. Set storage directories only at FSImage construction (Ivan Kelly
               via todd)
    HDFS-1926. Remove references to StorageDirectory from JournalManager
               interface (Ivan Kelly via todd)
    HDFS-1893. Change edit logs and images to be named based on txid (todd)
    HDFS-1985. Clean up image transfer servlet (todd)
    HDFS-1984. Enable multiple secondary namenodes to run simultaneously (todd)
    HDFS-1987. Re-enable TestCheckpoint.testSecondaryImageDownload which was
               not running previously. (todd)
    HDFS-1993. TestCheckpoint needs to clean up between cases (todd)
    HDFS-1992. Remove vestiges of NNStorageListener. (todd)
    HDFS-1991. Some refactoring of Secondary NameNode to be able to share more
               code with the BackupNode or CheckpointNode. (todd)
    HDFS-1994. Fix race conditions when running two rapidly checkpointing
               Secondary NameNodes. (todd)
    HDFS-2001. Remove use of previous.checkpoint and lastcheckpoint.tmp
               directories (todd)
    HDFS-2015. Remove checkpointTxId from VERSION file. (todd)
    HDFS-2016. Add infrastructure to remove or archive old and unneeded storage
               files within the name directories. (todd)
    HDFS-2047. Improve TestNamespace and TestEditLog in HDFS-1073 branch.
               (todd)
    HDFS-2048. Add upgrade tests and fix upgrade from 0.22 with corrupt image.
               (todd)
    HDFS-2027. Image inspector should return finalized logs before unfinalized
               logs. (todd)
    HDFS-2074. Determine edit log validity by truly reading and validating
               transactions. (todd)
    HDFS-2085. Finalize in-progress edit logs at startup. (todd)
    HDFS-2026. SecondaryNameNode should properly handle the case where the
               NameNode is reformatted. (todd)
    HDFS-2077. Address checkpoint upload when one of the storage dirs is failed
               (todd)
    HDFS-2078. NameNode should not clear directory when restoring removed
               storage. (todd)
    HDFS-2088. Move edits log archiving logic into FSEditLog/JournalManager
               (todd)
    HDFS-2093. Handle case where an entirely empty log is left during NN crash
               (todd)
    HDFS-2102. Zero-pad edits filename to make them lexically sortable. (Ivan
               Kelly via todd)
    HDFS-2010. Fix NameNode to exit if all edit streams become inaccessible.
               (atm via todd)
    HDFS-2123. Checkpoint interval should be based on txn count, not size.
               (todd)
    HDFS-1979. Fix backupnode for new edits/image layout. (todd)
    HDFS-2101. Fix remaining unit tests for new storage filenames. (todd)
    HDFS-2133. Address remaining TODOs and pre-merge cleanup on HDFS-1073
               branch.  (todd)
    HDFS-1780. Reduce need to rewrite FSImage on startup. (todd)
    HDFS-2104. Add a flag to the 2NN to format its checkpoint dirs on startup.
               (todd)
    HDFS-2135. Fix regression of HDFS-1955 in HDFS-1073 branch. (todd)
    HDFS-2160. Fix CreateEditsLog test tool in HDFS-1073 branch. (todd)
    HDFS-2168. Reenable TestEditLog.testFailedOpen and fix exposed bug. (todd)
    HDFS-2169. Clean up TestCheckpoint and remove TODOs (todd)
    HDFS-2170. Address remaining TODOs in HDFS-1073 branch. (todd)
    HDFS-2172. Address findbugs and javadoc warnings in HDFS-1073 branch. 
               (todd)

    HDFS-2445. Ensure failed tests exit with proper error code. (Jonathan 
    Eagles via acmurthy)

Release 0.22.1 - Unreleased

  INCOMPATIBLE CHANGES

  NEW FEATURES

  IMPROVEMENTS

  OPTIMIZATIONS

    HDFS-2718. Optimize OP_ADD in edits loading. (shv)

    HDFS-2886. CreateEditLogs should generate a realistic edit log. (shv)

  BUG FIXES

    HDFS-2877. If locking of a storage dir fails, it will remove the other
    NN's lock file on exit. (todd)

Release 0.22.0 - 2011-11-29

  INCOMPATIBLE CHANGES

    HDFS-1825. Remove thriftfs contrib. (nigel via eli)

  NEW FEATURES

    HDFS-992. Re-factor block access token implementation to conform to the 
    generic Token interface in Common (Kan Zhang and Jitendra Pandey via jghoman)

    HDFS-599. Allow NameNode to have a seprate port for service requests from
    client requests. (Dmytro Molkov via hairong)

    HDFS-1004. Update NN to support Kerberized SSL from HADOOP-6584. 
    (jghoman and Kan Zhang via jghoman)

    HDFS-1005. Fsck security. (borya and Kan Zhang via jghoman)

    HDFS-1006. getImage/putImage http requests should be https for the case 
    of security enabled. (borya and jghoman via jghoman)

    HDFS-1033. In secure clusters, NN and SNN should verify that the remote 
    principal during image and edits transfer. (jghoman)

    HDFS-1023. Allow http server to start as regular principal if https 
    principal not defined. (jghoman)

    HDFS-1150. Verify datanodes' identities to clients in secure clusters.
    (jghoman)

    HDFS-1330. Make RPCs to DataNodes timeout. (hairong)
    Added additional unit tests per HADOOP-6889. (John George via mattf)

    HDFS-202.  HDFS support of listLocatedStatus introduced in HADOOP-6870.
    HDFS piggyback block locations to each file status when listing a
    directory.  (hairong)

    HDFS-1361. Add -fileStatus operation to NNThroughputBenchmark. (shv)

    HDFS-1435. Provide an option to store fsimage compressed. (hairong)

    HDFS-903.  Support fsimage validation through MD5 checksum. (hairong)

    HDFS-1457. Provide an option to throttle image transmission between
    pimary and secondary NameNodes. (Yifei Lu and hairong via hairong)

    HDFS-1164. TestHdfsProxy is failing. (Todd Lipcon via cos)

    HDFS-811. Add metrics, failure reporting and additional tests for HDFS-457.
    (eli)

    HDFS-895. Allow hflush/sync to occur in parallel with new writes
    to the file. (Todd Lipcon via hairong)

    HDFS-528. Add ability for safemode to wait for a minimum number of 
    live datanodes (Todd Lipcon via eli)

    HDFS-1753. Resource Leak in StreamFile. (Uma Maheswara Rao G via eli)

  IMPROVEMENTS

    HDFS-1304. Add a new unit test for HftpFileSystem.open(..).  (szetszwo)

    HDFS-1096. fix for prev. commit. (boryas)

    HDFS-1096. allow dfsadmin/mradmin refresh of superuser proxy group
     mappings (boryas)

    HDFS-1146. Javadoc for getDelegationTokenSecretManager in FSNamesystem (jnp via boryas)

    HDFS-1132. Refactor TestFileStatus (Eli Collins via cos)

    HDFS-1163. normalize property names for JT/NN kerberos principal 
    names in configuration (from HADOOP 6633) (boryas)

    HDFS-1003. authorization checks for inter-server protocol 
    (based on HADOOP-6600) (boryas)

    HDFS-1061. Memory footprint optimization for INodeFile object. 
    (Bharath Mundlapudi via jghoman)

    HDFS-1079. Throw exceptions as specified by the AbstractFileSystem
    in HDFS implemenation and protocols. (suresh)

    HDFS-1112. Edit log buffer should not grow unfoundedly. (hairong)

    HDFS-1119. Introduce a GSet interface to BlocksMap.  (szetszwo)

    HDFS-1184. Replace tabs in code with spaces. (Jeff Ames via jghoman)

    HDFS-1185. Remove duplicate now() functions in DataNode, FSNamesysetm.
    (Jeff Ames via jghoman)

    HDFS-1183. Remove some duplicate code in NamenodeJspHelper.java.
    (Jeff Ames via jghoman)
    
    HDFS-1190.  Remove unused getNamenode() method from DataNode.
    (Jeff Ames via jghoman)

    HDFS-1110. Reuses objects for commonly used file names in namenode to
    reduce the heap usage. (suresh)

    HDFS-752. Add interfaces classification to to HDFS source code. (suresh)

    HDFS-947. An Hftp read request is redirected to a datanode that has 
    the most replicas of the blocks in the file. (Dmytro Molkov via dhruba)

    HDFS-1272. Fixes to take care of the changes in HADOOP-6845.
    (Jitendra Pandey via ddas)

    HDFS-1298 - Add support in HDFS for new statistics added in FileSystem
    to track the file system operations. (suresh)

    HDFS-1201. The HDFS component for HADOOP-6632. 
    (Kan Zhang & Jitendra Pandey via ddas)
    
    HDFS-1307 Add start time, end time and total time taken for FSCK to 
    FSCK report (suresh)

    HDFS-1302. The HDFS side of the changes corresponding to HADOOP-6861.
    (Jitendra Pandey & Owen O'Malley via ddas)
    
    HDFS-1315. Add fsck event to audit log and remove other audit log events 
    corresponding to FSCK listStatus and open calls. (suresh)

    HDFS-1178. The NameNode servlets should not use RPC to connect to the 
    NameNode. (Kan Zhang via jghoman)

    HDFS-1130. Adds dfs.cluster.administrator ACL configuration that can
    be used to control who can view the default hdfs servlets. (ddas)

    HDFS-1297. Fix some comments. (Jeff Ames via jghoman)

    HDFS-330.  Datanode Web UIs should provide robots.txt.
    (Allen Wittenauer via jghoman)

    HDFS-881.  Refactor DataNode Packet header into DataTransferProtocol.
    (Todd Lipcon via jghoman)

    HDFS-1036. docs for fetchdt

    HDFS-1318. Add JMX interface for read access to namenode and datanode
    web UI information. (Tanping Wang via suresh).

    HDFS-1356.  Provide information as to whether or not security is 
    enabled on web interface for NameNode (boryas)

    HDFS-1205. FSDatasetAsyncDiskService should name its threads.
    (Todd Lipcon via eli)

    HDFS-1111. Introduce getCorruptFileBlocks() for fsck. (Sriram Rao via shv)

    HDFS-1395. Add @Override to FSDataset methods that implement
    FSDatasetInterface methods. (suresh)

    HDFS-1383. Improve the error messages when using hftp://.  (szetszwo)

    HDFS-1093. Change the FSNamesystem lock to a read/write lock. (dhruba)

    HDFS-1407. Change DataTransferProtocol methods to use Block instead 
    of individual elements of Block. (suresh)

    HDFS-1417. Add @Override to SimulatedFSDataset methods that implement
    FSDatasetInterface methods. (suresh)

    HDFS-1426. Remove unused method BlockInfo#listCount. (hairong)

    HDFS-1472. Allow programmatic access to fsck output.
    (Ramkumar Vadali via dhruba)

    HADOOP-7007. Update the hudson-test-patch ant target to work with the
    latest test-patch.sh script (gkesavan)

    HDFS-1462. Refactor edit log loading to a separate class from edit log writing.
    (Todd Lipcon via eli)

    HDFS-1485. Fix typo in BlockPlacementPolicy. (Jingguo Yao via shv)

    HDFS-1035. Generate Eclipse's .classpath file from Ivy config. (nigel)

    HDFS-1408. Herriot NN and DN clients should vend statistics. (cos)

    HDFS-1491  Update Hdfs to match the change of methods from protected to public
    in AbstractFileSystem (Hadoop-6903) (sanjay)

    HDFS-1160. Improve some FSDataset warnings and comments. (eli)

    HDFS-556. Provide info on failed volumes in the web ui. (eli)

    HDFS-697. Enable asserts for tests by default. (eli)

    HDFS-1187. Modify fetchdt to allow renewing and canceling token.
    (Owen O'Malley and Kan Zhang via jghoman)

    HDFS-1387. Update HDFS permissions guide for security. (Todd Lipcon via eli)

    HDFS-455. Make NN and DN handle in a intuitive way comma-separated 
    configuration strings. (Michele Catasta via eli)

    HDFS-1071. savenamespace should write the fsimage to all configured 
    fs.name.dir in parallel (Dmytro Molkov via jghoman)
 
    HDFS-1055. Improve thread naming for DataXceivers. 
    (Todd Lipcon and Ramkumar Vadali via eli).

    HDFS-718. Configuration parameter to prevent accidental formatting of 
    HDFS filesystem. (Andrew Ryan via jghoman)

    HDFS-1500. TestOfflineImageViewer failing on trunk. (Todd Lipcon
    via hairong)

    HDFS-1483. DFSClient.getBlockLocations should indicate if corresponding
    blocks are corrupt. (Patrick Kling via hairong)

    HDFS-259. Remove intentionally corrupt 0.13 directory layout creation.
    (Todd Lipcon via eli)

    HDFS-1513. Fix a number of warnings. (eli)

    HDFS-1473. Refactor storage management into separate classes than fsimage
    file reading/writing. (Todd Lipcon via eli)

    HDFS-1582. Remove auto-generated native build files. (rvs via eli)

    HDFS-1456. Provide builder for constructing instances of MiniDFSCluster.
    (jghoman)

    HDFS-1861. Rename dfs.datanode.max.xcievers and bump its default value.
    (eli)

    HDFS-1052. HDFS Federation - Merge of umbrella jira changes from
    HDFS-1052 branch into trunk.

    HDFS-1835. DataNode should not depend on SHA1PRNG secure random generator
    to generate a storage ID. (John Carrino via todd)

    HDFS-1947. DFSClient should use mapreduce.task.attempt.id. (eli)

    HDFS-1957. Add documentation for HFTP. (Ari Rabkin via todd)

    HDFS-1454. Update the documentation to reflect that clients don't write
    blocks to local disk before copying to HDFS. (Harsh J Chouraria via todd)

    HDFS-1980. Move build/webapps deeper in the build directory heirarchy
    to aid eclipse users. (todd)

    HDFS-1619. Remove AC_TYPE* from the libhdfs. (Roman Shaposhnik via eli)

    HDFS-1948  Forward port 'hdfs-1520 lightweight namenode operation to
    trigger lease recovery' (stack)

    HDFS-1954. Improved corrupt files warning on NameNode web UI.
    (Patrick Hunt via shv)

    HDFS-1409. BackupNode registration throwing  
    UnsupportedActionException("register") instead of "journal".
    (Ching-Shen Chen via shv)

    HDFS-2054  BlockSender.sendChunk() prints ERROR for connection closures
    encountered during transferToFully() (Kihwal Lee via stack)

  OPTIMIZATIONS

    HDFS-1140. Speedup INode.getPathComponents. (Dmytro Molkov via shv)

    HDFS-1081. Performance regression in 
    DistributedFileSystem::getFileBlockLocations in secure systems (jghoman)

    HDFS-1114. Implement LightWeightGSet for BlocksMap in order to reduce
    NameNode memory footprint.  (szetszwo)

    HDFS-1320. Add LOG.isDebugEnabled() guard for each LOG.debug(..).
    (Erik Steffl via szetszwo)

    HDFS-1368. Add a block counter to DatanodeDescriptor. (hairong)

    HDFS-1434. Refactor Datanode#startDataNode method into smaller methods.
    (suresh)

    HDFS-941. The DFS client should cache and reuse open sockets to datanodes
    while performing reads. (bc Wong and Todd Lipcon via todd)

  BUG FIXES

    HDFS-1039. Adding test for  JspHelper.getUGI(jnp via boryas)

    HDFS-1019. Incorrect default values for delegation tokens in 
    hdfs-default.xml (jnp via boryas)

    HDFS-1039.  Service should be set in the token in JspHelper.getUGI(jnp via boryas)

    HDFS-1038. FIX. A test missed in a previous commit for this JIRA. (boryas)

    HDFS-1038. In nn_browsedfscontent.jsp fetch delegation token only 
    if security is enabled. (jnp via boryas)

    HDFS-1044. Cannot submit mapreduce job from secure client to 
    unsecure sever (boryas)

    HDFS-1021. specify correct server principal for RefreshAuthorizationPolicyProtocol 
    and RefreshUserToGroupMappingsProtocol protocols in DFSAdmin (for HADOOP-6612) (boryas)

    HDFS-970. fsync fsimage to disk before closing fsimage file.
    (Todd Lipcon via dhruba)

    HDFS-1027. Update copyright year to 2010. (Ravi Phulari via jghoman)

    HDFS-1080. SecondaryNameNode image transfer should use the defined http 
    address rather than local ip address. (jghoman)

    HDFS-1198. Resolving cross-realm principals. (Jitendra Pandey via jghoman)

    HDFS-1118. Fix socketleak on DFSClient. (Zheng Shao via dhruba)

    HDFS-1192. refreshSuperUserGroupsConfiguration should use server side 
    configuration for the refresh (for HADOOP-6815) (boryas)

    HDFS-1036. in DelegationTokenFetch dfs.getURI returns no port (boryas)

    HDFS-1017. browsedfs jsp should call JspHelper.getUGI rather 
    than using createRemoteUser() (jnp via boryas)

    HDFS-1250. Namenode should reject block reports and block received
    requests from dead datanodes (suresh)

    HDFS-1145. When NameNode is shutdown it does not try to exit
    safemode anymore. (dhruba)

    HDFS-1202. DataBlockScanner throws NPE when updated before 
    initialized. (Todd Lipcon via dhruba)

    HDFS-882. Datanode logs the hostname and port its listening on.
    (Steve Loughran via dhruba)

    HDFS-1238. ant eclipse-files has drifted again, (jghoman)

    HDFS-1045. In secure clusters, re-login is necessary for https 
    clients before opening connections. (jghoman)

    HDFS-1289. Datanode secure mode is broken. (Kan Zhang via jghoman)

    HDFS-1007. HFTP needs to be updated to use delegation tokens (boryas)

    HDFS-1085. HFTP read may fail silently on the client side if there is an
    exception on the server side.  (szetszwo)

    HDFS-1308. job conf key for the services name of DelegationToken for HFTP
    url is constructed incorrectly in HFTPFileSystem (boryas)

    HDFS-1319. Fix location of re-login for secondary namenode from HDFS-999. 
    (jghoman)

    HDFS-1317. Remove the FILEPATH_PATTERN from hdfsproxy.AuthorizationFilter.
    (Rohini Palaniswamy via szetszwo)

    HDFS-912. sed in build.xml on Solaris fails. (Allen Wittenauer via jghoman)

    HDFS-1296. using delegation token over hftp for long running 
    clients (boryas)

    HDFS-1334. open in HftpFileSystem does not add delegation tokens to the url.
    (Jitendra Pandey via jghoman)

    HDFS-1301.  TestHDFSProxy need to use server side conf for ProxyUser 
    stuff. (boryas)

    HDFS-1340. When security is turned off, there is a potential XSS attack. 
    This patch fixes it by removing delegationtoken string from the URL, 
    before returning a response to the client. (Jitendra Pandey via ddas)

    HDFS-1347.  TestDelegationToken uses mortbay.log for logging (boryas)

    HDFS-1157. Modifications introduced by HDFS-1150 are breaking aspect's
    bindings (cos)

    HDFS-1349. Remove empty java files. (Eli Collins)

    HDFS-1340. A null delegation token is appended to the url if security
    is disabled when browsing filesystem. (boryas)
    
    HDFS-1352. Fix jsvc.location. (Eli Collins via jghoman)

    HDFS-1284. TestBlockToken fails.  (Kan Zhang via jghoman)

    HDFS-1355. ant veryclean (clean-cache) doesn't clean enough. 
    (Luke Lu via jghoman)

    HDFS-1353. Remove most of getBlockLocation optimization. (jghoman)

    HDFS-1369. Invalid javadoc reference in FSDatasetMBean.java (Eli Collins)

    HDFS-829. hdfsJniHelper.c: #include <error.h> is not portable. 
    (Allen Wittenauer via jghoman)

    HDFS-1310. The ClientDatanodeProtocol proxy should be stopped in
    DFSInputStream.readBlockLength(..).  (sam rash via szetszwo)

    HDFS-1357. HFTP traffic served by DataNode shouldn't use service port 
    on NameNode. (Kan Zhang via jghoman)

    HDFS-1419. HDFS Federation: Three test cases need minor modification after 
    the new block id change (Tanping Wang via suresh)

    HDFS-96. HDFS supports blocks larger than 2 GB.
    (Patrick Kling via dhruba)

    HDFS-1433. Fix test failures - TestPread and TestFileLimit. (suresh)

    HDFS-1364. Makes long running HFTP-based applications do relogins
    if necessary. (Jitendra Pandey via ddas)

    HDFS-1399. Distinct minicluster services (e.g. NN and JT) overwrite each
    other's service policies.  (Aaron T. Myers via tomwhite)

    HDFS-1440. Fix TestComputeInvalidateWork failure. (suresh)
 
    HDFS-1498. FSDirectory#unprotectedConcat calls setModificationTime 
    on a file. (eli)

    HDFS-1625. Ignore disk space values in TestDataNodeMXBean.  (szetszwo)

    HDFS-1850. DN should transmit absolute failed volume count rather than 
    increments to the NN. (eli)

    HDFS-671. Documentation change for updated configuration keys. 
    (tomwhite via eli)

    HDFS-1544. Ivy resolve force mode should be turned off by default.
    (Luke Lu via tomwhite)

    HDFS-1615. seek() on closed DFS input stream throws NullPointerException
    (Scott Carey via todd)

    HDFS-1897. Documentation refers to removed option dfs.network.script
    (Andrew Whang via todd)

    HDFS-1621. Fix references to hadoop-common-${version} in build.xml
    (Jolly Chen via todd)

    HDFS-1505. saveNamespace appears to succeed even if all directories fail
    to save. (Aaron T. Myers via todd)

    HDFS-1921. saveNamespace can cause NN to be unable to come up on restart
    (Matt Foley via todd)

    HDFS-1925. SafeModeInfo should use the correct constant instead of a
    hard-coded value for its default. (Joey Echeverria via todd)

    HDFS-1575. Viewing block from web UI is broken. (Aaron T. Myers via todd)
    
    HDFS-1932. Ensure that HDFS configuration deprecations are set up in every
    spot that HDFS configurations are loaded. (Jolly Chen via todd)

    HDFS-1952. FSEditLog.open() appears to succeed even if all EDITS
    directories fail. (Andrew Wang via todd)

    HDFS-1965. IPCs done using block token-based tickets can't reuse
    connections (todd)

    HDFS-1978. All but first option in LIBHDFS_OPTS is ignored. (eli)

    HDFS-1964. Fix incorrect HTML unescaping in DatanodeJspHelper
    (Aaron T. Myers via todd)

    HDFS-1997. Image transfer process misreports client side exceptions.
    (todd via eli)

    HDFS-2000. Missing deprecation for io.bytes.per.checksum.
    (Aaron T. Myers vie eli)

    HDFS-977. DataNode.createInterDataNodeProtocolProxy() guards a log
    at the wrong level. (Harsh J Chouraria via todd)

    HDFS-1969. Running rollback on new-version namenode destroys the
    namespace. (todd)

    HDFS-2039. TestNameNodeMetrics uses a bad test root path, preventing it
    from running inside Eclipse. (todd)

    HDFS-988. saveNamespace race can corrupt the edits log. (eli)

    HDFS-2071. Use of isConnected() in DataXceiver is invalid. (Kihwal Lee
    via todd)

    HDFS-1981. NameNode does not saveNamespace() when editsNew is empty.
    (Uma Maheswara Rao G via shv)

    HDFS-2258. Reset lease limits to default values in TestLeaseRecovery2. (shv)

    HDFS-2232. Generalize regular expressions in TestHDFSCLI.
    (Plamen Jeliazkov via shv)

    HDFS-2290. Block with corrupt replica is not getting replicated.
    (Benoy Antony via shv)

    HDFS-2012. Balancer incorrectly treats nodes whose utilization equals
    avgUtilization. (Uma Maheswara Rao G via shv)

    HDFS-2491. TestBalancer can fail when datanode utilization and 
    avgUtilization is exactly same. (Uma Maheswara Rao G via shv)

    HDFS-2452. OutOfMemoryError in DataXceiverServer takes down the DataNode
    (Uma Maheswara Rao via cos)

    HDFS-2002. Incorrect computation of needed blocks in getTurnOffTip().
    (Plamen Jeliazkov via shv)

    HDFS-2573. TestFiDataXceiverServer is failing, not testing OOME (cos)

    HDFS-2514. Link resolution bug for intermediate symlinks with
    relative targets. (eli)

    HDFS-1786. Some cli test cases expect a "null" message
    (Uma Maheswara Rao G via todd)

    HDFS-1855. TestDatanodeBlockScanner.testBlockCorruptionRecoveryPolicy()
    part 2 fails in two different ways. (Matt Foley via eli)

    HDFS-2346. TestHost2NodesMap & TestReplicasMap will fail depending upon
    execution order of test methods (Laxman, Uma Maheswara Rao G via shv)

    HDFS-2287. TestParallelRead has a small off-by-one bug. (todd)

Release 0.21.1 - Unreleased

    HDFS-1466. TestFcHdfsSymlink relies on /tmp/test not existing. (eli)

    HDFS-874. TestHDFSFileContextMainOperations fails on weirdly 
    configured DNS hosts. (Todd Lipcon via eli)

    HDFS-1507. TestAbandonBlock should abandon a block. (eli)

    HDFS-1487. FSDirectory.removeBlock() should update diskspace count 
    of the block owner node (Zhong Wang via eli).

    HDFS-1467. Append pipeline never succeeds with more than one replica.
    (Todd Lipcon via eli)

    HDFS-1167. New property for local conf directory in system-test-hdfs.xml
    file. (Vinay Thota via cos)

    HDFS-1503. TestSaveNamespace fails. (Todd Lipcon via cos)

    HDFS-1524. Image loader should make sure to read every byte in image file.
    (hairong)

    HDFS-1523. TestLargeBlock is failing on trunk. (cos)

    HDFS-1502. TestBlockRecovery triggers NPE in assert. (hairong via cos)

    HDFS-1532. Exclude Findbugs warning in FSImageFormat$Saver. (Todd Lipcon
    via cos)

    HDFS-1527. SocketOutputStream.transferToFully fails for blocks >= 2GB on
    32 bit JVM. (Patrick Kling via cos)

    HDFS-1531. Clean up stack traces due to duplicate MXBean registration.
    (Todd Lipcon via cos)

    HDFS-613. TestBalancer and TestBlockTokenWithDFS fail Balancer assert.
    (Todd Lipcon via cos)

    HDFS-1511. 98 Release Audit warnings on trunk and branch-0.22.
    (jghoman)

    HDFS-1560. dfs.data.dir permissions should default to 700. 
    (Todd Lipcon via eli)

    HDFS-1550. NPE when listing a file with no location. (hairong)

    HDFS-1542. Add test for HADOOP-7082, a deadlock writing Configuration to
    HDFS. (todd)

    HDFS-1504. FSImageSaver should catch all exceptions, not just IOE. (todd)

    HDFS-884. DataNode throws IOException if all data directories are 
    unavailable. (Steve Loughran and shv)

    HDFS-1591. HDFS part of HADOOP-6642. (Chris Douglas, Po Cheung via shv)

    HDFS-900. Corrupt replicas are not processed correctly in block report (shv)

    HDFS-1529. Incorrect handling of interrupts in waitForAckedSeqno can cause
    deadlock (todd)

    HDFS-1597. Batched edit log syncs can reset synctxid and throw assertions
    (todd)

    HDFS-1602. Fix HADOOP-4885 for it is doesn't work as expected. (boryas)

    HDFS-1618. configure files that are generated as part of the released
    tarball need to have executable bit set (Roman Shaposhnik via cos)

    HDFS-981. test-contrib fails due to test-cactus failure (cos)

    HDFS-1001. DataXceiver and BlockReader disagree on when to send/recv
    CHECKSUM_OK. (bc Wong via eli)

    HDFS-1781. Fix the path for jsvc in bin/hdfs.  (John George via szetszwo)

    HDFS-1782. Fix an NPE in FSNamesystem.startFileInternal(..).
    (John George via szetszwo)

    HDFS-1821. Fix username resolution in NameNode.createSymlink(..) and
    FSDirectory.addSymlink(..).  (John George via szetszwo)

    HDFS-1806. TestBlockReport.blockReport_08() and _09() are timing-dependent
    and likely to fail on fast servers. (Matt Foley via eli)

    HDFS-1845. Symlink comes up as directory after namenode restart.
    (John George via eli)

    HDFS-1666. Disable failing hdfsproxy test TestAuthorizationFilter (todd)

    HDFS-1823. start-dfs.sh script fails if HADOOP_HOME is not set.
    (tomwhite via eli)

Release 0.21.1 - Unreleased

    HDFS-1411. Correct backup node startup command in hdfs user guide.
    (Ching-Shen Chen via shv)

  BUG FIXES

    HDFS-1363. Eliminate second synchronized sections in appendFile(). (shv)

    HDFS-1413. Fix broken links to HDFS Wiki. (shv)

    HDFS-1420. Clover build doesn't generate per-test coverage (cos)

    HDFS-1444. Test related code of build.xml is error-prone and needs to be
    re-aligned. (cos)

    HDFS-1343. Instrumented build should be concentrated in one build area (cos)

    HDFS-1452. ant compile-contrib is broken (cos)

    HDFS-1474. ant binary-system is broken (cos)

    HDFS-1292. Allow artifacts to be published to the staging Apache Nexus
    Maven Repository.  (Giridharan Kesavan via tomwhite)

    HDFS-1552. Remove java5 dependencies from build. (cos) 

    HDFS-1189. Quota counts missed between clear quota and set quota.
    (John George via szetszwo)

    HDFS-1665. Balancer misuses dfs.heartbeat.interval as milliseconds.
    (szetszwo)

    HDFS-1728. SecondaryNameNode.checkpointSize is in bytes but not in MB.
    (szetszwo)

    HDFS-1206. TestFiHFlush fails intermittently. (cos)

    HDFS-1548. Fault-injection tests are executed multiple times if invoked
    with run-test-hdfs-fault-inject target (cos)

    HDFS-1552. Remove java5 dependencies from build. (cos) 

    HDFS-996. JUnit tests should never depend on anything in conf (cos)

    HDFS-1612. Update HDFS design documentation for append, quota, symlink,
    block placement and checkpoint/backup node features.  (Joe Crobak
    via szetszwo)

    HDFS-1596. Replace fs.checkpoint.* with dfs.namenode.checkpoint.*
    in documentations.  (Harsh J Chouraria via szetszwo)

Release 0.21.0 - 2010-08-13

  INCOMPATIBLE CHANGES

    HDFS-538. Per the contract elucidated in HADOOP-6201, throw
    FileNotFoundException from FileSystem::listStatus rather than returning
    null. (Jakob Homan via cdouglas)

    HDFS-602. DistributedFileSystem mkdirs throws FileAlreadyExistsException
    instead of FileNotFoundException. (Boris Shkolnik via suresh)

    HDFS-544. Add a "rbw" subdir to DataNode data directory. (hairong)

    HDFS-576. Block report includes under-construction replicas. (shv)

    HDFS-636. SafeMode counts complete blocks only. (shv)

    HDFS-644. Lease recovery, concurrency support. (shv)

    HDFS-570. Get last block length from a data-node when opening a file
    being written to. (Tsz Wo (Nicholas), SZE via shv)

    HDFS-657. Remove unused legacy data-node protocol methods. (shv)

    HDFS-658. Block recovery for primary data-node. (shv)

    HDFS-660. Remove deprecated methods from InterDatanodeProtocol. (shv)

    HDFS-512. Block.equals() and compareTo() compare blocks based
    only on block Ids, ignoring generation stamps. (shv)

    HDFS-873. Configuration specifies data-node storage directories as URIs.
    (shv)

    HDFS-905. Use the new UserGroupInformation from HDFS-6299. 
    (jghoman via omalley)

    HDFS-984. Persistent delegation tokens. (Jitendra Pandey via shv)

    HDFS-1016. HDFS side change for HADOOP-6569. This jira changes the
    error message on the screen when cat a directory or a 
    non-existent file. (hairong)

  NEW FEATURES

    HDFS-1134. Large-scale Automated Framework. (cos)

    HDFS-436. Introduce AspectJ framework for HDFS code and tests.
    (Konstantin Boudnik via szetszwo)

    HDFS-447. Add LDAP lookup to hdfsproxy. (Zhiyong Zhang via cdouglas)

    HDFS-459. Introduce Job History Log Analyzer. (shv)

    HDFS-461. Tool to analyze file size distribution in HDFS. (shv)

    HDFS-492. Add two JSON JSP pages to the Namenode for providing corrupt
    blocks/replicas information.  (Bill Zeller via szetszwo)

    HDFS-578. Add support for new FileSystem method for clients to get server
    defaults. (Kan Zhang via suresh)

    HDFS-595. umask settings in configuration may now use octal or symbolic 
    instead of decimal. (Jakob Homan via suresh)

    HADOOP-6234. Updated hadoop-core and test jars to propagate new option 
    dfs.umaskmode in configuration. (Jakob Homan via suresh)

    HDFS-235. Add support for byte ranges in HftpFileSystem to serve
    range of bytes from a file. (Bill Zeller via suresh)

    HDFS-385. Add support for an experimental API that allows a module external
    to HDFS to specify how HDFS blocks should be placed. (dhruba)

    HADOOP-4952. Update hadoop-core and test jars to propagate new FileContext
    file system application interface. (Sanjay Radia via suresh).

    HDFS-567. Add block forensics contrib tool to print history of corrupt and
    missing blocks from the HDFS logs.
    (Bill Zeller, Jitendra Nath Pandey via suresh).

    HDFS-610. Support o.a.h.fs.FileContext.  (Sanjay Radia via szetszwo)

    HDFS-536. Support hflush at DFSClient. (hairong)

    HDFS-517. Introduce BlockInfoUnderConstruction to reflect block replica
    states while writing. (shv)

    HDFS-565. Introduce block committing logic during new block allocation
    and file close. (shv)

    HDFS-537. DataNode exposes a replica's meta info to BlockReceiver for the
    support of dfs writes/hflush. It also updates a replica's bytes received,
    bytes on disk, and bytes acked after receiving a packet. (hairong)

    HDFS-585. Datanode should serve up to visible length of a replica for read
    requests.  (szetszwo)

    HDFS-604. Block report processing for append. (shv)

    HDFS-619. Support replica recovery initialization in datanode for the new
    append design.  (szetszwo)

    HDFS-592. Allow clients to fetch a new generation stamp from NameNode for
    pipeline recovery. (hairong)

    HDFS-624. Support a new algorithm for pipeline recovery and pipeline setup
    for append. (hairong)

    HDFS-627. Support replica update in data-node.
    (Tsz Wo (Nicholas), SZE and Hairong Kuang via shv)

    HDFS-642. Support pipeline close and close error recovery. (hairong)

    HDFS-631. Rename configuration keys towards API standardization and
    backward compatibility. (Jitendra Nath Pandey via suresh)

    HDFS-669. Add unit tests framework (Mockito) (cos, Eli Collins)

    HDFS-731. Support new Syncable interface in HDFS. (hairong)

    HDFS-702. Add HDFS implementation of AbstractFileSystem. 
    (Sanjay Radio via suresh)

    HDFS-758. Add decommissioning status page to Namenode Web UI.
    (Jitendra Nath Pandey via suresh)

    HDFS-814. Add an api to get the visible length of a DFSDataInputStream.
    (szetszwo)

    HDFS-654. Add support new atomic rename functionality in HDFS for 
    supporting rename in FileContext. (suresh)

    HDFS-222. Support for concatenating of files into a single file
    without copying. (Boris Shkolnik via hairong)

    HDFS-933. Adds Delegation token based authentication in the NameNode.
    (Kan Zhang via ddas)

    HDFS-935. Adds a real user component in Delegation token.
    (Jitendra Nath Pandey via ddas)

    HDFS-245. Adds a symlink implementation to HDFS. This complements the new 
    symlink feature added in HADOOP-6421 (Eli Collins via Sanjay Radia)

    HDFS-1009. Support Kerberos authorization in HDFSProxy.  (Srikanth
    Sundarrajan via szetszwo)

    HDFS-1091. Implement listStatus that returns an iterator of FileStatus.
    (hairong)

  IMPROVEMENTS

    HDFS-381. Remove blocks from DataNode maps when corresponding file
    is deleted. (Suresh Srinivas via rangadi)

    HDFS-377. Separate codes which implement DataTransferProtocol.
    (szetszwo)

    HDFS-396. NameNode image and edits directories are specified as URIs.
    (Luca Telloli via rangadi)

    HDFS-444. Allow to change probability levels dynamically in the fault
    injection framework.  (Konstantin Boudnik via szetszwo)

    HDFS-352. Documentation for saveNamespace command. (Ravi Phulari via shv)

    HADOOP-6106. Updated hadoop-core and test jars from hudson trunk 
    build #12. (Giridharan Kesavan)

    HDFS-204. Add a new metrics FilesInGetListingOps to the Namenode.
    (Jitendra Nath Pandey via szetszwo)

    HDFS-278. HDFS Outputstream close does not hang forever. (dhruba)

    HDFS-443. Add a new metrics numExpiredHeartbeats to the Namenode.
    (Jitendra Nath Pandey via szetszwo)

    HDFS-475. Add new ant targets for fault injection jars and tests.
    (Konstantin Boudnik via szetszwo)

    HDFS-458. Create a new ant target, run-commit-test.  (Jakob Homan
    via szetszwo)

    HDFS-493. Change build.xml so that the fault-injected tests are executed
    only by the run-test-*-fault-inject targets.  (Konstantin Boudnik via
    szetszwo)

    HDFS-446. Improvements to Offline Image Viewer. (Jakob Homan via shv)

    HADOOP-6160. Fix releaseaudit target to run on specific directories.
    (gkesavan)

    HDFS-501. Use enum to define the constants in DataTransferProtocol.
    (szetszwo)

    HDFS-508. Factor out BlockInfo from BlocksMap. (shv)

    HDFS-510. Rename DatanodeBlockInfo to be ReplicaInfo.
    (Jakob Homan & Hairong Kuang via shv)

    HDFS-500. Deprecate NameNode methods deprecated in NameNodeProtocol.
    (Jakob Homan via shv)

    HDFS-514. Change DFSClient.namenode from public to private.  (Bill Zeller
    via szetszwo)

    HDFS-496. Use PureJavaCrc32 in HDFS.  (Todd Lipcon via szetszwo)

    HDFS-511. Remove redundant block searches in BlockManager. (shv)

    HDFS-504. Update the modification time of a file when the file 
    is closed. (Chun Zhang via dhruba)

    HDFS-498. Add development guide and documentation for the fault injection
    framework.  (Konstantin Boudnik via szetszwo)

    HDFS-524. Further DataTransferProtocol code refactoring.  (szetszwo)

    HDFS-529. Use BlockInfo instead of Block to avoid redundant block searches
    in BlockManager. (shv)

    HDFS-530. Refactor TestFileAppend* to remove code duplication.
    (Konstantin Boudnik via szetszwo)

    HDFS-451. Add fault injection tests for DataTransferProtocol.  (szetszwo)

    HDFS-409. Add more access token tests.  (Kan Zhang via szetszwo)

    HDFS-546. DatanodeDescriptor iterates blocks as BlockInfo. (shv)

    HDFS-457. Do not shutdown datanode if some, but not all, volumes fail.
    (Boris Shkolnik via szetszwo)

    HDFS-548. TestFsck takes nearly 10 minutes to run. (hairong)

    HDFS-539. Refactor fault injeciton pipeline test util for future reuse.
    (Konstantin Boudnik via szetszwo)

    HDFS-552. Change TestFiDataTransferProtocol to junit 4 and add a few new
    tests.  (szetszwo)

    HDFS-563. Simplify the codes in FSNamesystem.getBlockLocations(..).
    (szetszwo)

    HDFS-581. Introduce an iterator over blocks in the block report array.(shv)

    HDFS-549. Add a new target, run-with-fault-inject-testcaseonly, which
    allows an execution of non-FI tests in FI-enable environment.  (Konstantin
    Boudnik via szetszwo)

    HDFS-173. Namenode will not block until a large directory deletion 
    completes. It allows other operations when the deletion is in progress. 
    (suresh)

    HDFS-551. Create new functional test for a block report. (Konstantin
    Boudnik via hairong)

    HDFS-288. Redundant computation in hashCode() implementation.
    (szetszwo via tomwhite)

    HDFS-412. Hadoop JMX usage makes Nagios monitoring impossible.
    (Brian Bockelman via tomwhite)

    HDFS-472. Update hdfsproxy documentation. Adds a setup guide and design
    document. (Zhiyong Zhang via cdouglas)

    HDFS-617. Support non-recursive create().  (Kan Zhang via szetszwo)

    HDFS-618. Support non-recursive mkdir().  (Kan Zhang via szetszwo)

    HDFS-574. Split the documentation between the subprojects.
    (Corinne Chandel via omalley)

    HDFS-598. Eclipse launch task for HDFS. (Eli Collins via tomwhite)

    HDFS-641. Move all of the components that depend on map/reduce to 
    map/reduce. (omalley)

    HDFS-509. Redesign DataNode volumeMap to include all types of Replicas.
    (hairong)

    HDFS-562. Add a test for NameNode.getBlockLocations(..) to check read from
    un-closed file.  (szetszwo)

    HDFS-543. Break FSDatasetInterface#writToBlock() into writeToRemporary,
    writeToRBW, ad append. (hairong)

    HDFS-603. Add a new interface, Replica, which is going to replace the use
    of Block in datanode.  (szetszwo)

    HDFS-589. Change block write protocol to support pipeline recovery.
    (hairong)

    HDFS-652. Replace BlockInfo.isUnderConstruction() with isComplete() (shv)

    HDFS-648. Change some methods in AppendTestUtil to public.  (Konstantin
    Boudnik via szetszwo)

    HDFS-662. Unnecessary info message from DFSClient. (hairong)

    HDFS-518. Create new tests for Append's hflush. (Konstantin Boudnik
    via szetszwo)

    HDFS-688. Add configuration resources to DFSAdmin. (shv)

    HDFS-29. Validate the consistency of the lengths of replica and its file 
    in replica recovery.  (szetszwo)

    HDFS-680. Add new access method to a copy of a block's replica. (shv)

    HDFS-704. Unify build property names to facilitate cross-projects
    modifications (cos)

    HDFS-705. Create an adapter to access some of package-private methods of
    DataNode from tests (cos)

    HDFS-710. Add actions with constraints to the pipeline fault injection
    tests and change SleepAction to support uniform random sleeping over an
    interval.  (szetszwo)

    HDFS-713. Need to properly check the type of the test class from an aspect
    (cos)

    HDFS-716. Define a pointcut for pipeline close and add a few fault
    injection tests to simulate out of memory problem.  (szetszwo)

    HDFS-719. Add 6 fault injection tests for pipeline close to simulate slow
    datanodes and disk errors.  (szetszwo)

    HDFS-616. Create functional tests for new design of the block report. (cos)
    
    HDFS-584. Fail the fault-inject build if any advices are mis-bound. (cos)

    HDFS-730. Add 4 fault injection tests to simulate non-responsive datanode
    and out-of-memory problem for pipeline close ack.  (szetszwo)

    HDFS-728. Create a comprehensive functional test for append. (hairong)

    HDFS-736. commitBlockSynchronization() updates block GS and length 
    in-place. (shv)

    HADOOP-5107. Use Maven ant tasks to publish the subproject jars.
    (Giridharan Kesavan via omalley)

    HDFS-521. Create new tests for pipeline (cos)

    HDFS-764. Places the Block Access token implementation in hdfs project.
    (Kan Zhang via ddas)

    HDFS-787. Upgrade some libraries to be consistent with common and 
    mapreduce. (omalley)

    HDFS-519. Create new tests for lease recovery (cos)

    HDFS-804. New unit tests for concurrent lease recovery (cos)

    HDFS-813. Enable the append test in TestReadWhileWriting.  (szetszwo)

    HDFS-145. Cleanup inconsistent block length handling code in
    FSNameSystem#addStoredBlock. (hairong)

    HDFS-127. Reset failure count in DFSClient for each block acquiring
    operation.  (Igor Bolotin via szetszwo)

    HDFS-520. Create new tests for block recovery. (hairong)

    HDFS-1067. Create block recovery tests that handle errors. (hairong)

    HDFS-1107. Turn on append by default. (shv)

    HDFS-968. Use StringBuilder instead of StringBuffer for better
    performance. (Kay Kay via suresh)
    
    HDFS-703. Replace current fault injection implementation with one
    from (cos)

    HDFS-754. Reduce ivy console output to observable level (cos)

    HDFS-832. HDFS side of HADOOP-6222. (cos)

    HDFS-840. Change tests to use FileContext test helper introduced in
    HADOOP-6394. (Jitendra Nath Pandey via suresh)

    HDFS-685. Use the user-to-groups mapping service in the NameNode. 
    (boryas, acmurthy)

    HDFS-755. Read multiple checksum chunks at once in DFSInputStream.
    (Todd Lipcon via tomwhite)

    HDFS-786. Implement getContentSummary in HftpFileSystem.
    (Tsz Wo (Nicholas), SZE via cdouglas)

    HDFS-587. Add support for specifying queue name in mapreduce tests.
    (Erik Steffl via suresh)

    HDFS-902 Move contrib/raid to MapReduce. (Eli Collins via omalley)

    HDFS-800. The last block of a file under construction may change to the
    COMPLETE state in response to getAdditionalBlock or completeFileInternal.
    (hairong)

    HDFS-899. Delegation Token Implementation
     and corresponding changes in Namenode and DFS Api to issue, 
    renew and cancel delegation tokens. (jnp via boryas)

    HDFS-844. Log the filename when file locking fails. (tomwhite)

    HDFS-914. Refactor DFSOutputStream and DFSInputStream out of DFSClient.
    (Todd Lipcon via tomwhite)

    HDFS-949. Move DelegationToken into Common so that it can be used by
    MapReduce. (omalley)

    HDFS-930. Better error message for DATA_TRANSFER_VERSION mismatched.
    (Kay Kay via szetszwo)

    HDFS-986. Delegation token renewing and cancelling should provide
    meaningful exceptions when there are failures instead of returning 
    false. (omalley)

    HADOOP-6579. Upgrade the commons-codec library to 1.4. (omalley)

    HDFS-991. Allow authentication to the web ui via a delegation token. 
    (omalley)

    HDFS-994. Allow fetching of delegation token from NameNode for hftp.
    (Jakob Homan via acmurthy) 

    HDFS-998. Quote blocks streamed through jsps. (cdouglas)

    HDFS-729. NameNode API to list files that have missing blocks.
    (Rodrigo Schmidt via dhruba)

    HDFS-850. The WebUI display more details about namenode memory usage.
    (Dmytro Molkov via dhruba)

    HDFS-826. The DFSOutputStream has a API that returns the number of
    active datanode(s) in the current pipeline. (dhruba)

    HDFS-985. HDFS should issue multiple RPCs for listing a large
    directory. (hairong)

    HDFS-1043. NNThroughputBenchmark modifications to support benchmarking of
    server-side user group resolution. (shv)

    HDFS-892. Optionally use Avro reflection for Namenode RPC.  This
    is not a complete implementation yet, but rather a starting point.
    (cutting)
    
    HDFS-854. Datanode should scan devices in parallel to generate
    block report. (Dmytro Molkov via jhoman)

    HDFS-1032. fsck has an option to list corrupt files.
    (Andre Oriai via dhruba)

    HDFS-1024. SecondaryNameNode verifies size of fsimage and edits file.
    (Dmytro Molkov via dhruba)
    
    HDFS-1011. hdfsproxy: Improve log messages by restoring the previous
    thread name.  (Srikanth Sundarrajan via szetszwo)

    HDFS-997. Allow datanode storage directory permissions to be configurable.
    (Luke Lu via cdouglas)

    HDFS-1012. hdfsproxy: Support for fully qualified HDFS path in addition to
    simple unqualified path.  (Srikanth Sundarrajan via szetszwo)

    HDFS-993. Namenode should issue a delegation token only for kerberos 
    authenticated clients.(jnp via boryas)

    HDFS-1087. Modify audit log to use a StringBuilder rather than a Formatter.
    (cdouglas)

    HDFS-1083. Update TestHDFSCLI not to expect exception class name
    in error messages. (suresh)

    HDFS-1099. Add test for umask backward compatibility. (suresh)

    HDFS-1092. Use logging rather than System.err in MiniDFSCluster.
    (Kay Kay via jghoman)

    HDFS-1047. Install/deploy source jars to Maven repo. 
    (Patrick Angeles via jghoman)

    HDFS-666. Unit test for FsShell -text. (cdouglas via jghoman)

    HDFS-1054. Remove unnecessary sleep after failure in nextBlockOutputStream.
    (Todd Lipcon via jghoman)

    HDFS-921. Convert TestDFSClientRetries::testNotYetReplicatedErrors
    to Mockito. (jghoman)

    HDFS-1100. Override unwrapException in TestFcHdfsSymlink to test 
    symlink API conformance. (Eli Collins via suresh).

    HDFS-1089. Remove uses of FileContext#isFile, isDirectory, and exists.
    (Eli Collins via hairong)

    HDFS-1028. Efficient splitting of path components reduces the time
    to load in fsimage by 20%. (Dmytro Molkov via dhruba)

    HDFS-1109. HFTP supports filenames that contains the character "+".
    (Dmytro Molkov via dhruba)

    HDFS-853. The HDFS webUI displays the balanced-ness of the cluster.
    (Dmytro Molkov via dhruba)

    HDFS-1126. Change HDFS to depend on Hadoop 'common' artifacts instead
    of 'core'. (tomwhite)

    HDFS-995.  Replace usage of FileStatus#isDir().  (Eli Collins via
    tomwhite)

    HDFS-1161.  Make DN minimum valid volumes configurable.
    (Eli Collins via tomwhite)

    HDFS-1181. Move configuration and script files post split. (tomwhite)

    HDFS-1170.  Add more assertions to TestLargeDirectoryDelete.
    (Steve Loughran via tomwhite)

    HDFS-1199. Extract a subset of tests for smoke (DOA) validation. (cos)

    HDFS-1174. New properties for suspend and resume process. (Vinay Thota via
    cos)

    HDFS-1277. [Herriot] New property for multi user list. (Vinay Thota via
    cos)

    HDFS-806. Add new unit tests to the 10-mins 'run-commit-test' target (cos)

  OPTIMIZATIONS

    HDFS-946. NameNode should not return full path name when lisitng a
    diretory or getting the status of a file. (hairong)

  BUG FIXES

    HDFS-76. Better error message to users when commands fail because of 
    lack of quota. Allow quota to be set even if the limit is lower than
    current consumption. (Boris Shkolnik via rangadi)

    HADOOP-4687. HDFS is split from Hadoop Core. It is a subproject under 
    Hadoop (Owen O'Malley)

    HADOOP-6096. Fix Eclipse project and classpath files following project
    split. (tomwhite)

    HDFS-195. Handle expired tokens when write pipeline is reestablished.
    (Kan Zhang via rangadi)

    HDFS-181. Validate src path in FSNamesystem.getFileInfo(..).  (Todd
    Lipcon via szetszwo)

    HDFS-441. Remove TestFTPFileSystem.  (szetszwo)

    HDFS-440. Fix javadoc broken links in DFSClient.  (szetszwo)

    HDFS-480. Fix a typo in the jar name in build.xml.  
    (Konstantin Shvachko via gkesavan)

    HDFS-438. Check for NULL before invoking GenericArgumentParser in
    DataNode. (Raghu Angadi)

    HDFS-415. BlockReceiver hangs in case of certain runtime exceptions.
    (Konstantin Boudnik via rangadi)

    HDFS-462. loadFSImage should close edits file. (Jakob Homan via shv)

    HDFS-489. Update TestHDFSCLI for the -skipTrash option in rm. (Jakob Homan
    via szetszwo)

    HDFS-445. pread() does not pick up changes to block locations. 
    (Kan Zhang via rangadi) 

    HDFS-463. CreateEditLog utility broken after HDFS-396 (URI for
    FSImage). (Suresh Srinivas via rangadi)

    HDFS-484. Fix bin-package and package target to package jar files.
    (gkesavan)

    HDFS-490. Eliminate the deprecated warnings introduced by H-5438.
    (He Yongqiang via szetszwo)

    HDFS-119. Fix a bug in logSync(), which causes NameNode block forever.
    (Suresh Srinivas via shv)

    HDFS-534. Include avro in ivy.  (szetszwo)

    HDFS-532. Allow applications to know that a read request failed 
    because block is missing. (dhruba)

    HDFS-561. Fix write pipeline READ_TIMEOUT in DataTransferProtocol.
    (Kan Zhang via szetszwo)

    HDFS-553. BlockSender reports wrong failed position in ChecksumException.
    (hairong)

    HDFS-568. Set mapred.job.tracker.retire.jobs to false in
    src/test/mapred-site.xml for mapreduce tests to run.  (Amareshwari
    Sriramadasu via szetszwo)
 
    HDFS-15. All replicas end up on 1 rack. (Jitendra Nath Pandey via hairong)
 
    HDFS-586. TestBlocksWithNotEnoughRacks sometimes fails.
    (Jitendra Nath Pandey via hairong)

    HADOOP-6243. Fixed a NullPointerException in handling deprecated keys.
    (Sreekanth Ramakrishnan via yhemanth)

    HDFS-605. Do not run fault injection tests in the run-test-hdfs-with-mr
    target.  (Konstantin Boudnik via szetszwo)

    HDFS-606. Fix ConcurrentModificationException in invalidateCorruptReplicas()
    (shv)

    HDFS-601. TestBlockReport obtains data directories directly from
    MiniHDFSCluster. (Konstantin Boudnik via shv)

    HDFS-614. TestDatanodeBlockScanner obtains data directories directly from
    MiniHDFSCluster. (shv)

    HDFS-612. Remove the use of org.mortbay.log.Log in FSDataset.  (szetszwo)

    HDFS-622. checkMinReplication should count live nodes only. (shv)

    HDFS-629. Remove ReplicationTargetChooser.java along with fixing 
    import warnings generated by Eclipse. (dhruba)

    HDFS-637. DataNode sends a Success ack when block write fails. (hairong)

    HDFS-640. Fixed TestHDFSFileContextMainOperations.java build failure. (suresh)

    HDFS-547. TestHDFSFileSystemContract#testOutputStreamClosedTwice
    sometimes fails with CloseByInterruptException. (hairong)

    HDFS-588. Fix TestFiDataTransferProtocol and TestAppend2 failures. (shv)

    HDFS-550. DataNode restarts may introduce corrupt/duplicated/lost replicas
    when handling detached replicas. (hairong)

    HDFS-659. If the the last block is not complete, update its length with
    one of its replica's length stored in datanode.  (szetszwo)

    HDFS-649. Check null pointers for DataTransferTest.  (Konstantin Boudnik
    via szetszwo)

    HDFS-661. DataNode upgrade fails on non-existant current directory.
    (hairong)

    HDFS-597. Mofication introduced by HDFS-537 breakes an advice binding in
    FSDatasetAspects.  (Konstantin Boudnik via szetszwo)

    HDFS-665. TestFileAppend2 sometimes hangs. (hairong)

    HDFS-676. Fix NPE in FSDataset.updateReplicaUnderRecovery() (shv)

    HDFS-673. BlockReceiver#PacketResponder should not remove a packet from
    the ack queue before its ack is sent. (hairong)

    HDFS-682. Fix bugs in TestBlockUnderConstruction.  (szetszwo)

    HDFS-668. TestFileAppend3#TC7 sometimes hangs. (hairong)

    HDFS-679. Appending to a partial chunk incorrectly assumes the
    first packet fills up the partial chunk. (hairong)

    HDFS-722. Fix callCreateBlockWriteStream pointcut in FSDatasetAspects.
    (szetszwo)

    HDFS-690. TestAppend2#testComplexAppend failed on "Too many open files".
    (hairong)

    HDFS-725. Support the build error fix for HADOOP-6327.  (Sanjay Radia via
    szetszwo)

    HDFS-625. Fix NullPointerException thrown from ListPathServlet. (suresh)

    HDFS-735. TestReadWhileWriting has wrong line termination symbols (cos)

    HDFS-691. Fix an overflow error in DFSClient.DFSInputStream.available().
    (szetszwo)

    HDFS-733. TestBlockReport fails intermittently. (cos)

    HDFS-774. Intermittent race condition in TestFiPipelines (cos)

    HDFS-741. TestHFlush test doesn't seek() past previously written part of
    the file (cos, szetszwo)

    HDFS-706. Intermittent failures in TestFiHFlush (cos)
 
    HDFS-646. Fix test-patch failure by adding test-contrib ant target.
    (gkesavan)

    HDFS-791. Build is broken after HDFS-787 patch has been applied (cos)

    HDFS-792. TestHDFSCLI is failing. (Todd Lipcon via cos)

    HDFS-781. Namenode metrics PendingDeletionBlocks is not decremented.
    (Suresh)

    HDFS-192. Fix TestBackupNode failures. (shv)

    HDFS-797. TestHDFSCLI much slower after HDFS-265 merge. (Todd Lipcon via cos)

    HDFS-824. Stop lease checker in TestReadWhileWriting.  (szetszwo)

    HDFS-823. CheckPointer should use addInternalServlet for image-fetching
    servlet (jghoman)

    HDFS-456. Fix URI generation for windows file paths. (shv)

    HDFS-812. FSNamesystem#internalReleaseLease throws NullPointerException on
    a single-block file's lease recovery. (cos)

    HDFS-724. Pipeline hangs if one of the block receiver is not responsive.
    (hairong)

    HDFS-564. Adding pipeline tests 17-35. (hairong)

    HDFS-849. TestFiDataTransferProtocol2#pipeline_Fi_18 sometimes fails.
    (hairong)

    HDFS-762. Balancer causes Null Pointer Exception. 
    (Cristian Ivascu via dhruba)

    HDFS-868. Fix link to Hadoop Upgrade Wiki. (Chris A. Mattmann via shv)
    
    HDFS-880. TestNNLeaseRecovery fails on windows (cos, shv)

    HDFS-699. Primary datanode should compare replicas' on disk lengths.
    (hairong)

    HDFS-897. Fix a bug related to generation stamp comparison in 
    ReplicasMap. (suresh)

    HDFS-793. Data node should receive the whole packet ack message before it
    constructs and sends its own ack message for the packet. (hairong)

    HDFS-101. DFS write pipeline: DFSClient sometimes does not detect second
    datanode failure. (hairong)

    HDFS-822. Appends to already-finalized blocks can rename across volumes.
    (hairong)

    HDFS-1046. Fix Tomcat version in hdfsproxy/build.xml.  (Srikanth
    Sundarrajan via szetszwo)

    HDFS-1072. Fix TestReadWhileWriting failure. (Erik Steffl via shv)

    HDFS-913. Rename fault injection test TestRename.java to TestFiRename.java
    to include it in tests run by ant target run-test-hdfs-fault-inject.
    (suresh)
  
    HDFS-695. RaidNode should read in configuration from hdfs-site.xml.
    (dhruba)

    HDFS-726. Eclipse .classpath template has outdated jar files and is
    missing some new ones. (cos)

    HDFS-750. Fix build failure due to TestRename. (suresh)

    HDFS-712. Move libhdfs from mapreduce subproject to hdfs subproject.
    (Eli Collins via dhruba)

    HDFS-757. Enable Unit test for HDFS Raid. (dhruba)

    HDFS-611. Prevent DataNode heartbeat times from increasing even when
    the DataNode has many blocks to delete. (Zheng Shao via dhruba)

    HDFS-751. Fix TestCrcCorruption to pick up the correct datablocks to
    corrupt. (dhruba)
    
    HDFS-763. Fix slightly misleading report from DataBlockScanner 
    about corrupted scans. (dhruba)

    HDFS-727. bug setting block size hdfsOpenFile (Eli Collins via cos)

    HDFS-756. libhdfs unit tests do not run. (Eli Collins via cos)

    HDFS-783. libhdfs tests brakes code coverage runs with Clover (cos)

    HDFS-785. Add Apache license to several namenode unit tests. 
    (Ravi Phulari via jghoman)

    HDFS-802. Update Eclipse configuration to match changes to Ivy
    configuration (Edwin Chan via cos)

    HDFS-423. Unbreak FUSE build and fuse_dfs_wrapper.sh (Eli Collins via cos)

    HDFS-825. Build fails to pull latest hadoop-core-* artifacts (cos)

    HDFS-94. The Heap Size printed in the NameNode WebUI is accurate.
    (Dmytro Molkov via dhruba)

    HDFS-767. An improved retry policy when the DFSClient is unable to fetch a
    block from the datanode.  (Ning Zhang via dhruba)

    HDFS-775. FSDataset calls getCapacity() twice. (stevel)
    
    HDFS-885. Datanode toString() NPEs on null dnRegistration. (stevel)

    HDFS-877. Client-driven block verification not functioning. (Todd
    Lipcon via hairong)

    HDFS-630. In DFSOutputStream.nextBlockOutputStream(), the client can
    exclude specific datanodes when locating the next block.
    (Cosmin Lehene via Stack)

    HDFS-922. Remove unnecessary semicolon added by HDFS-877 that causes
    problems for Eclipse compilation. (jghoman)

    HDFS-927  DFSInputStream retries too many times for new block locations
    (Todd Lipcon via Stack)

    HDFS-938. Replace calls to UGI.getUserName() with UGI.getShortUserName()
    (jghoman)

    HDFS-894. DatanodeID.ipcPort is not updated when existing node 
    re-registers. (Todd Lipcon via tomwhite)

    HDFS-965. Split TestDelegationToken in to two parts and fix configuration
    to allow proxy users in the test. (Jitendra Pandey via omalley)

    HDFS-999. Secondary namenode should login using kerberos if security is 
    configured (boryas)

    HDFS-856. Hardcoded replication level for new files in fuse-dfs.
    (Brian Bockelman via tomwhite)

    HDFS-857. Incorrect type for fuse-dfs capacity can cause "df" to return
    negative values on 32-bit machines. (Brian Bockelman via tomwhite)

    HDFS-858. Incorrect return codes for fuse-dfs. (Brian Bockelman via
    tomwhite)

    HDFS-859. fuse-dfs utime behavior causes issues with tar.
    (Brian Bockelman via tomwhite)

    HDFS-861. fuse-dfs does not support O_RDWR. (Brian Bockelman via tomwhite)

    HDFS-961. dfs_readdir incorrectly parses paths. (Eli Collins via tomwhite)

    HDFS-1015. Fix intermittent failure in TestSecurityTokenEditLog.
    (Jitendra Nath Pandey via suresh)

    HDFS-939. libhdfs test is broken. (Eli Collins via tomwhite)
    
    HDFS-1074. hdfsproxy: Fix bugs in TestProxyUtil.  (Srikanth Sundarrajan
    via szetszwo)

    HDFS-481. hdfsproxy: Bug Fixes + HdfsProxy to use proxy user to
    impresonate the real user.  (Srikanth Sundarrajan via szetszwo)

    HDFS-482. Move HsftpFileSystem's ssl.client.do.not.authenticate.server
    configuration setting to ssl-client.xml.  (Srikanth Sundarrajan via
    szetszwo)

    HDFS-1010. hdfsproxy: Retrieve groups from UnixUserGroupInformation
    instead of LdapEntry.  (Srikanth Sundarrajan via szetszwo)

    HDFS-466. hdfs_write infinite loop when dfs fails and cannot write
    files > 2 GB. (Pete Wyckoff via tomwhite)

    HDFS-651. HDFS Docs - fix listing of docs in the doc menu.
    (Corinne Chandel via tomwhite)

    HDFS-1014. Error in reading delegation tokens from edit logs.
    (Jitendra Nath Pandey via jhoman)

    HDFS-1088. Prevent renaming a symbolik link to its target.
    (Eli Collins via suresh)

    HDFS-966. NameNode does not recovers lease when it is in safemode.
    (dhruba)

    HDFS-833. Datanode shutdown should log problems with Storage.unlockAll()
    (Steve Loughran via dhruba)

    HDFS-1101. TestDiskError.testLocalDirs() fails. (cdouglas via jghoman)

    HDFS-1031. Enhance the webUi to list a few of the corrupted files in HDFS.
    (Andre Orian via dhruba)

    HDFS-1078. Create static and dynamic versions of libhdfs.
    (Sam Rash via dhruba)

    HDFS-1104. Fsck triggers full GC on NameNode. (hairong)

    HDFS-1141. Closing a file is successful only if the client still has a
    valid lease. (Todd Lipcon via dhruba)

    HDFS-1138. Prevent erroneous updation of modification time of a directory
    when fsimage loads. (Dmytro Molkov via dhruba)

    HDFS-1000. Updates libhdfs to the new API for UGI (ddas)

    HDFS-609. Create a file with the append flag does not work in HDFS.
    (tomwhite)

    HDFS-1255. Fix failing test-libhdfs.sh test. (tomwhite)

    HDFS-1256. libhdfs is missing from the tarball. (tomwhite)

    HDFS-1057. Concurrent readers hit ChecksumExceptions if following a
    writer to very end of file. (sam rash via hairong)

    HDFS-1212. Harmonize HDFS JAR library versions with Common. (tomwhite)

    HDFS-1159. clean-cache target removes wrong ivy cache (cos)

    HDFS-1193. -mvn-system-deploy target is broken which inturn fails the
    mvn-deploy task leading to unstable mapreduce build (Giridharan
    Kesavan via cos)

    HDFS-1299. 'compile-fault-inject' never should be called directly. (cos)

    HDFS-1311. Running tests with 'testcase' cause triple execution of the
    same test case (Cos)

    HDFS-1267. fuse-dfs does not compile. (Devaraj Das via tomwhite)

    HDFS-1598.  Directory listing on hftp:// does not show .*.crc files.
    (szetszwo)

    HDFS-1750. ListPathsServlet should not use HdfsFileStatus.getLocalName()
    to get file name since it may return an empty string.  (szetszwo)

Release 0.20.3 - Unreleased

  IMPROVEMENTS

  BUG FIXES

    HDFS-1041. DFSClient.getFileChecksum(..) should retry if connection to
    the first datanode fails.  (szetszwo)

    HDFS-909. Wait until edits syncing is finishes before purging edits.
    (Todd Lipcon via shv)

    HDFS-1258. Clearing namespace quota on "/" corrupts fs image.
    (Aaron T. Myers via szetszwo)

    HDFS-1406. TestCLI fails on Ubuntu with default /etc/hosts. (cos)

Release 0.20.203.0 - 2011-5-11

  IMPROVEMENTS

    HADOOP-7259. Contrib modules should include the build.properties from
    the enclosing hadoop directory. (omalley)

  BUG FIXES

    HDFS-132. Fix namenode to not report files deleted metrics for deletions
    done while replaying edits during startup. (suresh & shv)

    HDFS-955. New implementation of saveNamespace() to avoid loss of edits 
    when name-node fails during saving. (shv)

Release 0.20.2 - 2009-09-01

  IMPROVEMENTS

    HDFS-737. Add full path name of the file to the block information and 
    summary of total number of files, blocks, live and deadnodes to 
    metasave output. (Jitendra Nath Pandey via suresh)

    HDFS-919. Create test to validate the BlocksVerified metric (Gary Murry
    via cos)

    HDFS-907. Add tests for getBlockLocations and totalLoad metrics.
    (Ravi Phulari via cos)
    
  BUG FIXES

    HDFS-686. NullPointerException is thrown while merging edit log and image.
    (hairong)

    HDFS-677. Rename failure when both source and destination quota exceeds
    results in deletion of source. (suresh)

    HDFS-709. Fix TestDFSShell failure due to rename bug introduced by 
    HDFS-677. (suresh)

    HDFS-579. Fix DfsTask to follow the semantics of 0.19, regarding non-zero
    return values as failures. (Christian Kunz via cdouglas)

    HDFS-723. Fix deadlock in DFSClient#DFSOutputStream. (hairong)

    HDFS-596. Fix memory leak in hdfsFreeFileInfo() for libhdfs.
    (Zhang Bingjun via dhruba)

    HDFS-185. Disallow chown, chgrp, chmod, setQuota, and setSpaceQuota when
    name-node is in safemode. (Ravi Phulari via shv)

    HDFS-187. Initialize secondary namenode http address in TestStartup.
    (Todd Lipcon via szetszwo)

    HDFS-464. Fix memory leaks in libhdfs. (Christian Kunz via suresh)
    
    HDFS-1377. Quota bug for partial blocks allows quotas to be violated. (eli)

Release 0.20.1 - 2009-09-01

  IMPROVEMENTS

    HDFS-438. Improve help message for space quota command. (Raghu Angadi)

  BUG FIXES

    HDFS-167. Fix a bug in DFSClient that caused infinite retries on write.
    (Bill Zeller via szetszwo)

    HDFS-527. Remove/deprecate unnecessary DFSClient constructors.  (szetszwo)

    HDFS-525. The SimpleDateFormat object in ListPathsServlet is not thread
    safe. (Suresh Srinivas and cdouglas)

    HDFS-761. Fix failure to process rename operation from edits log due to 
    quota verification. (suresh)