# Hadoop 0.23.10 Release Notes

These release notes include new developer and user-facing incompatibilities, features, and major improvements.

## Changes since Hadoop 0.23.9

- YARN-1419. Minor bug reported by Jonathan Eagles and fixed by Jonathan Eagles (scheduler)
  TestFifoScheduler.testAppAttemptMetrics fails intermittently under jdk7

  > QueueMetrics holds its data in a static variable causing metrics to bleed over from test to test. clearQueueMetrics is to be called for tests that need to measure metrics correctly for a single test. jdk7 comes into play since tests are run out of order, and in the case make the metrics unreliable.

- YARN-1386. Critical bug reported by Jason Lowe and fixed by Jason Lowe (nodemanager)
  NodeManager mistakenly loses resources and relocalizes them

  > When a local resource that should already be present is requested again, the nodemanager checks to see if it still present. However the method it uses to check for presence is via File.exists() as the user of the nodemanager process. If the resource was a private resource localized for another user, it will be localized to a location that is not accessible by the nodemanager user. Therefore File.exists() returns false, the nodemanager mistakenly believes the resource is no longer available, and it proceeds to localize it over and over.

- YARN-1243. Major bug reported by Sanjay Upadhyay and fixed by Jason Lowe (capacityscheduler)
  ResourceManager: Error in handling event type NODE_UPDATE to the scheduler - NPE at SchedulerApp.java:411

  > 2013-09-26 03:25:02,262 [ResourceManager Event Processor] FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event type NODE_UPDATE to the scheduler java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApp.unreserve(SchedulerApp.java:411) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.unreserve(LeafQueue.java:1333) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1261) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignOffSwitchContainers(LeafQueue.java:1137) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:1092) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignReservedContainer(LeafQueue.java:887) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:788) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:594) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:656) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:80) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:340) at java.lang.Thread.run(Thread.java:722) Yarn Resource manager exits at this NPE

- YARN-1176. Critical bug reported by Thomas Graves and fixed by Jonathan Eagles (resourcemanager)
  RM web services ClusterMetricsInfo total nodes doesn't include unhealthy nodes

  > In the web services api for the cluster/metrics, the totalNodes reported doesn't include the unhealthy nodes. this.totalNodes = activeNodes + lostNodes + decommissionedNodes + rebootedNodes;

- YARN-1119. Major test reported by Robert Parker and fixed by Mit Desai (resourcemanager)
  Add ClusterMetrics checks to tho TestRMNodeTransitions tests

  > YARN-1101 identified an issue where UNHEALTHY nodes could double decrement the active nodes. We should add checks for RUNNING node transitions.

- YARN-1101. Major bug reported by Robert Parker and fixed by Robert Parker (resourcemanager)
  Active nodes can be decremented below 0

  > The issue is in RMNodeImpl where both RUNNING and UNHEALTHY states that transition to a deactive state (LOST, DECOMMISSIONED, REBOOTED) use the same DeactivateNodeTransition class. The DeactivateNodeTransition class naturally decrements the active node, however the in cases where the node has transition to UNHEALTHY the active count has already been decremented.

- YARN-1036. Major bug reported by Ravi Prakash and fixed by Ravi Prakash (nodemanager)
  Distributed Cache gives inconsistent result if cache files get deleted from task tracker

  > This is a JIRA to backport MAPREDUCE-4342. I had to open a new JIRA because that one had been closed.

- YARN-1031. Major bug reported by Jonathan Eagles and fixed by Jonathan Eagles
  JQuery UI components reference external css in branch-23

- YARN-985. Major improvement reported by Ravi Prakash and fixed by Ravi Prakash (nodemanager)
  Nodemanager should log where a resource was localized

  > When a resource is localized, we should log WHERE on the local disk it was localized. This helps in debugging afterwards (e.g. if the disk was to go bad).

- YARN-949. Major bug reported by Kihwal Lee and fixed by Kihwal Lee (nodemanager)
  Failed log aggregation can leave a file open.

  > If log aggregation fails on a node manager, the output file in hdfs can be left open.

- YARN-707. Blocker improvement reported by Bikas Saha and fixed by Jason Lowe
  Add user info in the YARN ClientToken

  > If user info is present in the client token then it can be used to do limited authz in the AM.

- YARN-573. Critical sub-task reported by Omkar Vinit Joshi and fixed by Omkar Vinit Joshi
  Shared data structures in Public Localizer and Private Localizer are not Thread safe.

  > PublicLocalizer 1) pending accessed by addResource (part of event handling) and run method (as a part of PublicLocalizer.run() ). PrivateLocalizer 1) pending accessed by addResource (part of event handling) and findNextResource (i.remove()). Also update method should be fixed. It too is sharing pending list.

- YARN-337. Major bug reported by Jason Lowe and fixed by Jason Lowe (resourcemanager)
  RM handles killed application tracking URL poorly

  > When the ResourceManager kills an application, it leaves the proxy URL redirecting to the original tracking URL for the application even though the ApplicationMaster is no longer there to service it. It should redirect it somewhere more useful, like the RM's web page for the application, where the user can find that the application was killed and links to the AM logs. In addition, sometimes the AM during teardown from the kill can attempt to unregister and provide an updated tracking URL, but unfortunately the RM has "forgotten" the AM due to the kill and refuses to process the unregistration. Instead it logs: {noformat} 2013-01-09 17:37:49,671 [IPC Server handler 2 on 8030] ERROR org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: AppAttemptId doesnt exist in cache appattempt_1357575694478_28614_000001 {noformat} It should go ahead and process the unregistration to update the tracking URL since the application offered it.

- YARN-155. Major bug reported by Thomas Graves and fixed by Thomas Graves
  TestAppManager intermittently fails with jdk7

  > Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.851 sec <<< FAILURE! testRMAppSubmit(org.apache.hadoop.yarn.server.resourcemanager.TestAppManager) Time elapsed: 0.017 sec <<< FAILURE! junit.framework.AssertionFailedError: app event type is wrong before expected:<KILL> but was:<APP_REJECTED> at junit.framework.Assert.fail(Assert.java:47) at junit.framework.Assert.failNotEquals(Assert.java:283) at junit.framework.Assert.assertEquals(Assert.java:64) at org.apache.hadoop.yarn.server.resourcemanager.TestAppManager.setupDispatcher(TestAppManager.java:329) at org.apache.hadoop.yarn.server.resourcemanager.TestAppManager.testRMAppSubmit(TestAppManager.java:354) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) I can reproduce the failure on jdk6 if I move the testRMAppSubmit to the bottom of the file, so my initial hunch is this is due to test order.

- MAPREDUCE-5640. Trivial improvement reported by Jason Lowe and fixed by Jason Lowe (test)
  Rename TestLineRecordReader in jobclient module

- MAPREDUCE-5610. Major test reported by Jonathan Eagles and fixed by Jonathan Eagles
  TestSleepJob fails in jdk7

- MAPREDUCE-5587. Major bug reported by Jonathan Eagles and fixed by Jonathan Eagles
  TestTextOutputFormat fails on JDK7

- MAPREDUCE-5586. Major bug reported by Jonathan Eagles and fixed by Jonathan Eagles
  TestCopyMapper#testCopyFailOnBlockSizeDifference fails when run from hadoop-tools/hadoop-distcp directory

- MAPREDUCE-5585. Major bug reported by Jonathan Eagles and fixed by Jonathan Eagles
  TestCopyCommitter#testNoCommitAction Fails on JDK7

- MAPREDUCE-5569. Major bug reported by Nathan Roberts and fixed by Nathan Roberts
  FloatSplitter is not generating correct splits

- MAPREDUCE-5561. Critical bug reported by Cindy Li and fixed by Karthik Kambatla
  org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl testcase failing on trunk

- MAPREDUCE-5543. Blocker bug reported by Jason Lowe and fixed by Jason Lowe (mrv2)
  In-memory map outputs can be leaked after shuffle completes in 0.23

- MAPREDUCE-5513. Major bug reported by Jason Lowe and fixed by Robert Parker
  ConcurrentModificationException in JobControl

- MAPREDUCE-5504. Major bug reported by Thomas Graves and fixed by Kousuke Saruta (client)
  mapred queue -info inconsistent with types

- MAPREDUCE-5475. Blocker bug reported by Jason Lowe and fixed by Jason Lowe (mr-am , mrv2)
  MRClientService does not verify ACLs properly

- MAPREDUCE-5468. Blocker bug reported by Yesha Vora and fixed by Vinod Kumar Vavilapalli
  AM recovery does not work for map only jobs

- MAPREDUCE-5440. Major bug reported by Robert Parker and fixed by Robert Parker (mrv2)
  TestCopyCommitter Fails on JDK7

- MAPREDUCE-5425. Major bug reported by Ashwin Shankar and fixed by Robert Parker (jobhistoryserver)
  Junit in TestJobHistoryServer failing in jdk 7

- MAPREDUCE-5419. Major bug reported by Robert Parker and fixed by Robert Parker (mrv2)
  TestSlive is getting FileNotFound Exception

- MAPREDUCE-5414. Major bug reported by Nemon Lou and fixed by Nemon Lou (test)
  TestTaskAttempt fails jdk7 with NullPointerException

- MAPREDUCE-5380. Major bug reported by Stephen Chu and fixed by Stephen Chu
  Invalid mapred command should return non-zero exit code

- MAPREDUCE-5373. Major bug reported by Chuan Liu and fixed by Jonathan Eagles
  TestFetchFailure.testFetchFailureMultipleReduces could fail intermittently

- MAPREDUCE-5317. Major bug reported by Ravi Prakash and fixed by Ravi Prakash (mrv2)
  Stale files left behind for failed jobs

- MAPREDUCE-5251. Major bug reported by Jason Lowe and fixed by Ashwin Shankar (mrv2)
  Reducer should not implicate map attempt if it has insufficient space to fetch map output

- MAPREDUCE-5001. Major bug reported by Brock Noland and fixed by Sandy Ryza
  LocalJobRunner has race condition resulting in job failures

- MAPREDUCE-4716. Major bug reported by Thomas Graves and fixed by Thomas Graves (jobhistoryserver)
  TestHsWebServicesJobsQuery.testJobsQueryStateInvalid fails with jdk7

- MAPREDUCE-4579. Major bug reported by Thomas Graves and fixed by Thomas Graves
  TestTaskAttempt fails jdk7

- MAPREDUCE-4571. Major bug reported by Thomas Graves and fixed by Thomas Graves (webapps)
  TestHsWebServicesJobs fails on jdk7

- MAPREDUCE-3193. Major bug reported by Ramgopal N and fixed by Devaraj K (mrv1 , mrv2)
  FileInputFormat doesn't read files recursively in the input path dir

- MAPREDUCE-1981. Major improvement reported by Hairong Kuang and fixed by Hairong Kuang (job submission)
  Improve getSplits performance by using listLocatedStatus

- HDFS-5557. Critical bug reported by Kihwal Lee and fixed by Kihwal Lee
  Write pipeline recovery for the last packet in the block may cause rejection of valid replicas

- HDFS-5526. Blocker bug reported by Tsz Wo (Nicholas), SZE and fixed by Kihwal Lee (datanode)
  Datanode cannot roll back to previous layout version

- HDFS-5438. Critical bug reported by Kihwal Lee and fixed by Kihwal Lee (namenode)
  Flaws in block report processing can cause data loss

- HDFS-5346. Major bug reported by Kihwal Lee and fixed by Ravi Prakash (namenode , performance)
  Avoid unnecessary call to getNumLiveDataNodes() for each block during IBR processing

- HDFS-5239. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (namenode)
  Allow FSNamesystem lock fairness to be configurable

- HDFS-5047. Major bug reported by Kihwal Lee and fixed by Robert Parker (namenode)
  Supress logging of full stack trace of quota and lease exceptions

- HDFS-5028. Major bug reported by zhaoyunjiong and fixed by zhaoyunjiong
  LeaseRenewer throw java.util.ConcurrentModificationException when timeout

- HDFS-5010. Major improvement reported by Kihwal Lee and fixed by Kihwal Lee (namenode , performance)
  Reduce the frequency of getCurrentUser() calls from namenode

- HDFS-4998. Major bug reported by Kihwal Lee and fixed by Kihwal Lee (test)
  TestUnderReplicatedBlocks fails intermittently

- HDFS-4995. Major bug reported by Kihwal Lee and fixed by Kihwal Lee (namenode)
  Make getContentSummary() less expensive

- HDFS-4994. Minor bug reported by Kihwal Lee and fixed by Robert Parker (namenode)
  Audit log getContentSummary() calls

- HDFS-4993. Major bug reported by Kihwal Lee and fixed by Robert Parker
  fsck can fail if a file is renamed or deleted

- HDFS-4984. Major bug reported by Kihwal Lee and fixed by Jing Zhao (namenode)
  Incorrect Quota counting in INodeFile

- HDFS-4972. Major bug reported by Kihwal Lee and fixed by Kihwal Lee (namenode)
  [branch-0.23] permission check and operation are done in a separate lock for getBlockLocations()

- HDFS-4943. Minor bug reported by Jerry He and fixed by Jerry He (webhdfs)
  WebHdfsFileSystem does not work when original file path has encoded chars

- HDFS-4482. Blocker bug reported by Uma Maheswara Rao G and fixed by Uma Maheswara Rao G (namenode)
  ReplicationMonitor thread can exit with NPE due to the race between delete and replication of same file.

- HDFS-4329. Major bug reported by Andy Isaacson and fixed by Cristina L. Abad (hdfs-client)
  DFSShell issues with directories with spaces in name

- HDFS-3970. Major bug reported by Vinay and fixed by Vinay (datanode)
  BlockPoolSliceStorage#doRollback(..) should use BlockPoolSliceStorage instead of DataStorage to read prev version file.

- HDFS-3020. Critical bug reported by Todd Lipcon and fixed by Todd Lipcon (namenode)
  Auto-logSync based on edit log buffer size broken

- HADOOP-10091. Blocker bug reported by Jason Lowe and fixed by Jason Lowe (fs)
  Job with a har archive as input fails on 0.23

- HADOOP-9956. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (ipc)
  RPC listener inefficiently assigns connections to readers

- HADOOP-9955. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (ipc)
  RPC idle connection closing is extremely inefficient

- HADOOP-9757. Major bug reported by Jason Lowe and fixed by Cristina L. Abad (fs)
  Har metadata cache can grow without limit

- HADOOP-9748. Critical sub-task reported by Daryn Sharp and fixed by Daryn Sharp (security)
  Reduce blocking on UGI.ensureInitialized

- HADOOP-9686. Major improvement reported by Jason Lowe and fixed by Jason Lowe (conf)
  Easy access to final parameters in Configuration

- HADOOP-9476. Major bug reported by Kihwal Lee and fixed by Robert Parker (security , test)
  Some test cases in TestUserGroupInformation fail if ran after testSetLoginUser.

- HADOOP-8704. Major improvement reported by Thomas Graves and fixed by Jonathan Eagles
  add request logging to jetty/httpserver