Hadoop 2.0.4-alpha Release Notes

• YARN-470. Major bug reported by Hitesh Shah and fixed by Siddharth Seth (nodemanager)
  Support a way to disable resource monitoring on the NodeManager
  

  Currently, the memory management monitor's check is disabled when the maxMem is set to -1. However, the maxMem is also sent to the RM when the NM registers with it ( to define the max limit of allocate-able resources ). We need an explicit flag to disable monitoring to avoid the problems caused by the overloading of the max memory value.

• YARN-449. Blocker bug reported by Siddharth Seth and fixed by
  HBase test failures when running against Hadoop 2
  

  Post YARN-429, unit tests for HBase continue to fail since the classpath for the MRAppMaster is not being set correctly. Reverting YARN-129 may fix this, but I'm not sure that's the correct solution. My guess is, as Alexandro pointed out in YARN-129, maven classloader magic is messing up java.class.path.

• YARN-443. Major improvement reported by Thomas Graves and fixed by Thomas Graves (nodemanager)
  allow OS scheduling priority of NM to be different than the containers it launches
  

  It would be nice if we could have the nodemanager run at a different OS scheduling priority than the containers so that you can still communicate with the nodemanager if the containers out of control. On linux we could launch the nodemanager at a higher priority, but then all the containers it launches would also be at that higher priority, so we need a way for the container executor to launch them at a lower priority. I'm not sure how this applies to windows if at all.

• YARN-429. Blocker bug reported by Siddharth Seth and fixed by Siddharth Seth (resourcemanager)
  capacity-scheduler config missing from yarn-test artifact
  

  MiniYARNCluster and MiniMRCluster are unusable by downstream projects with the 2.0.3-alpha release, since the capacity-scheduler configuration is missing from the test artifact. hadoop-yarn-server-tests-3.0.0-SNAPSHOT-tests.jar should include the default capacity-scheduler configuration. Also, this doesn't need to be part of the default classpath - and should be moved out of the top level directory in the dist package.

• MAPREDUCE-5117. Blocker bug reported by Roman Shaposhnik and fixed by Siddharth Seth (security)
  With security enabled HS delegation token renewer fails
  

• MAPREDUCE-5088. Blocker bug reported by Roman Shaposhnik and fixed by Daryn Sharp
  MR Client gets an renewer token exception while Oozie is submitting a job
  

• MAPREDUCE-5083. Major bug reported by Siddharth Seth and fixed by Siddharth Seth (mrv2)
  MiniMRCluster should use a random component when creating an actual cluster
  

  Committed to branch-2.0.4. Modified changes.txt in trunk, branch-2 and branch-2.0.4 accordingly.

• MAPREDUCE-5053. Major bug reported by Robert Parker and fixed by Robert Parker
  java.lang.InternalError from decompression codec cause reducer to fail
  

• MAPREDUCE-5023. Critical bug reported by Kendall Thrapp and fixed by Ravi Prakash (jobhistoryserver , webapps)
  History Server Web Services missing Job Counters
  

• MAPREDUCE-5006. Major bug reported by Alejandro Abdelnur and fixed by Sandy Ryza (contrib/streaming)
  streaming tests failing
  

• MAPREDUCE-4549. Blocker bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans (mrv2)
  Distributed cache conflicts breaks backwards compatability
  

• HDFS-4649. Blocker bug reported by Daryn Sharp and fixed by Daryn Sharp (namenode , security , webhdfs)
  Webhdfs cannot list large directories
  

• HDFS-4646. Minor bug reported by Jagane Sundar and fixed by (namenode)
  createNNProxyWithClientProtocol ignores configured timeout value
  

• HDFS-4581. Major bug reported by Rohit Kochar and fixed by Rohit Kochar (datanode)
  DataNode#checkDiskError should not be called on network errors
  

• HDFS-4577. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (webhdfs)
  Webhdfs operations should declare if authentication is required
  

• HDFS-4571. Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (webhdfs)
  WebHDFS should not set the service hostname on the server side
  

• HDFS-4567. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (webhdfs)
  Webhdfs does not need a token for token operations
  

• HDFS-4566. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (webhdfs)
  Webdhfs token cancelation should use authentication
  

• HDFS-4560. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (webhdfs)
  Webhdfs cannot use tokens obtained by another user
  

• HDFS-4548. Blocker sub-task reported by Daryn Sharp and fixed by Daryn Sharp
  Webhdfs doesn't renegotiate SPNEGO token
  

• HDFS-3344. Major bug reported by Tsz Wo (Nicholas), SZE and fixed by Kihwal Lee (namenode)
  Unreliable corrupt blocks counting in TestProcessCorruptBlocks
  

• HADOOP-9444. Blocker bug reported by Konstantin Boudnik and fixed by Roman Shaposhnik (conf)
  $var shell substitution in properties are not expanded in hadoop-policy.xml
  

• HADOOP-9406. Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (build)
  hadoop-client leaks dependency on JDK tools jar
  

• HADOOP-9405. Minor bug reported by Andrew Wang and fixed by Andrew Wang (test , tools)
  TestGridmixSummary#testExecutionSummarizer is broken
  

• HADOOP-9399. Minor bug reported by Todd Lipcon and fixed by Konstantin Boudnik (build)
  protoc maven plugin doesn't work on mvn 3.0.2
  

  Committed to 2.0.4-alpha branch

• HADOOP-9379. Trivial improvement reported by Arpit Gupta and fixed by Arpit Gupta
  capture the ulimit info after printing the log to the console
  

• HADOOP-9374. Major improvement reported by Daryn Sharp and fixed by Daryn Sharp (security)
  Add tokens from -tokenCacheFile into UGI
  

• HADOOP-9301. Blocker bug reported by Roman Shaposhnik and fixed by Alejandro Abdelnur (build)
  hadoop client servlet/jsp/jetty/tomcat JARs creating conflicts in Oozie & HttpFS
  

• HADOOP-9299. Blocker bug reported by Roman Shaposhnik and fixed by Daryn Sharp (security)
  kerberos name resolution is kicking in even when kerberos is not configured
  

Hadoop 2.0.3-alpha Release Notes

• YARN-372. Minor task reported by Siddharth Seth and fixed by Siddharth Seth
  Move InlineDispatcher from hadoop-yarn-server-resourcemanager to hadoop-yarn-common
  

  InlineDispatcher is a utility used in unit tests. Belongs in yarn-common instead of yarn-server-resource-manager.

• YARN-364. Major bug reported by Jason Lowe and fixed by Jason Lowe
  AggregatedLogDeletionService can take too long to delete logs
  

  AggregatedLogDeletionService uses the yarn.log-aggregation.retain-seconds property to determine which logs should be deleted, but it uses the same value to determine how often to check for old logs. This means logs could actually linger up to twice as long as configured.

• YARN-360. Critical bug reported by Daryn Sharp and fixed by Daryn Sharp
  Allow apps to concurrently register tokens for renewal
  

  {{DelegationTokenRenewer#addApplication}} has an unnecessary {{synchronized}} keyword. This serializes job submissions and can add unnecessary latency and/or hang all submissions if there are problems renewing the token.

• YARN-357. Major bug reported by Daryn Sharp and fixed by Daryn Sharp (resourcemanager)
  App submission should not be synchronized
  

  MAPREDUCE-2953 fixed a race condition with querying of app status by making {{RMClientService#submitApplication}} synchronously invoke {{RMAppManager#submitApplication}}. However, the {{synchronized}} keyword was also added to {{RMAppManager#submitApplication}} with the comment: bq. I made the submitApplication synchronized to keep it consistent with the other routines in RMAppManager although I do not believe it needs it since the rmapp datastructure is already a concurrentMap and I don't see anything else that would be an issue. It's been observed that app submission latency is being unnecessarily impacted.

• YARN-355. Blocker bug reported by Daryn Sharp and fixed by Daryn Sharp (resourcemanager)
  RM app submission jams under load
  

  The RM performs a loopback connection to itself to renew its own tokens. If app submissions consume all RPC handlers for {{ClientRMProtocol}}, then app submissions block because it cannot loopback to itself to do the renewal.

• YARN-354. Blocker bug reported by Liang Xie and fixed by Liang Xie
  WebAppProxyServer exits immediately after startup
  

  Please see HDFS-4426 for detail, i found the yarn WebAppProxyServer is broken by HADOOP-9181 as well, here's the hot fix, and i verified manually in our test cluster. I'm really applogized for bring about such trouble...

• YARN-343. Major bug reported by Thomas Graves and fixed by Xuan Gong (capacityscheduler)
  Capacity Scheduler maximum-capacity value -1 is invalid
  

  I tried to start the resource manager using the capacity scheduler with a particular queues maximum-capacity set to -1 which is supposed to disable it according to the docs but I got the following exception: java.lang.IllegalArgumentException: Illegal value of maximumCapacity -0.01 used in call to setMaxCapacity for queue foo at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.checkMaxCapacity(CSQueueUtils.java:31) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.setupQueueConfigs(LeafQueue.java:220) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.<init>(LeafQueue.java:191) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:310) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:325) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:232) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:202)

• YARN-336. Major bug reported by Sandy Ryza and fixed by Sandy Ryza (scheduler)
  Fair scheduler FIFO scheduling within a queue only allows 1 app at a time
  

  The fair scheduler allows apps to be scheduled in FIFO fashion within a queue. Currently, when this setting is turned on, the scheduler only allows one app to run at a time. While apps submitted earlier should get first priority for allocations, when there is space remaining, other apps should have a change to get at them.

• YARN-334. Critical bug reported by Thomas Graves and fixed by Thomas Graves
  Maven RAT plugin is not checking all source files
  

  yarn side of HADOOP-9097 Running 'mvn apache-rat:check' passes, but running RAT by hand (by downloading the JAR) produces some warnings for Java files, amongst others.

• YARN-331. Major improvement reported by Sandy Ryza and fixed by Sandy Ryza (scheduler)
  Fill in missing fair scheduler documentation
  

  In the fair scheduler documentation, a few config options are missing: locality.threshold.node locality.threshold.rack max.assign aclSubmitApps minSharePreemptionTimeout

• YARN-330. Major bug reported by Hitesh Shah and fixed by Sandy Ryza (nodemanager)
  Flakey test: TestNodeManagerShutdown#testKillContainersOnShutdown
  

  =Seems to be timing related as the container status RUNNING as returned by the ContainerManager does not really indicate that the container task has been launched. Sleep of 5 seconds is not reliable. Running org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerShutdown Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 9.353 sec <<< FAILURE! testKillContainersOnShutdown(org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerShutdown) Time elapsed: 9283 sec <<< FAILURE! junit.framework.AssertionFailedError: Did not find sigterm message at junit.framework.Assert.fail(Assert.java:47) at junit.framework.Assert.assertTrue(Assert.java:20) at org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerShutdown.testKillContainersOnShutdown(TestNodeManagerShutdown.java:162) Logs: 2013-01-09 14:13:08,401 INFO [AsyncDispatcher event handler] container.Container (ContainerImpl.java:handle(835)) - Container container_0_0000_01_000000 transitioned from NEW to LOCALIZING 2013-01-09 14:13:08,412 INFO [AsyncDispatcher event handler] localizer.LocalizedResource (LocalizedResource.java:handle(194)) - Resource file:hadoop-common/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerShutdown/tmpDir/scriptFile.sh transitioned from INIT to DOWNLOADING 2013-01-09 14:13:08,412 INFO [AsyncDispatcher event handler] localizer.ResourceLocalizationService (ResourceLocalizationService.java:handle(521)) - Created localizer for container_0_0000_01_000000 2013-01-09 14:13:08,589 INFO [LocalizerRunner for container_0_0000_01_000000] localizer.ResourceLocalizationService (ResourceLocalizationService.java:writeCredentials(895)) - Writing credentials to the nmPrivate file hadoop-common/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerShutdown/nm0/nmPrivate/container_0_0000_01_000000.tokens. Credentials list: 2013-01-09 14:13:08,628 INFO [LocalizerRunner for container_0_0000_01_000000] nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:createUserCacheDirs(373)) - Initializing user nobody 2013-01-09 14:13:08,709 INFO [main] containermanager.ContainerManagerImpl (ContainerManagerImpl.java:getContainerStatus(538)) - Returning container_id {, app_attempt_id {, application_id {, id: 0, cluster_timestamp: 0, }, attemptId: 1, }, }, state: C_RUNNING, diagnostics: "", exit_status: -1000, 2013-01-09 14:13:08,781 INFO [LocalizerRunner for container_0_0000_01_000000] nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:startLocalizer(99)) - Copying from hadoop-common/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerShutdown/nm0/nmPrivate/container_0_0000_01_000000.tokens to hadoop-common/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerShutdown/nm0/usercache/nobody/appcache/application_0_0000/container_0_0000_01_000000.tokens

• YARN-328. Major improvement reported by Suresh Srinivas and fixed by Suresh Srinivas (resourcemanager)
  Use token request messages defined in hadoop common
  

  YARN changes related to HADOOP-9192 to reuse the protobuf messages defined in common.

• YARN-325. Blocker bug reported by Jason Lowe and fixed by Arun C Murthy (capacityscheduler)
  RM CapacityScheduler can deadlock when getQueueInfo() is called and a container is completing
  

  If a client calls getQueueInfo on a parent queue (e.g.: the root queue) and containers are completing then the RM can deadlock. getQueueInfo() locks the ParentQueue and then calls the child queues' getQueueInfo() methods in turn. However when a container completes, it locks the LeafQueue then calls back into the ParentQueue. When the two mix, it's a recipe for deadlock. Stacktrace to follow.

• YARN-320. Blocker bug reported by Daryn Sharp and fixed by Daryn Sharp (resourcemanager)
  RM should always be able to renew its own tokens
  

  YARN-280 introduced fast-fail for job submissions with bad tokens. Unfortunately, other stack components like oozie and customers are acquiring RM tokens with a hardcoded dummy renewer value. These jobs would fail after 24 hours because the RM token couldn't be renewed, but fast-fail is failing them immediately. The RM should always be able to renew its own tokens submitted with a job. The renewer field may continue to specify an external user who can renew.

• YARN-319. Major bug reported by shenhong and fixed by shenhong (resourcemanager , scheduler)
  Submit a job to a queue that not allowed in fairScheduler, client will hold forever.
  

  RM use fairScheduler, when client submit a job to a queue, but the queue do not allow the user to submit job it, in this case, client will hold forever.

• YARN-315. Major improvement reported by Suresh Srinivas and fixed by Suresh Srinivas
  Use security token protobuf definition from hadoop common
  

  YARN part of HADOOP-9173.

• YARN-302. Major bug reported by Sandy Ryza and fixed by Sandy Ryza (resourcemanager , scheduler)
  Fair scheduler assignmultiple should default to false
  

  The MR1 default was false. When true, it results in overloading some machines with many tasks and underutilizing others.

• YARN-301. Major bug reported by shenhong and fixed by shenhong (resourcemanager , scheduler)
  Fair scheduler throws ConcurrentModificationException when iterating over app's priorities
  

  In my test cluster, fairscheduler appear to concurrentModificationException and RM crash, here is the message: 2012-12-30 17:14:17,171 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event type NODE_UPDATE to the scheduler java.util.ConcurrentModificationException at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100) at java.util.TreeMap$KeyIterator.next(TreeMap.java:1154) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AppSchedulable.assignContainer(AppSchedulable.java:297) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.assignContainer(FSLeafQueue.java:181) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.nodeUpdate(FairScheduler.java:780) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:842) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:98) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:340) at java.lang.Thread.run(Thread.java:662)

• YARN-300. Major bug reported by shenhong and fixed by Sandy Ryza (resourcemanager , scheduler)
  After YARN-271, fair scheduler can infinite loop and not schedule any application.
  

  After yarn-271, when yarn.scheduler.fair.max.assign<=0, when a node was been reserved, fairScheduler will infinite loop and not schedule any application.

• YARN-293. Critical bug reported by Devaraj K and fixed by Robert Joseph Evans (nodemanager)
  Node Manager leaks LocalizerRunner object for every Container
  

  Node Manager creates a new LocalizerRunner object for every container and puts in ResourceLocalizationService.LocalizerTracker.privLocalizers map but it never removes from the map.

• YARN-288. Major bug reported by Sandy Ryza and fixed by Sandy Ryza (resourcemanager , scheduler)
  Fair scheduler queue doesn't accept any jobs when ACLs are configured.
  

  If a queue is configured with an ACL for who can submit jobs, no jobs are allowed, even if a user on the list tries. This is caused by using the scheduler thinking the user is "yarn", because it calls UserGroupInformation.getCurrentUser() instead of UserGroupInformation.createRemoteUser() with the given user name.

• YARN-286. Major new feature reported by Tom White and fixed by Tom White (applications)
  Add a YARN ApplicationClassLoader
  

  Add a classloader that provides webapp-style class isolation for use by applications. This is the YARN part of MAPREDUCE-1700 (which was already developed in that JIRA).

• YARN-285. Major improvement reported by Derek Dagit and fixed by Derek Dagit
  RM should be able to provide a tracking link for apps that have already been purged
  

  As applications complete, the RM tracks their IDs in a completed list. This list is routinely truncated to limit the total number of application remembered by the RM. When a user clicks the History for a job, either the browser is redirected to the application's tracking link obtained from the stored application instance. But when the application has been purged from the RM, an error is displayed. In very busy clusters the rate at which applications complete can cause applications to be purged from the RM's internal list within hours, which breaks the proxy URLs users have saved for their jobs. We would like the RM to provide valid tracking links persist so that users are not frustrated by broken links.

• YARN-283. Major bug reported by Sandy Ryza and fixed by Sandy Ryza (scheduler)
  Fair scheduler fails to get queue info without root prefix
  

  If queue1 exists, and a client calls "mapred queue -info queue1", an exception is thrown. If they use root.queue1, it works correctly.

• YARN-282. Major bug reported by Sandy Ryza and fixed by Sandy Ryza
  Fair scheduler web UI double counts Apps Submitted
  

  Each app submitted is reported twice under "Apps Submitted"

• YARN-280. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (resourcemanager)
  RM does not reject app submission with invalid tokens
  

  The RM will launch an app with invalid tokens. The tasks will languish with failed connection retries, followed by task reattempts, followed by app reattempts.

• YARN-278. Major bug reported by Sandy Ryza and fixed by Sandy Ryza (resourcemanager , scheduler)
  Fair scheduler maxRunningApps config causes no apps to make progress
  

  This occurs because the scheduler erroneously chooses apps to offer resources to that are not runnable, then later decides they are not runnable, and doesn't try to give the resources to anyone else.

• YARN-277. Major improvement reported by Bikas Saha and fixed by Bikas Saha
  Use AMRMClient in DistributedShell to exemplify the approach
  

• YARN-272. Major bug reported by Sandy Ryza and fixed by Sandy Ryza (scheduler)
  Fair scheduler log messages try to print objects without overridden toString methods
  

  A lot of junk gets printed out like this: 2012-12-11 17:31:52,998 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSSchedulerApp: Application application_1355270529654_0003 reserved container org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl@324f0f97 on node host: c1416.hal.cloudera.com:46356 #containers=7 available=0 used=8192, currently has 4 at priority org.apache.hadoop.yarn.api.records.impl.pb.PriorityPBImpl@33; currentReservation 4096

• YARN-271. Major bug reported by Sandy Ryza and fixed by Sandy Ryza (resourcemanager , scheduler)
  Fair scheduler hits IllegalStateException trying to reserve different apps on same node
  

  After the fair scheduler reserves a container on a node, it doesn't check for reservations it just made when trying to make more reservations during the same heartbeat.

• YARN-267. Major bug reported by Sandy Ryza and fixed by Sandy Ryza (resourcemanager , scheduler)
  Fix fair scheduler web UI
  

  The fair scheduler web UI was broken by MAPREDUCE-4720. The queues area is not shown, and changes are required to still show the fair share inside the applications table.

• YARN-266. Critical bug reported by Ravi Prakash and fixed by Ravi Prakash (resourcemanager)
  RM and JHS Web UIs are blank because AppsBlock is not escaping string properly
  

  e.g. Job names with a line feed "\n" are causing a line feed in the JSON array being written out (since we are only using StringEscapeUtils.escapeHtml() ) and the Javascript parser complains that string quotes are unclosed. This

• YARN-264. Major bug reported by Karthik Kambatla and fixed by Karthik Kambatla
  y.s.rm.DelegationTokenRenewer attempts to renew token even after removing an app
  

  yarn.s.rm.security.DelegationTokenRenewer uses TimerTask/Timer. When such a timer task is canceled, already scheduled tasks run to completion. The task should check for such cancellation before running. Also, delegationTokens needs to be synchronized on all accesses.

• YARN-258. Major bug reported by Ravi Prakash and fixed by Ravi Prakash (resourcemanager)
  RM web page UI shows Invalid Date for start and finish times
  

  Whenever the number of jobs was greater than a 100, two javascript arrays were being populated. appsData and appsTableData. appsData was winning out (because it was coming out later) and so renderHadoopDate was trying to render a <br title=""...> string.

• YARN-254. Major improvement reported by Sandy Ryza and fixed by Sandy Ryza (resourcemanager , scheduler)
  Update fair scheduler web UI for hierarchical queues
  

  The fair scheduler should have a web UI similar to the capacity scheduler that shows nested queues.

• YARN-253. Critical bug reported by Tom White and fixed by Tom White (nodemanager)
  Container launch may fail if no files were localized
  

  This can be demonstrated with DistributedShell. The containers running the shell do not have any files to localize (if there is no shell script to copy) so if they run on a different NM to the AM (which does localize files), then they will fail since the appcache directory does not exist.

• YARN-251. Major bug reported by Tom White and fixed by Tom White (resourcemanager)
  Proxy URI generation fails for blank tracking URIs
  

  If the URI is an empty string (the default if not set), then a warning is displayed. A null URI displays no such warning. These two cases should be handled in the same way.

• YARN-230. Major sub-task reported by Bikas Saha and fixed by Bikas Saha (resourcemanager)
  Make changes for RM restart phase 1
  

  As described in YARN-128, phase 1 of RM restart puts in place mechanisms to save application state and read them back after restart. Upon restart, the NM's are asked to reboot and the previously running AM's are restarted. After this is done, RM HA and work preserving restart can continue in parallel. For more details please refer to the design document in YARN-128

• YARN-229. Major sub-task reported by Bikas Saha and fixed by Bikas Saha (resourcemanager)
  Remove old code for restart
  

  Much of the code is dead/commented out and is not executed. Removing it will help with making and understanding new changes.

• YARN-225. Critical bug reported by Devaraj K and fixed by Devaraj K (resourcemanager)
  Proxy Link in RM UI thows NPE in Secure mode
  

  {code:xml} java.lang.NullPointerException at org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.doGet(WebAppProxyServlet.java:241) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:975) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) {code}

• YARN-224. Major bug reported by Sandy Ryza and fixed by Sandy Ryza
  Fair scheduler logs too many nodeUpdate INFO messages
  

  The RM logs are filled with an INFO message the fair scheduler logs every time it receives a nodeUpdate. It should be taken out or demoted to debug.

• YARN-223. Critical bug reported by Radim Kolar and fixed by Radim Kolar
  Change processTree interface to work better with native code
  

  Problem is that on every update of processTree new object is required. This is undesired when working with processTree implementation in native code. replace ProcessTree.getProcessTree() with updateProcessTree(). No new object allocation is needed and it simplify application code a bit.

• YARN-222. Major improvement reported by Sandy Ryza and fixed by Sandy Ryza (resourcemanager , scheduler)
  Fair scheduler should create queue for each user by default
  

  In MR1 the fair scheduler's default behavior was to create a pool for each user. The YARN fair scheduler has this capability, but it should be turned on by default, for consistency.

• YARN-219. Critical sub-task reported by Robert Joseph Evans and fixed by Robert Joseph Evans (nodemanager)
  NM should aggregate logs when application finishes.
  

  The NM should only aggregate logs when the application finishes. This will reduce the load on the NN, especially with respect to lease renewal.

• YARN-217. Blocker bug reported by Devaraj K and fixed by Devaraj K (resourcemanager)
  yarn rmadmin commands fail in secure cluster
  

  All the rmadmin commands fail in secure mode with the "protocol org.apache.hadoop.yarn.server.nodemanager.api.RMAdminProtocolPB is unauthorized" message in RM logs.

• YARN-216. Major improvement reported by Todd Lipcon and fixed by Robert Joseph Evans
  Remove jquery theming support
  

  As of today we have 9.4MB of JQuery themes in our code tree. In addition to being a waste of space, it's a highly questionable feature. I've never heard anyone complain that the Hadoop interface isn't themeable enough, and there's far more value in consistency across installations than there is in themeability. Let's rip it out.

• YARN-214. Major bug reported by Jason Lowe and fixed by Jonathan Eagles (resourcemanager)
  RMContainerImpl does not handle event EXPIRE at state RUNNING
  

  RMContainerImpl has a race condition where a container can enter the RUNNING state just as the container expires. This results in an invalid event transition error: {noformat} 2012-11-11 05:31:38,954 [ResourceManager Event Processor] ERROR org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: Can't handle this event at current state org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: EXPIRE at RUNNING at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301) at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43) at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443) at org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:205) at org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:44) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApp.containerCompleted(SchedulerApp.java:203) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.completedContainer(LeafQueue.java:1337) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.completedContainer(CapacityScheduler.java:739) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:659) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:80) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:340) at java.lang.Thread.run(Thread.java:619) {noformat} EXPIRE needs to be handled (well at least ignored) in the RUNNING state to account for this race condition.

• YARN-212. Blocker bug reported by Nathan Roberts and fixed by Nathan Roberts (nodemanager)
  NM state machine ignores an APPLICATION_CONTAINER_FINISHED event when it shouldn't
  

  The NM state machines can make the following two invalid state transitions when a speculative attempt is killed shortly after it gets started. When this happens the NM keeps the log aggregation context open for this application and therefore chews up FDs and leases on the NN, eventually running the NN out of FDs and bringing down the entire cluster. 2012-11-07 05:36:33,774 [AsyncDispatcher event handler] WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Can't handle this event at current state org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: APPLICATION_CONTAINER_FINISHED at INITING 2012-11-07 05:36:33,775 [AsyncDispatcher event handler] WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Can't handle this event at current state: Current: [DONE], eventType: [INIT_CONTAINER] org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: INIT_CONTAINER at DONE

• YARN-206. Major bug reported by Jason Lowe and fixed by Jason Lowe (resourcemanager)
  TestApplicationCleanup.testContainerCleanup occasionally fails
  

  testContainerCleanup is occasionally failing with the error: testContainerCleanup(org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup): expected:<2> but was:<1>

• YARN-204. Major bug reported by Aleksey Gorshkov and fixed by Aleksey Gorshkov (applications)
  test coverage for org.apache.hadoop.tools
  

  Added some tests for org.apache.hadoop.tools

• YARN-202. Critical bug reported by Kihwal Lee and fixed by Kihwal Lee
  Log Aggregation generates a storm of fsync() for namenode
  

  When the log aggregation is on, write to each aggregated container log causes hflush() to be called. For large clusters, this can creates a lot of fsync() calls for namenode. We have seen 6-7x increase in the average number of fsync operations compared to 1.0.x on a large busy cluster. Over 99% of fsync ops were for log aggregation writing to tmp files.

• YARN-201. Critical bug reported by Jason Lowe and fixed by Jason Lowe (capacityscheduler)
  CapacityScheduler can take a very long time to schedule containers if requests are off cluster
  

  When a user runs a job where one of the input files is a large file on another cluster, the job can create many splits on nodes which are unreachable for computation from the current cluster. The off-switch delay logic in LeafQueue can cause the ResourceManager to allocate containers for the job very slowly. In one case the job was only getting one container every 23 seconds, and the queue had plenty of spare capacity.

• YARN-189. Blocker bug reported by Thomas Graves and fixed by Thomas Graves (resourcemanager)
  deadlock in RM - AMResponse object
  

  we ran into a deadlock in the RM. ============================= "1128743461@qtp-1252749669-5201": waiting for ownable synchronizer 0x00002aabbc87b960, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync), which is held by "AsyncDispatcher event handler" "AsyncDispatcher event handler": waiting to lock monitor 0x00002ab0bba3a370 (object 0x00002aab3d4cd698, a org.apache.hadoop.yarn.api.records.impl.pb.AMResponsePBImpl), which is held by "IPC Server handler 36 on 8030" "IPC Server handler 36 on 8030": waiting for ownable synchronizer 0x00002aabbc87b960, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync), which is held by "AsyncDispatcher event handler" Java stack information for the threads listed above: =================================================== "1128743461@qtp-1252749669-5201": at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00002aabbc87b960> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:941) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1261) at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:594) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.getFinalApplicationStatus(RMAppAttemptImpl.java:2 95) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.getFinalApplicationStatus(RMAppImpl.java:222) at org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices.getApps(RMWebServices.java:328) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaM ... ... .. "AsyncDispatcher event handler": at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.unregisterAttempt(ApplicationMasterService.java:307) - waiting to lock <0x00002aab3d4cd698> (a org.apache.hadoop.yarn.api.records.impl.pb.AMResponsePBImpl) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$BaseFinalTransition.transition(RMAppAttemptImpl.java:647) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$FinalTransition.transition(RMAppAttemptImpl.java:809) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$FinalTransition.transition(RMAppAttemptImpl.java:796) at org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:357) at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298) at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43) at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443) - locked <0x00002aabbb673090> (a org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:478) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:81) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:436) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:417) at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126) at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75) at java.lang.Thread.run(Thread.java:619) "IPC Server handler 36 on 8030": at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00002aabbc87b960> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.pullJustFinishedContainers(RMAppAttemptImpl.java:437) at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:285) - locked <0x00002aab3d4cd698> (a org.apache.hadoop.yarn.api.records.impl.pb.AMResponsePBImpl) at org.apache.hadoop.yarn.api.impl.pb.service.AMRMProtocolPBServiceImpl.allocate(AMRMProtocolPBServiceImpl.java:56) at org.apache.hadoop.yarn.proto.AMRMProtocol$AMRMProtocolService$2.callBlockingMethod(AMRMProtocol.java:87) at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:353) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1528) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1524) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1212) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1522)

• YARN-188. Major test reported by Aleksey Gorshkov and fixed by Aleksey Gorshkov (capacityscheduler)
  Coverage fixing for CapacityScheduler
  

  some tests for CapacityScheduler YARN-188-branch-0.23.patch patch for branch 0.23 YARN-188-branch-2.patch patch for branch 2 YARN-188-trunk.patch patch for trunk

• YARN-187. Major new feature reported by Sandy Ryza and fixed by Sandy Ryza (scheduler)
  Add hierarchical queues to the fair scheduler
  

• YARN-186. Major test reported by Aleksey Gorshkov and fixed by Aleksey Gorshkov (resourcemanager , scheduler)
  Coverage fixing LinuxContainerExecutor
  

  Added some tests for LinuxContainerExecuror YARN-186-branch-0.23.patch patch for branch-0.23 YARN-186-branch-2.patch patch for branch-2 ARN-186-trunk.patch patch for trank

• YARN-184. Major improvement reported by Sandy Ryza and fixed by Sandy Ryza
  Remove unnecessary locking in fair scheduler, and address findbugs excludes.
  

  In YARN-12, locks were added to all fields of QueueManager to address findbugs. In addition, findbugs exclusions were added in response to MAPREDUCE-4439, without a deep look at the code.

• YARN-183. Minor improvement reported by Sandy Ryza and fixed by Sandy Ryza (scheduler)
  Clean up fair scheduler code
  

  The fair scheduler code has a bunch of minor stylistic issues.

• YARN-181. Critical bug reported by Siddharth Seth and fixed by Siddharth Seth (resourcemanager)
  capacity-scheduler.xml move breaks Eclipse import
  

  Eclipse doesn't seem to handle "testResources" which resolve to an absolute path. YARN-140 moved capacity-scheduler.cfg a couple of levels up to the hadoop-yarn project.

• YARN-180. Critical bug reported by Thomas Graves and fixed by Arun C Murthy (capacityscheduler)
  Capacity scheduler - containers that get reserved create container token to early
  

  The capacity scheduler has the ability to 'reserve' containers. Unfortunately before it decides that it goes to reserved rather then assigned, the Container object is created which creates a container token that expires in roughly 10 minutes by default. This means that by the time the NM frees up enough space on that node for the container to move to assigned the container token may have expired.

• YARN-179. Blocker bug reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli (capacityscheduler)
  Bunch of test failures on trunk
  

  {{CapacityScheduler.setConf()}} mandates a YarnConfiguration. It doesn't need to, throughout all of YARN, components only depend on Configuration and depend on the callers to provide correct configuration. This is causing multiple tests to fail.

• YARN-178. Critical bug reported by Radim Kolar and fixed by Radim Kolar
  Fix custom ProcessTree instance creation
  

  1. In current pluggable resourcecalculatorprocesstree is not passed root process id to custom implementation making it unusable. 2. pstree do not extend Configured as it should Added constructor with pid argument with testsuite. Also added test that pstree is correctly configured.

• YARN-177. Critical bug reported by Thomas Graves and fixed by Arun C Murthy (capacityscheduler)
  CapacityScheduler - adding a queue while the RM is running has wacky results
  

  Adding a queue to the capacity scheduler while the RM is running and then running a job in the queue added results in very strange behavior. The cluster Total Memory can either decrease or increase. We had a cluster where total memory decreased to almost 1/6th the capacity. Running on a small test cluster resulted in the capacity going up by simply adding a queue and running wordcount. Looking at the RM logs, used memory can go negative but other logs show the number positive: 2012-10-21 22:56:44,796 [ResourceManager Event Processor] INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.0375 absoluteUsedCapacity=0.0375 used=memory: 7680 cluster=memory: 204800 2012-10-21 22:56:45,831 [ResourceManager Event Processor] INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: completedContainer queue=root usedCapacity=-0.0225 absoluteUsedCapacity=-0.0225 used=memory: -4608 cluster=memory: 204800

• YARN-170. Major bug reported by Sandy Ryza and fixed by Sandy Ryza (nodemanager)
  NodeManager stop() gets called twice on shutdown
  

  The stop method in the NodeManager gets called twice when the NodeManager is shut down via the shutdown hook. The first is the stop that gets called directly by the shutdown hook. The second occurs when the NodeStatusUpdaterImpl is stopped. The NodeManager responds to the NodeStatusUpdaterImpl stop stateChanged event by stopping itself. This is so that NodeStatusUpdaterImpl can notify the NodeManager to stop, by stopping itself in response to a request from the ResourceManager This could be avoided if the NodeStatusUpdaterImpl were to stop the NodeManager by calling its stop method directly.

• YARN-169. Minor improvement reported by Anthony Rojas and fixed by Anthony Rojas (nodemanager)
  Update log4j.appender.EventCounter to use org.apache.hadoop.log.metrics.EventCounter
  

  We should update the log4j.appender.EventCounter in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/container-log4j.properties to use *org.apache.hadoop.log.metrics.EventCounter* rather than *org.apache.hadoop.metrics.jvm.EventCounter* to avoid triggering the following warning: {code}WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files{code}

• YARN-166. Major bug reported by Thomas Graves and fixed by Thomas Graves (capacityscheduler)
  capacity scheduler doesn't allow capacity < 1.0
  

  1.x supports queue capacity < 1, but in 0.23 the capacity scheduler doesn't. This is an issue for us since we have a large cluster running 1.x that currently has a queue with capacity 0.5%.

• YARN-165. Blocker improvement reported by Jason Lowe and fixed by Jason Lowe (resourcemanager)
  RM should point tracking URL to RM web page for app when AM fails
  

  Currently when an ApplicationMaster fails the ResourceManager is updating the tracking URL to an empty string, see RMAppAttemptImpl.ContainerFinishedTransition. Unfortunately when the client attempts to follow the proxy URL it results in a web page showing an HTTP 500 error and an ugly backtrace because "http://" isn't a very helpful tracking URL. It would be much more helpful if the proxy URL redirected to the RM webapp page for the specific application. That page shows the various AM attempts and pointers to their logs which will be useful for debugging the problems that caused the AM attempts to fail.

• YARN-163. Major bug reported by Jason Lowe and fixed by Jason Lowe (nodemanager)
  Retrieving container log via NM webapp can hang with multibyte characters in log
  

  ContainerLogsBlock.printLogs currently assumes that skipping N bytes in the log file is the same as skipping N characters, but that is not true when the log contains multibyte characters. This can cause the loop that skips a portion of the log to try to skip past the end of the file and loop forever (or until Jetty kills the worker thread).

• YARN-161. Major bug reported by Chris Nauroth and fixed by Chris Nauroth (api)
  Yarn Common has multiple compiler warnings for unchecked operations
  

  The warnings are in classes StateMachineFactory, RecordFactoryProvider, RpcFactoryProvider, and YarnRemoteExceptionFactoryProvider. OpenJDK 1.6.0_24 actually treats these as compilation errors, causing the build to fail.

• YARN-159. Major bug reported by Thomas Graves and fixed by Thomas Graves (resourcemanager)
  RM web ui applications page should be sorted to display last app first
  

  RM web ui applications page should be sorted to display last app first. It currently sorts with smallest application id first, which is the first apps that were submitted. After you have one page worth of apps its much more useful for it to sort such that the biggest appid (last submitted app) shows up first.

• YARN-151. Major bug reported by Robert Joseph Evans and fixed by Ravi Prakash
  Browser thinks RM main page JS is taking too long
  

  The main RM page with the default settings of 10,000 applications can cause browsers to think that the JS on the page is stuck and ask you if you want to kill it. This is a big usability problem.

• YARN-150. Major bug reported by Bikas Saha and fixed by Bikas Saha
  AppRejectedTransition does not unregister app from master service and scheduler
  

  AttemptStartedTransition() adds the app to the ApplicationMasterService and scheduler. when the scheduler rejects the app then AppRejectedTransition() forgets to unregister it from the ApplicationMasterService.

• YARN-146. Major new feature reported by Sandy Ryza and fixed by Sandy Ryza (resourcemanager)
  Add unit tests for computing fair share in the fair scheduler
  

  MR1 had TestComputeFairShares. This should go into the YARN fair scheduler.

• YARN-145. Major new feature reported by Sandy Ryza and fixed by Sandy Ryza (resourcemanager)
  Add a Web UI to the fair share scheduler
  

  The fair scheduler had a UI in MR1. Port the capacity scheduler web UI and modify appropriately for the fair share scheduler.

• YARN-140. Major bug reported by Ahmed Radwan and fixed by Ahmed Radwan (capacityscheduler)
  Add capacity-scheduler-default.xml to provide a default set of configurations for the capacity scheduler.
  

  When setting up the capacity scheduler users are faced with problems like: {code} FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager java.lang.IllegalArgumentException: Illegal capacity of -1 for queue root {code} Which basically arises from missing basic configurations, which in many cases, there is no need to explicitly provide, and a default configuration will be sufficient. For example, to address the error above, the user need to add a capacity of 100 to the root queue. So, we need to add a capacity-scheduler-default.xml, this will be helpful to provide the basic set of default configurations required to run the capacity scheduler. The user can still override existing configurations or provide new ones in capacity-scheduler.xml. This is similar to *-default.xml vs *-site.xml for yarn, core, mapred, hdfs, etc.

• YARN-139. Major bug reported by Nathan Roberts and fixed by Vinod Kumar Vavilapalli (api)
  Interrupted Exception within AsyncDispatcher leads to user confusion
  

  Successful applications tend to get InterruptedExceptions during shutdown. The exception is harmless but it leads to lots of user confusion and therefore could be cleaned up. 2012-09-28 14:50:12,477 WARN [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Interrupted Exception while stopping java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1143) at java.lang.Thread.join(Thread.java:1196) at org.apache.hadoop.yarn.event.AsyncDispatcher.stop(AsyncDispatcher.java:105) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:99) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:89) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:437) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:402) at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126) at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75) at java.lang.Thread.run(Thread.java:619) 2012-09-28 14:50:12,477 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.service.AbstractService: Service:Dispatcher is stopped. 2012-09-28 14:50:12,477 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.mapreduce.v2.app.MRAppMaster is stopped. 2012-09-28 14:50:12,477 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Exiting MR AppMaster..GoodBye

• YARN-136. Major bug reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli (resourcemanager)
  Make ClientTokenSecretManager part of RMContext
  

  Helps to add it to the context instead of passing it all around as an extra parameter.

• YARN-135. Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli (resourcemanager)
  ClientTokens should be per app-attempt and be unregistered on App-finish.
  

  Two issues: - ClientTokens are per app-attempt but are created per app. - Apps don't get unregistered from RMClientTokenSecretManager.

• YARN-134. Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli
  ClientToAMSecretManager creates keys without checking for validity of the appID
  

• YARN-133. Major bug reported by Thomas Graves and fixed by Ravi Prakash (resourcemanager)
  update web services docs for RM clusterMetrics
  

  Looks like jira https://issues.apache.org/jira/browse/MAPREDUCE-3747 added in more RM cluster metrics but the docs didn't get updated: http://hadoop.apache.org/docs/r0.23.3/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Metrics_API

• YARN-131. Major bug reported by Ahmed Radwan and fixed by Ahmed Radwan (capacityscheduler)
  Incorrect ACL properties in capacity scheduler documentation
  

  The CapacityScheduler apt file incorrectly specifies the property names controlling acls for application submission and queue administration. {{yarn.scheduler.capacity.root.<queue-path>.acl_submit_jobs}} should be {{yarn.scheduler.capacity.root.<queue-path>.acl_submit_applications}} {{yarn.scheduler.capacity.root.<queue-path>.acl_administer_jobs}} should be {{yarn.scheduler.capacity.root.<queue-path>.acl_administer_queue}} Uploading a patch momentarily.

• YARN-129. Major improvement reported by Tom White and fixed by Tom White (client)
  Simplify classpath construction for mini YARN tests
  

  The test classpath includes a special file called 'mrapp-generated-classpath' (or similar in distributed shell) that is constructed at build time, and whose contents are a classpath with all the dependencies needed to run the tests. When the classpath for a container (e.g. the AM) is constructed the contents of mrapp-generated-classpath is read and added to the classpath, and the file itself is then added to the classpath so that later when the AM constructs a classpath for a task container it can propagate the test classpath correctly. This mechanism can be drastically simplified by propagating the system classpath of the current JVM (read from the java.class.path property) to a launched JVM, but only if running in the context of the mini YARN cluster. Any tests that use the mini YARN cluster will automatically work with this change. Although any that explicitly deal with mrapp-generated-classpath can be simplified.

• YARN-127. Major bug reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli
  Move RMAdmin tool to the client package
  

  It belongs to the client package and not the RM clearly.

• YARN-116. Major bug reported by xieguiming and fixed by xieguiming (resourcemanager)
  RM is missing ability to add include/exclude files without a restart
  

  The "yarn.resourcemanager.nodes.include-path" default value is "", if we need to add an include file, we must currently restart the RM. I suggest that for adding an include or exclude file, there should be no need to restart the RM. We may only execute the refresh command. The HDFS NameNode already has this ability. Fix is to the modify HostsFileReader class instances: From: {code} public HostsFileReader(String inFile, String exFile) {code} To: {code} public HostsFileReader(Configuration conf, String NODES_INCLUDE_FILE_PATH,String DEFAULT_NODES_INCLUDE_FILE_PATH, String NODES_EXCLUDE_FILE_PATH,String DEFAULT_NODES_EXCLUDE_FILE_PATH) {code} And thus, we can read the config file dynamically when a {{refreshNodes}} is invoked and therefore have no need to restart the ResourceManager.

• YARN-103. Major improvement reported by Bikas Saha and fixed by Bikas Saha
  Add a yarn AM - RM client module
  

  Add a basic client wrapper library to the AM RM protocol in order to prevent proliferation of code being duplicated everywhere. Provide helper functions to perform reverse mapping of container requests to RM allocation resource request table format.

• YARN-102. Trivial bug reported by Devaraj K and fixed by Devaraj K (resourcemanager)
  Move the apache licence header to the top of the file in MemStore.java
  

• YARN-94. Major bug reported by Vinod Kumar Vavilapalli and fixed by Hitesh Shah (applications/distributed-shell)
  DistributedShell jar should point to Client as the main class by default
  

  Today, it says so.. {code} $ $YARN_HOME/bin/yarn jar $YARN_HOME/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-$VERSION.jar RunJar jarFile [mainClass] args... {code}

• YARN-93. Major bug reported by Jason Lowe and fixed by Jason Lowe (resourcemanager)
  Diagnostics missing from applications that have finished but failed
  

  If an application finishes in the YARN sense but fails in the app framework sense (e.g.: a failed MapReduce job) then diagnostics are missing from the RM web page for the application. The RM should be reporting diagnostic messages even for successful YARN applications.

• YARN-82. Minor bug reported by Andy Isaacson and fixed by Hemanth Yamijala (nodemanager)
  YARN local-dirs defaults to /tmp/nm-local-dir
  

  {{yarn.nodemanager.local-dirs}} defaults to {{/tmp/nm-local-dir}}. It should be {hadoop.tmp.dir}/nm-local-dir or similar. Among other problems, this can prevent multiple test clusters from starting on the same machine. Thanks to Hemanth Yamijala for reporting this issue.

• YARN-78. Major bug reported by Bikas Saha and fixed by Bikas Saha (applications)
  Change UnmanagedAMLauncher to use YarnClientImpl
  

  YARN-29 added a common client impl to talk to the RM. Use that in the UnmanagedAMLauncher.

• YARN-72. Major bug reported by Hitesh Shah and fixed by Sandy Ryza (nodemanager)
  NM should handle cleaning up containers when it shuts down
  

  Ideally, the NM should wait for a limited amount of time when it gets a shutdown signal for existing containers to complete and kill the containers ( if we pick an aggressive approach ) after this time interval. For NMs which come up after an unclean shutdown, the NM should look through its directories for existing container.pids and try and kill an existing containers matching the pids found.

• YARN-57. Major improvement reported by Radim Kolar and fixed by Radim Kolar (nodemanager)
  Plugable process tree
  

  Trunk version of Pluggable process tree. Work based on MAPREDUCE-4204

• YARN-50. Blocker sub-task reported by Siddharth Seth and fixed by Siddharth Seth
  Implement renewal / cancellation of Delegation Tokens
  

  Currently, delegation tokens issues by the RM and History server cannot be renewed or cancelled. This needs to be implemented.

• YARN-43. Major bug reported by Thomas Graves and fixed by Thomas Graves
  TestResourceTrackerService fail intermittently on jdk7
  

  Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.73 sec <<< FAILURE! testDecommissionWithIncludeHosts(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService) Time elapsed: 0.086 sec <<< FAILURE! junit.framework.AssertionFailedError: expected:<0> but was:<1> at junit.framework.Assert.fail(Assert.java:47) at junit.framework.Assert.failNotEquals(Assert.java:283) at junit.framework.Assert.assertEquals(Assert.java:64) at junit.framework.Assert.assertEquals(Assert.java:195) at junit.framework.Assert.assertEquals(Assert.java:201) at org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService.testDecommissionWithIncludeHosts(TestResourceTrackerService.java:90)

• YARN-40. Major bug reported by Devaraj K and fixed by Devaraj K (client)
  Provide support for missing yarn commands
  

  1. status <app-id> 2. kill <app-id> (Already issue present with Id : MAPREDUCE-3793) 3. list-apps [all] 4. nodes-report

• YARN-33. Major bug reported by Mayank Bansal and fixed by Mayank Bansal (nodemanager)
  LocalDirsHandler should validate the configured local and log dirs
  

  WHen yarn.nodemanager.log-dirs is with file:// URI then startup of node manager creates the directory like file:// under CWD. WHich should not be there. Thanks, Mayank

• YARN-32. Major bug reported by Thomas Graves and fixed by Vinod Kumar Vavilapalli
  TestApplicationTokens fails intermintently on jdk7
  

  TestApplicationsTokens fails intermintently on jdk7.

• YARN-30. Major bug reported by Thomas Graves and fixed by Thomas Graves
  TestNMWebServicesApps, TestRMWebServicesApps and TestRMWebServicesNodes fail on jdk7
  

  It looks like the string changed from "const class" to "constant". Tests run: 19, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 6.786 sec <<< FAILURE! testNodeAppsStateInvalid(org.apache.hadoop.yarn.server.nodemanager.webapp.TestNMWebServicesApps) Time elapsed: 0.248 sec <<< FAILURE! java.lang.AssertionError: exception message doesn't match, got: No enum constant org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationState.FOO_STATE expected: No enum const class org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationState.FOO_STATE

• YARN-28. Major bug reported by Thomas Graves and fixed by Thomas Graves
  TestCompositeService fails on jdk7
  

  test TestCompositeService fails when run with jdk7. It appears it expects test testCallSequence to be called first and the sequence numbers to start at 0. On jdk7 its not being called first and sequence number has already been incremented.

• YARN-23. Major improvement reported by Karthik Kambatla and fixed by Karthik Kambatla (scheduler)
  FairScheduler: FSQueueSchedulable#updateDemand() - potential redundant aggregation
  

  In FS, FSQueueSchedulable#updateDemand() limits the demand to maxTasks only after iterating though all the pools and computing the final demand. By checking if the demand has reached maxTasks in every iteration, we can avoid redundant work, at the expense of one condition check every iteration.

• YARN-3. Major sub-task reported by Arun C Murthy and fixed by Andrew Ferguson
  Add support for CPU isolation/monitoring of containers
  

• YARN-2. Major new feature reported by Arun C Murthy and fixed by Arun C Murthy (capacityscheduler , scheduler)
  Enhance CS to schedule accounting for both memory and cpu cores
  

  With YARN being a general purpose system, it would be useful for several applications (MPI et al) to specify not just memory but also CPU (cores) for their resource requirements. Thus, it would be useful to the CapacityScheduler to account for both.

• MAPREDUCE-4977. Major improvement reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (documentation)
  Documentation for pluggable shuffle and pluggable sort
  

• MAPREDUCE-4971. Minor improvement reported by Arun C Murthy and fixed by Arun C Murthy
  Minor extensibility enhancements
  

• MAPREDUCE-4969. Major bug reported by Arpit Agarwal and fixed by Arpit Agarwal (test)
  TestKeyValueTextInputFormat test fails with Open JDK 7
  

• MAPREDUCE-4953. Major bug reported by Andy Isaacson and fixed by Andy Isaacson (pipes)
  HadoopPipes misuses fprintf
  

• MAPREDUCE-4949. Minor improvement reported by Sandy Ryza and fixed by Sandy Ryza (examples)
  Enable multiple pi jobs to run in parallel
  

• MAPREDUCE-4948. Critical bug reported by Junping Du and fixed by Junping Du (client)
  TestYARNRunner.testHistoryServerToken failed on trunk
  

• MAPREDUCE-4946. Critical bug reported by Jason Lowe and fixed by Jason Lowe (mr-am)
  Type conversion of map completion events leads to performance problems with large jobs
  

• MAPREDUCE-4936. Critical bug reported by Daryn Sharp and fixed by Arun C Murthy (mrv2)
  JobImpl uber checks for cpu are wrong
  

• MAPREDUCE-4934. Critical bug reported by Thomas Graves and fixed by Thomas Graves (build)
  Maven RAT plugin is not checking all source files
  

• MAPREDUCE-4928. Major improvement reported by Suresh Srinivas and fixed by Suresh Srinivas (applicationmaster , security)
  Use token request messages defined in hadoop common
  

  Protobuf message GetDelegationTokenRequestProto field renewer is made requried from optional. This change is not wire compatible with the older releases.

• MAPREDUCE-4925. Major bug reported by Karthik Kambatla and fixed by Karthik Kambatla (examples)
  The pentomino option parser may be buggy
  

• MAPREDUCE-4924. Trivial bug reported by Robert Kanter and fixed by Robert Kanter (mrv1)
  flakey test: org.apache.hadoop.mapred.TestClusterMRNotification.testMR
  

• MAPREDUCE-4923. Minor bug reported by Sandy Ryza and fixed by Sandy Ryza (mrv1 , mrv2 , task)
  Add toString method to TaggedInputSplit
  

• MAPREDUCE-4921. Blocker bug reported by Daryn Sharp and fixed by Daryn Sharp (client)
  JobClient should acquire HS token with RM principal
  

• MAPREDUCE-4920. Major bug reported by Vinod Kumar Vavilapalli and fixed by Suresh Srinivas
  Use security token protobuf definition from hadoop common
  

• MAPREDUCE-4913. Major bug reported by Jason Lowe and fixed by Jason Lowe (mr-am)
  TestMRAppMaster#testMRAppMasterMissingStaging occasionally exits
  

• MAPREDUCE-4907. Major improvement reported by Sandy Ryza and fixed by Sandy Ryza (mrv1 , tasktracker)
  TrackerDistributedCacheManager issues too many getFileStatus calls
  

• MAPREDUCE-4905. Major test reported by Aleksey Gorshkov and fixed by Aleksey Gorshkov
  test org.apache.hadoop.mapred.pipes
  

• MAPREDUCE-4902. Trivial bug reported by Albert Chu and fixed by Albert Chu
  Fix typo "receievd" should be "received" in log output
  

• MAPREDUCE-4899. Major improvement reported by Derek Dagit and fixed by Derek Dagit
  Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID
  

• MAPREDUCE-4895. Major bug reported by Dennis Y and fixed by Dennis Y
  Fix compilation failure of org.apache.hadoop.mapred.gridmix.TestResourceUsageEmulators
  

• MAPREDUCE-4894. Blocker bug reported by Siddharth Seth and fixed by Siddharth Seth (jobhistoryserver , mrv2)
  Renewal / cancellation of JobHistory tokens
  

• MAPREDUCE-4893. Major bug reported by Bikas Saha and fixed by Bikas Saha (applicationmaster)
  MR AppMaster can do sub-optimal assignment of containers to map tasks leading to poor node locality
  

• MAPREDUCE-4890. Critical bug reported by Jason Lowe and fixed by Jason Lowe (mr-am)
  Invalid TaskImpl state transitions when task fails while speculating
  

• MAPREDUCE-4884. Major bug reported by Chris Nauroth and fixed by Chris Nauroth (contrib/streaming , test)
  streaming tests fail to start MiniMRCluster due to "Queue configuration missing child queue names for root"
  

• MAPREDUCE-4861. Major bug reported by Karthik Kambatla and fixed by Karthik Kambatla
  Cleanup: Remove unused mapreduce.security.token.DelegationTokenRenewal
  

• MAPREDUCE-4856. Major bug reported by Sandy Ryza and fixed by Sandy Ryza (test)
  TestJobOutputCommitter uses same directory as TestJobCleanup
  

• MAPREDUCE-4848. Major bug reported by Jason Lowe and fixed by Jerry Chen (mr-am)
  TaskAttemptContext cast error during AM recovery
  

• MAPREDUCE-4845. Major improvement reported by Sandy Ryza and fixed by Sandy Ryza (client)
  ClusterStatus.getMaxMemory() and getUsedMemory() exist in MR1 but not MR2
  

• MAPREDUCE-4842. Blocker bug reported by Jason Lowe and fixed by Mariappan Asokan (mrv2)
  Shuffle race can hang reducer
  

• MAPREDUCE-4838. Major improvement reported by Arun C Murthy and fixed by Zhijie Shen
  Add extra info to JH files
  

• MAPREDUCE-4836. Major bug reported by Ravi Prakash and fixed by Ravi Prakash
  Elapsed time for running tasks on AM web UI tasks page is 0
  

• MAPREDUCE-4833. Critical bug reported by Robert Joseph Evans and fixed by Robert Parker (applicationmaster , mrv2)
  Task can get stuck in FAIL_CONTAINER_CLEANUP
  

• MAPREDUCE-4832. Critical bug reported by Robert Joseph Evans and fixed by Jason Lowe (applicationmaster)
  MR AM can get in a split brain situation
  

• MAPREDUCE-4825. Major bug reported by Jason Lowe and fixed by Jason Lowe (mr-am)
  JobImpl.finished doesn't expect ERROR as a final job state
  

• MAPREDUCE-4822. Trivial improvement reported by Robert Joseph Evans and fixed by Chu Tong (jobhistoryserver)
  Unnecessary conversions in History Events
  

• MAPREDUCE-4819. Blocker bug reported by Jason Lowe and fixed by Bikas Saha (mr-am)
  AM can rerun job after reporting final job status to the client
  

• MAPREDUCE-4817. Critical bug reported by Jason Lowe and fixed by Thomas Graves (applicationmaster , mr-am)
  Hardcoded task ping timeout kills tasks localizing large amounts of data
  

• MAPREDUCE-4813. Critical bug reported by Jason Lowe and fixed by Jason Lowe (applicationmaster)
  AM timing out during job commit
  

• MAPREDUCE-4811. Minor improvement reported by Ravi Prakash and fixed by Ravi Prakash (jobhistoryserver , mrv2)
  JobHistoryServer should show when it was started in WebUI About page
  

• MAPREDUCE-4810. Minor improvement reported by Jason Lowe and fixed by Jerry Chen (applicationmaster)
  Add admin command options for ApplicationMaster
  

• MAPREDUCE-4809. Major sub-task reported by Arun C Murthy and fixed by Mariappan Asokan
  Change visibility of classes for pluggable sort changes
  

• MAPREDUCE-4808. Major new feature reported by Arun C Murthy and fixed by Mariappan Asokan
  Refactor MapOutput and MergeManager to facilitate reuse by Shuffle implementations
  

• MAPREDUCE-4807. Major sub-task reported by Arun C Murthy and fixed by Mariappan Asokan
  Allow MapOutputBuffer to be pluggable
  

• MAPREDUCE-4803. Minor test reported by Mariappan Asokan and fixed by Mariappan Asokan (test)
  Duplicate copies of TestIndexCache.java
  

• MAPREDUCE-4802. Major improvement reported by Ravi Prakash and fixed by Ravi Prakash (mr-am , mrv2 , webapps)
  Takes a long time to load the task list on the AM for large jobs
  

• MAPREDUCE-4801. Critical bug reported by Jason Lowe and fixed by Jason Lowe
  ShuffleHandler can generate large logs due to prematurely closed channels
  

• MAPREDUCE-4797. Major bug reported by Jason Lowe and fixed by Jason Lowe (applicationmaster)
  LocalContainerAllocator can loop forever trying to contact the RM
  

• MAPREDUCE-4787. Major bug reported by Ravi Prakash and fixed by Robert Parker (test)
  TestJobMonitorAndPrint is broken
  

• MAPREDUCE-4786. Major bug reported by Ravi Prakash and fixed by Ravi Prakash (mrv2)
  Job End Notification retry interval is 5 milliseconds by default
  

• MAPREDUCE-4782. Blocker bug reported by Mark Fuhs and fixed by Mark Fuhs (client)
  NLineInputFormat skips first line of last InputSplit
  

• MAPREDUCE-4778. Major bug reported by Sandy Ryza and fixed by Sandy Ryza (jobtracker , scheduler)
  Fair scheduler event log is only written if directory exists on HDFS
  

• MAPREDUCE-4777. Minor improvement reported by Sandy Ryza and fixed by Sandy Ryza
  In TestIFile, testIFileReaderWithCodec relies on testIFileWriterWithCodec
  

• MAPREDUCE-4774. Major bug reported by Ivan A. Veselovsky and fixed by Jason Lowe (applicationmaster , mrv2)
  JobImpl does not handle asynchronous task events in FAILED state
  

• MAPREDUCE-4772. Critical bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans (mrv2)
  Fetch failures can take way too long for a map to be restarted
  

• MAPREDUCE-4771. Major bug reported by Jason Lowe and fixed by Jason Lowe (mrv2)
  KeyFieldBasedPartitioner not partitioning properly when configured
  

• MAPREDUCE-4764. Major improvement reported by Ivan A. Veselovsky and fixed by
  repair test org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
  

• MAPREDUCE-4763. Minor improvement reported by Ivan A. Veselovsky and fixed by
  repair test org.apache.hadoop.mapreduce.security.TestUmbilicalProtocolWithJobToken
  

• MAPREDUCE-4752. Major improvement reported by Robert Joseph Evans and fixed by Robert Joseph Evans (mrv2)
  Reduce MR AM memory usage through String Interning
  

• MAPREDUCE-4751. Major bug reported by Ravi Prakash and fixed by Vinod Kumar Vavilapalli
  AM stuck in KILL_WAIT for days
  

• MAPREDUCE-4748. Blocker bug reported by Robert Joseph Evans and fixed by Jason Lowe (mrv2)
  Invalid event: T_ATTEMPT_SUCCEEDED at SUCCEEDED
  

• MAPREDUCE-4746. Major bug reported by Robert Parker and fixed by Robert Parker (applicationmaster)
  The MR Application Master does not have a config to set environment variables
  

• MAPREDUCE-4741. Minor bug reported by Jason Lowe and fixed by Vinod Kumar Vavilapalli (applicationmaster , mrv2)
  WARN and ERROR messages logged during normal AM shutdown
  

• MAPREDUCE-4740. Blocker bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans (mrv2)
  only .jars can be added to the Distributed Cache classpath
  

• MAPREDUCE-4736. Trivial improvement reported by Brandon Li and fixed by Brandon Li (test)
  Remove obsolete option [-rootDir] from TestDFSIO
  

• MAPREDUCE-4733. Major bug reported by Jason Lowe and fixed by Jason Lowe (applicationmaster , mrv2)
  Reducer can fail to make progress during shuffle if too many reducers complete consecutively
  

• MAPREDUCE-4730. Blocker bug reported by Jason Lowe and fixed by Jason Lowe (applicationmaster , mrv2)
  AM crashes due to OOM while serving up map task completion events
  

• MAPREDUCE-4729. Major bug reported by Thomas Graves and fixed by Vinod Kumar Vavilapalli (jobhistoryserver)
  job history UI not showing all job attempts
  

• MAPREDUCE-4724. Major bug reported by Thomas Graves and fixed by Thomas Graves (jobhistoryserver)
  job history web ui applications page should be sorted to display last app first
  

• MAPREDUCE-4723. Major improvement reported by Sandy Ryza and fixed by Sandy Ryza
  Fix warnings found by findbugs 2
  

• MAPREDUCE-4721. Major bug reported by Ravi Prakash and fixed by Ravi Prakash (jobhistoryserver)
  Task startup time in JHS is same as job startup time.
  

• MAPREDUCE-4720. Major bug reported by Robert Joseph Evans and fixed by Ravi Prakash
  Browser thinks History Server main page JS is taking too long
  

• MAPREDUCE-4712. Major bug reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli (jobhistoryserver)
  mr-jobhistory-daemon.sh doesn't accept --config
  

• MAPREDUCE-4705. Critical bug reported by Jason Lowe and fixed by Jason Lowe (jobhistoryserver , mrv2)
  Historyserver links expire before the history data does
  

• MAPREDUCE-4703. Major improvement reported by Ahmed Radwan and fixed by Ahmed Radwan (mrv1 , mrv2 , test)
  Add the ability to start the MiniMRClientCluster using the configurations used before it is being stopped.
  

• MAPREDUCE-4681. Major bug reported by Arun C Murthy and fixed by Arun C Murthy
  HDFS-3910 broke MR tests
  

• MAPREDUCE-4678. Minor bug reported by Chris McConnell and fixed by Chris McConnell (examples)
  Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException
  

• MAPREDUCE-4674. Minor bug reported by Robert Justice and fixed by Robert Justice
  Hadoop examples secondarysort has a typo "secondarysrot" in the usage
  

• MAPREDUCE-4666. Minor improvement reported by Jason Lowe and fixed by Jason Lowe (jobhistoryserver)
  JVM metrics for history server
  

• MAPREDUCE-4654. Critical bug reported by Colin Patrick McCabe and fixed by Sandy Ryza (test)
  TestDistCp is @ignored
  

• MAPREDUCE-4637. Major bug reported by Tom White and fixed by Mayank Bansal (mrv2)
  Killing an unassigned task attempt causes the job to fail
  

  Handle TaskAttempt diagnostic updates while in the NEW and UNASSIGNED states.

• MAPREDUCE-4616. Minor improvement reported by Tony Burton and fixed by Tony Burton (documentation)
  Improvement to MultipleOutputs javadocs
  

• MAPREDUCE-4607. Major bug reported by Bikas Saha and fixed by Bikas Saha
  Race condition in ReduceTask completion can result in Task being incorrectly failed
  

• MAPREDUCE-4596. Major task reported by Siddharth Seth and fixed by Siddharth Seth (applicationmaster , mrv2)
  Split StateMachine state from states seen by MRClientProtocol (for Job, Task, TaskAttempt)
  

• MAPREDUCE-4554. Major bug reported by Benoy Antony and fixed by Benoy Antony (job submission , security)
  Job Credentials are not transmitted if security is turned off
  

• MAPREDUCE-4521. Major bug reported by Jason Lowe and fixed by Ravi Prakash (mrv2)
  mapreduce.user.classpath.first incompatibility with 0.20/1.x
  

• MAPREDUCE-4520. Major new feature reported by Arun C Murthy and fixed by Arun C Murthy
  Add experimental support for MR AM to schedule CPUs along-with memory
  

• MAPREDUCE-4517. Minor improvement reported by James Kinley and fixed by Jason Lowe (applicationmaster)
  Too many INFO messages written out during AM to RM heartbeat
  

• MAPREDUCE-4479. Major bug reported by Mariappan Asokan and fixed by Mariappan Asokan (test)
  Fix parameter order in assertEquals() in TestCombineInputFileFormat.java
  

• MAPREDUCE-4458. Major improvement reported by Robert Joseph Evans and fixed by Robert Parker (mrv2)
  Warn if java.library.path is used for AM or Task
  

• MAPREDUCE-4425. Critical bug reported by Siddharth Seth and fixed by Jason Lowe (mrv2)
  Speculation + Fetch failures can lead to a hung job
  

• MAPREDUCE-4279. Major bug reported by Rahul Jain and fixed by Devaraj K (jobtracker)
  getClusterStatus() fails with null pointer exception when running jobs in local mode
  

• MAPREDUCE-4278. Major bug reported by Araceli Henley and fixed by Sandy Ryza
  cannot run two local jobs in parallel from the same gateway.
  

• MAPREDUCE-4272. Major bug reported by Luke Lu and fixed by Yu Gao (task)
  SortedRanges.Range#compareTo is not spec compliant
  

• MAPREDUCE-4266. Major task reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (build)
  remove Ant remnants from MR
  

• MAPREDUCE-4229. Major improvement reported by Todd Lipcon and fixed by Miomir Boljanovic (jobtracker)
  Counter names' memory usage can be decreased by interning
  

• MAPREDUCE-4123. Critical bug reported by Nishan Shetty and fixed by Devaraj K (mrv2)
  ./mapred groups gives NoClassDefFoundError
  

• MAPREDUCE-4049. Major sub-task reported by Avner BenHanoch and fixed by Avner BenHanoch (performance , task , tasktracker)
  plugin for generic shuffle service
  

  Allow ReduceTask loading a third party plugin for shuffle (and merge) instead of the default shuffle.

• MAPREDUCE-3678. Major new feature reported by Bejoy KS and fixed by Harsh J (mrv1 , mrv2)
  The Map tasks logs should have the value of input split it processed
  

  A map-task's syslogs now carries basic info on the InputSplit it processed.

• MAPREDUCE-2454. Minor new feature reported by Mariappan Asokan and fixed by Mariappan Asokan
  Allow external sorter plugin for MR
  

  MAPREDUCE-4807 Allow external implementations of the sort phase in a Map task

• MAPREDUCE-2264. Major bug reported by Adam Kramer and fixed by Devaraj K (jobtracker)
  Job status exceeds 100% in some cases
  

• MAPREDUCE-1806. Major bug reported by Paul Yang and fixed by Gera Shegalov (harchive)
  CombineFileInputFormat does not work with paths not on default FS
  

• MAPREDUCE-1700. Major bug reported by Tom White and fixed by Tom White (task)
  User supplied dependencies may conflict with MapReduce system JARs
  

• HDFS-4468. Minor bug reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE
  Fix TestHDFSCLI and TestQuota for HADOOP-9252
  

• HDFS-4462. Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (namenode)
  2NN will fail to checkpoint after an HDFS upgrade from a pre-federation version of HDFS
  

• HDFS-4458. Major bug reported by wenwupeng and fixed by Binglin Chang (balancer)
  start balancer failed with "Failed to create file [/system/balancer.id]" if configure IP on fs.defaultFS
  

• HDFS-4456. Major new feature reported by Tsz Wo (Nicholas), SZE and fixed by Plamen Jeliazkov (webhdfs)
  Add concat to HttpFS and WebHDFS REST API docs
  

• HDFS-4452. Critical bug reported by Konstantin Shvachko and fixed by Konstantin Shvachko (namenode)
  getAdditionalBlock() can create multiple blocks if the client times out and retries.
  

• HDFS-4451. Major bug reported by Joshua Blatt and fixed by (balancer)
  hdfs balancer command returns exit code 1 on success instead of 0
  

  This is an incompatible change from release 2.0.2-alpha and prior releases. Balancer tool exited with exit code 1 on success. It is changed to exit with exit code 0 on success. Non 0 exit code indicates failure.

• HDFS-4445. Blocker sub-task reported by Vinay and fixed by Vinay
  All BKJM ledgers are not checked while tailing, So failover will fail.
  

• HDFS-4444. Trivial bug reported by Stephen Chu and fixed by Stephen Chu
  Add space between total transaction time and number of transactions in FSEditLog#printStatistics
  

• HDFS-4443. Trivial bug reported by Christian Rohling and fixed by Christian Rohling (namenode)
  Remove trailing '`' character from HDFS nodelist jsp
  

• HDFS-4428. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  FsDatasetImpl should disclose what the error is when a rename fails
  

• HDFS-4426. Blocker bug reported by Jason Lowe and fixed by Arpit Agarwal (namenode)
  Secondary namenode shuts down immediately after startup
  

• HDFS-4415. Major bug reported by Robert Kanter and fixed by Robert Kanter
  HostnameFilter should handle hostname resolution failures and continue processing
  

• HDFS-4404. Critical bug reported by liaowenrui and fixed by Todd Lipcon (ha , hdfs-client)
  Create file failure when the machine of first attempted NameNode is down
  

• HDFS-4403. Minor bug reported by Todd Lipcon and fixed by Todd Lipcon (hdfs-client)
  DFSClient can infer checksum type when not provided by reading first byte
  

  The HDFS implementation of getFileChecksum() can now operate correctly against earlier-version datanodes which do not include the checksum type information in their checksum response. The checksum type is automatically inferred by issuing a read of the first byte of each block.

• HDFS-4393. Minor improvement reported by Brandon Li and fixed by Brandon Li
  Empty request and responses in protocol translators can be static final members
  

• HDFS-4392. Trivial improvement reported by Andrew Purtell and fixed by Andrew Purtell (test)
  Use NetUtils#getFreeSocketPort in MiniDFSCluster
  

• HDFS-4385. Critical bug reported by Thomas Graves and fixed by Thomas Graves (build)
  Maven RAT plugin is not checking all source files
  

• HDFS-4384. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (libhdfs)
  test_libhdfs_threaded gets SEGV if JNIEnv cannot be initialized
  

• HDFS-4381. Major improvement reported by Jing Zhao and fixed by Jing Zhao (namenode)
  Document fsimage format details in FSImageFormat class javadoc
  

• HDFS-4377. Trivial bug reported by Eli Collins and fixed by Eli Collins
  Some trivial DN comment cleanup
  

• HDFS-4375. Major improvement reported by Suresh Srinivas and fixed by Suresh Srinivas (namenode , security)
  Use token request messages defined in hadoop common
  

• HDFS-4369. Blocker bug reported by Suresh Srinivas and fixed by Suresh Srinivas (namenode)
  GetBlockKeysResponseProto does not handle null response
  

  Protobuf message GetBlockKeysResponseProto member keys is made optional from required so that null values can be passed over the wire. This is an incompatible wire protocol change and does not affect the API backward compatibility.

• HDFS-4367. Blocker bug reported by Suresh Srinivas and fixed by Suresh Srinivas (namenode)
  GetDataEncryptionKeyResponseProto does not handle null response
  

  Member dataEncryptionKey of the protobuf message GetDataEncryptionKeyResponseProto is made optional instead of required. This is incompatible change is not likely to affect the existing users (that are using HDFS FileSystem and other public APIs).

• HDFS-4364. Blocker bug reported by Suresh Srinivas and fixed by Suresh Srinivas
  GetLinkTargetResponseProto does not handle null path
  

  Protobuf message GetLinkTargetResponseProto member targetPath is made optional from required so that null values can be passed over the wire. This is an incompatible wire protocol change and does not affect the API backward compatibility.

• HDFS-4363. Major bug reported by Suresh Srinivas and fixed by Suresh Srinivas
  Combine PBHelper and HdfsProtoUtil and remove redundant methods
  

• HDFS-4362. Critical bug reported by Suresh Srinivas and fixed by Suresh Srinivas
  GetDelegationTokenResponseProto does not handle null token
  

• HDFS-4359. Major bug reported by Liang Xie and fixed by Liang Xie (datanode)
  remove an unnecessary synchronized keyword in BPOfferService.java
  

• HDFS-4351. Major bug reported by Andrew Wang and fixed by Andrew Wang (namenode)
  Fix BlockPlacementPolicyDefault#chooseTarget when avoiding stale nodes
  

• HDFS-4350. Major bug reported by Andrew Wang and fixed by Andrew Wang
  Make enabling of stale marking on read and write paths independent
  

  This patch makes an incompatible configuration change, as described below: In releases 1.1.0 and other point releases 1.1.x, the configuration parameter "dfs.namenode.check.stale.datanode" could be used to turn on checking for the stale nodes. This configuration is no longer supported in release 1.2.0 onwards and is renamed as "dfs.namenode.avoid.read.stale.datanode". How feature works and configuring this feature: As described in HDFS-3703 release notes, datanode stale period can be configured using parameter "dfs.namenode.stale.datanode.interval" in seconds (default value is 30 seconds). NameNode can be configured to use this staleness information for reads using configuration "dfs.namenode.avoid.read.stale.datanode". When this parameter is set to true, namenode picks a stale datanode as the last target to read from when returning block locations for reads. Using staleness information for writes is as described in the releases notes of HDFS-3912.

• HDFS-4349. Major test reported by Konstantin Shvachko and fixed by Konstantin Shvachko (namenode , test)
  Test reading files from BackupNode
  

• HDFS-4347. Major bug reported by Konstantin Shvachko and fixed by Plamen Jeliazkov (namenode , test)
  TestBackupNode can go into infinite loop "Waiting checkpoint to complete."
  

• HDFS-4344. Major bug reported by tamtam180 and fixed by Andy Isaacson (namenode)
  dfshealth.jsp throws NumberFormatException when dfs.hosts/dfs.hosts.exclude includes port number
  

• HDFS-4326. Major task reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur
  bump up Tomcat version for HttpFS to 6.0.36
  

• HDFS-4315. Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (datanode)
  DNs with multiple BPs can have BPOfferServices fail to start due to unsynchronized map access
  

• HDFS-4308. Major bug reported by Konstantin Shvachko and fixed by Plamen Jeliazkov (namenode)
  addBlock() should persist file blocks once
  

• HDFS-4307. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  SocketCache should use monotonic time
  

• HDFS-4306. Major bug reported by Binglin Chang and fixed by Binglin Chang
  PBHelper.convertLocatedBlock miss convert BlockToken
  

• HDFS-4302. Major bug reported by Eugene Koontz and fixed by Eugene Koontz (ha , namenode)
  Precondition in EditLogFileInputStream's length() method is checked too early in NameNode startup, causing fatal exception
  

• HDFS-4295. Major bug reported by Stephen Chu and fixed by Stephen Chu (security)
  Using port 1023 should be valid when starting Secure DataNode
  

• HDFS-4294. Major bug reported by Robert Parker and fixed by Robert Parker
  Backwards compatibility is not maintained for TestVolumeId
  

• HDFS-4292. Minor bug reported by Binglin Chang and fixed by Binglin Chang
  Sanity check not correct in RemoteBlockReader2.newBlockReader
  

• HDFS-4291. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  edit log unit tests leave stray test_edit_log_file around
  

• HDFS-4288. Critical bug reported by Daryn Sharp and fixed by Daryn Sharp (namenode)
  NN accepts incremental BR as IBR in safemode
  

• HDFS-4282. Major bug reported by Junping Du and fixed by Todd Lipcon (namenode , test)
  TestEditLog.testFuzzSequences FAILED in all pre-commit test
  

• HDFS-4279. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (namenode)
  NameNode does not initialize generic conf keys when started with -recover
  

• HDFS-4274. Minor bug reported by Chris Nauroth and fixed by Chris Nauroth (datanode)
  BlockPoolSliceScanner does not close verification log during shutdown
  

• HDFS-4270. Minor bug reported by Derek Dagit and fixed by Derek Dagit (namenode)
  Replications of the highest priority should be allowed to choose a source datanode that has reached its max replication limit
  

• HDFS-4268. Major bug reported by Konstantin Shvachko and fixed by Konstantin Shvachko (namenode)
  Remove redundant enum NNHAStatusHeartbeat.State
  

• HDFS-4259. Minor improvement reported by Harsh J and fixed by Harsh J (hdfs-client)
  Improve pipeline DN replacement failure message
  

• HDFS-4247. Blocker sub-task reported by Daryn Sharp and fixed by Daryn Sharp (namenode)
  saveNamespace should be tolerant of dangling lease
  

• HDFS-4242. Major bug reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (namenode)
  Map.Entry is incorrectly used in LeaseManager
  

• HDFS-4238. Major bug reported by Vinay and fixed by Todd Lipcon (ha)
  [HA] Standby namenode should not do purging of shared storage edits.
  

• HDFS-4236. Blocker bug reported by Allen Wittenauer and fixed by Alejandro Abdelnur
  Regression: HDFS-4171 puts artificial limit on username length
  

• HDFS-4232. Blocker bug reported by Daryn Sharp and fixed by Daryn Sharp (namenode)
  NN fails to write a fsimage with stale leases
  

• HDFS-4231. Major improvement reported by Konstantin Shvachko and fixed by Konstantin Shvachko (ha , namenode)
  Introduce HAState for BackupNode
  

• HDFS-4216. Major bug reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (namenode)
  Adding symlink should not ignore QuotaExceededException
  

• HDFS-4214. Trivial improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (tools)
  OfflineEditsViewer should print out the offset at which it encountered an error
  

• HDFS-4213. Major new feature reported by Jing Zhao and fixed by Jing Zhao (hdfs-client , namenode)
  When the client calls hsync, allows the client to update the file length in the NameNode
  

• HDFS-4199. Minor test reported by Ivan A. Veselovsky and fixed by Ivan A. Veselovsky
  Provide test for HdfsVolumeId
  

• HDFS-4186. Critical bug reported by Kihwal Lee and fixed by Kihwal Lee (namenode)
  logSync() is called with the write lock held while releasing lease
  

• HDFS-4182. Critical bug reported by Todd Lipcon and fixed by Robert Joseph Evans (namenode)
  SecondaryNameNode leaks NameCache entries
  

• HDFS-4181. Critical bug reported by Kihwal Lee and fixed by Kihwal Lee (namenode)
  LeaseManager tries to double remove and prints extra messages
  

• HDFS-4179. Major bug reported by Konstantin Shvachko and fixed by Konstantin Shvachko (namenode)
  BackupNode: allow reads, fix checkpointing, safeMode
  

• HDFS-4178. Major bug reported by Andy Isaacson and fixed by Andy Isaacson (scripts)
  shell scripts should not close stderr
  

• HDFS-4172. Minor bug reported by Derek Dagit and fixed by Derek Dagit (namenode)
  namenode does not URI-encode parameters when building URI for datanode request
  

• HDFS-4171. Major bug reported by Harsh J and fixed by Alejandro Abdelnur
  WebHDFS and HttpFs should accept only valid Unix user names
  

• HDFS-4164. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (fuse-dfs)
  fuse_dfs: add -lrt to the compiler command line on Linux
  

• HDFS-4162. Minor bug reported by Derek Dagit and fixed by Derek Dagit (datanode)
  Some malformed and unquoted HTML strings are returned from datanode web ui
  

• HDFS-4156. Major bug reported by Eli Collins and fixed by Eli Reisman
  Seeking to a negative position should throw an IOE
  

• HDFS-4155. Major improvement reported by Liang Xie and fixed by Liang Xie (libhdfs)
  libhdfs implementation of hsync API
  

• HDFS-4153. Major improvement reported by Liang Xie and fixed by Liang Xie (journal-node)
  Add START_MSG/SHUTDOWN_MSG for JournalNode
  

• HDFS-4143. Minor improvement reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (namenode)
  Change INodeFile.blocks to private
  

• HDFS-4140. Major bug reported by Andy Isaacson and fixed by Colin Patrick McCabe (fuse-dfs)
  fuse-dfs handles open(O_TRUNC) poorly
  

• HDFS-4139. Major bug reported by Andy Isaacson and fixed by Colin Patrick McCabe (fuse-dfs)
  fuse-dfs RO mode still allows file truncation
  

• HDFS-4132. Major bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (libhdfs)
  when libwebhdfs is not enabled, nativeMiniDfsClient frees uninitialized memory
  

• HDFS-4130. Major sub-task reported by Han Xiao and fixed by Han Xiao (ha , performance)
  BKJM: The reading for editlog at NN starting using bkjm is not efficient
  

• HDFS-4127. Minor bug reported by Junping Du and fixed by Junping Du (namenode)
  Log message is not correct in case of short of replica
  

• HDFS-4122. Major bug reported by Suresh Srinivas and fixed by Suresh Srinivas (datanode , hdfs-client , namenode)
  Cleanup HDFS logs and reduce the size of logged messages
  

  The change from this jira changes the content of some of the log messages. No log message are removed. Only the content of the log messages is changed to reduce the size. If you have a tool that depends on the exact content of the log, please look at the patch and make appropriate updates to the tool.

• HDFS-4121. Minor improvement reported by Binglin Chang and fixed by Binglin Chang
  Add namespace declarations in hdfs .proto files for languages other than java
  

• HDFS-4112. Major bug reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (namenode)
  A few improvements on INodeDirectory
  

• HDFS-4110. Trivial improvement reported by Liang Xie and fixed by Liang Xie (journal-node)
  Refine JNStorage log
  

• HDFS-4107. Major bug reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (namenode)
  Add utility methods to cast INode to INodeFile and INodeFileUnderConstruction
  

• HDFS-4106. Minor bug reported by Jing Zhao and fixed by Jing Zhao (namenode , test)
  BPServiceActor#lastHeartbeat, lastBlockReport and lastDeletedReport should be declared as volatile
  

• HDFS-4105. Major bug reported by Arpit Gupta and fixed by Arpit Gupta
  the SPNEGO user for secondary namenode should use the web keytab
  

• HDFS-4104. Minor bug reported by Andy Isaacson and fixed by Andy Isaacson
  dfs -test -d prints inappropriate error on nonexistent directory
  

• HDFS-4100. Major sub-task reported by Liang Xie and fixed by Liang Xie (datanode , journal-node , security)
  Fix all findbug security warings
  

• HDFS-4099. Minor bug reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (namenode)
  Clean up replication code and add more javadoc
  

• HDFS-4090. Critical bug reported by Kihwal Lee and fixed by Kihwal Lee (hdfs-client)
  getFileChecksum() result incompatible when called against zero-byte files.
  

• HDFS-4088. Minor improvement reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (namenode)
  Remove "throws QuotaExceededException" from an INodeDirectoryWithQuota constructor
  

• HDFS-4080. Major bug reported by Kihwal Lee and fixed by Kihwal Lee (namenode)
  Add a separate logger for block state change logs to enable turning off those logs
  

• HDFS-4075. Critical bug reported by Kihwal Lee and fixed by Kihwal Lee (namenode)
  Reduce recommissioning overhead
  

• HDFS-4074. Trivial improvement reported by Brandon Li and fixed by Brandon Li (namenode)
  Remove empty constructors for INode
  

• HDFS-4073. Minor improvement reported by Tsz Wo (Nicholas), SZE and fixed by Jing Zhao (namenode)
  Two minor improvements to FSDirectory
  

• HDFS-4072. Minor bug reported by Jing Zhao and fixed by Jing Zhao (namenode)
  On file deletion remove corresponding blocks pending replication
  

• HDFS-4068. Minor improvement reported by Eli Collins and fixed by Eli Collins (datanode)
  DatanodeID and DatanodeInfo member should be private
  

• HDFS-4061. Major bug reported by Eli Collins and fixed by Eli Collins
  TestBalancer and TestUnderReplicatedBlocks need timeouts
  

• HDFS-4059. Minor sub-task reported by Jing Zhao and fixed by Jing Zhao (datanode , namenode)
  Add number of stale DataNodes to metrics
  

  This jira adds a new metric with name "StaleDataNodes" under metrics context "dfs" of type Gauge. This tracks the number of DataNodes marked as stale. A DataNode is marked stale when the heartbeat message from the DataNode is not received within the configured time ""dfs.namenode.stale.datanode.interval". Please see hdfs-default.xml documentation corresponding to ""dfs.namenode.stale.datanode.interval" for more details on how to configure this feature. When this feature is not configured, this metrics would return zero.

• HDFS-4058. Major improvement reported by Eli Collins and fixed by Eli Collins (datanode)
  DirectoryScanner may fail with IOOB if the directory scanning threads return out of volume order
  

• HDFS-4055. Major bug reported by Binglin Chang and fixed by Binglin Chang
  TestAuditLogs is flaky
  

• HDFS-4049. Critical bug reported by Todd Lipcon and fixed by Todd Lipcon (datanode , performance)
  hflush performance regression due to nagling delays
  

• HDFS-4048. Major improvement reported by Stephen Chu and fixed by Stephen Chu
  Use ERROR instead of INFO for volume failure logs
  

• HDFS-4046. Minor bug reported by Binglin Chang and fixed by Binglin Chang (datanode , namenode)
  ChecksumTypeProto use NULL as enum value which is illegal in C/C++
  

• HDFS-4044. Major bug reported by Binglin Chang and fixed by Binglin Chang (datanode)
  Duplicate ChecksumType definition in HDFS .proto files
  

• HDFS-4041. Major improvement reported by Chris Nauroth and fixed by Chris Nauroth (build)
  Hadoop HDFS Maven protoc calls must not depend on external sh script
  

• HDFS-4038. Minor sub-task reported by Vinay and fixed by Vinay (ha)
  Override toString() for BookKeeperEditLogInputStream
  

• HDFS-4037. Major improvement reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (namenode)
  Rename the getReplication() method in BlockCollection to getBlockReplication()
  

• HDFS-4036. Major bug reported by Tsz Wo (Nicholas), SZE and fixed by Jing Zhao (namenode)
  FSDirectory.unprotectedAddFile(..) should not throw UnresolvedLinkException
  

• HDFS-4035. Major sub-task reported by Eli Collins and fixed by Eli Collins
  LightWeightGSet and LightWeightHashSet increment a volatile without synchronization
  

• HDFS-4034. Major sub-task reported by Eli Collins and fixed by Eli Collins
  Remove redundant null checks
  

• HDFS-4033. Major sub-task reported by Eli Collins and fixed by Eli Collins
  Miscellaneous findbugs 2 fixes
  

• HDFS-4032. Major sub-task reported by Eli Collins and fixed by Eli Collins
  Specify the charset explicitly rather than rely on the default
  

• HDFS-4031. Major sub-task reported by Eli Collins and fixed by Eli Collins (namenode)
  Update findbugsExcludeFile.xml to include findbugs 2 exclusions
  

• HDFS-4030. Major sub-task reported by Eli Collins and fixed by Eli Collins (namenode)
  BlockManager excessBlocksCount and postponedMisreplicatedBlocksCount should be AtomicLongs
  

• HDFS-4029. Major sub-task reported by Eli Collins and fixed by Eli Collins (namenode)
  GenerationStamp should use an AtomicLong
  

• HDFS-4022. Blocker bug reported by suja s and fixed by Vinay
  Replication not happening for appended block
  

• HDFS-4021. Minor bug reported by Colin Patrick McCabe and fixed by Christopher Conner (namenode)
  Misleading error message when resources are low on the NameNode
  

• HDFS-4020. Major bug reported by Eli Collins and fixed by Eli Collins
  TestRBWBlockInvalidation may time out
  

• HDFS-4018. Minor bug reported by Eli Collins and fixed by Eli Collins
  TestDataNodeMultipleRegistrations#testMiniDFSClusterWithMultipleNN is missing some cluster cleanup
  

• HDFS-4008. Minor improvement reported by Eli Collins and fixed by Eli Collins (test)
  TestBalancerWithEncryptedTransfer needs a timeout
  

• HDFS-4007. Minor test reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (test)
  Rehabilitate bit-rotted unit tests under hadoop-hdfs-project/hadoop-hdfs/src/test/unit/
  

• HDFS-4006. Major bug reported by Eli Collins and fixed by Todd Lipcon (namenode)
  TestCheckpoint#testSecondaryHasVeryOutOfDateImage occasionally fails due to unexpected exit
  

• HDFS-4000. Major improvement reported by Eli Collins and fixed by Colin Patrick McCabe
  TestParallelLocalRead fails with "input ByteBuffers must be direct buffers"
  

• HDFS-3999. Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur
  HttpFS OPEN operation expects len parameter, it should be length
  

• HDFS-3997. Trivial bug reported by Mithun Radhakrishnan and fixed by Mithun Radhakrishnan (namenode)
  OfflineImageViewer incorrectly passes value of imageVersion when visiting IS_COMPRESSED element
  

• HDFS-3996. Minor bug reported by Eli Collins and fixed by Eli Collins
  Add debug log removed in HDFS-3873 back
  

• HDFS-3992. Minor bug reported by Ivan A. Veselovsky and fixed by Ivan A. Veselovsky
  Method org.apache.hadoop.hdfs.TestHftpFileSystem.tearDown() sometimes throws NPEs
  

• HDFS-3990. Critical bug reported by Daryn Sharp and fixed by Daryn Sharp (namenode)
  NN's health report has severe performance problems
  

• HDFS-3985. Major bug reported by Eli Collins and fixed by (test)
  Add timeouts to TestMulitipleNNDataBlockScanner
  

• HDFS-3979. Major bug reported by Lars Hofhansl and fixed by Lars Hofhansl (datanode)
  Fix hsync semantics
  

• HDFS-3970. Major bug reported by Vinay and fixed by Andrew Wang (datanode)
  BlockPoolSliceStorage#doRollback(..) should use BlockPoolSliceStorage instead of DataStorage to read prev version file.
  

• HDFS-3964. Minor bug reported by Eli Collins and fixed by Eli Collins (namenode)
  Make NN log of fs.defaultFS debug rather than info
  

• HDFS-3957. Minor improvement reported by Andrew Wang and fixed by Andrew Wang
  Change MutableQuantiles to use a shared thread for rolling over metrics
  

• HDFS-3951. Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (datanode)
  datanode web ui does not work over HTTPS when datanode is started in secure mode
  

• HDFS-3949. Minor bug reported by Eli Collins and fixed by Eli Collins (namenode)
  NameNodeRpcServer#join should join on both client and server RPC servers
  

• HDFS-3948. Minor bug reported by Eli Collins and fixed by Jing Zhao (test)
  TestWebHDFS#testNamenodeRestart occasionally fails
  

• HDFS-3944. Major task reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur
  Httpfs resolveAuthority() is not resolving host correctly
  

• HDFS-3939. Minor improvement reported by Eli Collins and fixed by Eli Collins (namenode)
  NN RPC address cleanup
  

• HDFS-3938. Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (documentation)
  remove current limitations from HttpFS docs
  

• HDFS-3936. Major bug reported by Eli Collins and fixed by Eli Collins
  MiniDFSCluster shutdown races with BlocksMap usage
  

• HDFS-3935. Major sub-task reported by Eli Collins and fixed by Andy Isaacson
  QJM: Add JournalNode to the start / stop scripts
  

• HDFS-3932. Major bug reported by Eli Collins and fixed by Eli Collins
  NameNode Web UI broken if the rpc-address is set to the wildcard
  

• HDFS-3931. Minor bug reported by Eli Collins and fixed by Andy Isaacson (test)
  TestDatanodeBlockScanner#testBlockCorruptionPolicy2 is broken
  

• HDFS-3925. Minor improvement reported by Andrew Wang and fixed by Andrew Wang
  Prettify PipelineAck#toString() for printing to a log
  

• HDFS-3924. Major bug reported by Andrew Wang and fixed by Andrew Wang (hdfs-client)
  Multi-byte id in HdfsVolumeId
  

• HDFS-3923. Major sub-task reported by Jing Zhao and fixed by Jing Zhao
  libwebhdfs testing code cleanup
  

• HDFS-3921. Major bug reported by Stephen Chu and fixed by Aaron T. Myers
  NN will prematurely consider blocks missing when entering active state while still in safe mode
  

• HDFS-3920. Major sub-task reported by Jing Zhao and fixed by Jing Zhao
  libwebdhfs code cleanup: string processing and using strerror consistently to handle all errors
  

• HDFS-3919. Minor bug reported by Andy Isaacson and fixed by Andy Isaacson (test)
  MiniDFSCluster:waitClusterUp can hang forever
  

• HDFS-3916. Major bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (webhdfs)
  libwebhdfs (C client) code cleanups
  

• HDFS-3912. Major sub-task reported by Jing Zhao and fixed by Jing Zhao
  Detecting and avoiding stale datanodes for writing
  

• HDFS-3910. Minor improvement reported by Eli Collins and fixed by Eli Collins (test)
  DFSTestUtil#waitReplication should timeout
  

• HDFS-3896. Minor improvement reported by Jeff Lord and fixed by Jeff Lord
  Add descriptions for dfs.namenode.rpc-address and dfs.namenode.servicerpc-address to hdfs-default.xml
  

• HDFS-3831. Critical bug reported by Jason Lowe and fixed by Jason Lowe (security)
  Failure to renew tokens due to test-sources left in classpath
  

• HDFS-3829. Major bug reported by Trevor Robinson and fixed by Trevor Robinson (test)
  TestHftpURLTimeouts fails intermittently with JDK7
  

• HDFS-3824. Major bug reported by Trevor Robinson and fixed by Trevor Robinson (test)
  TestHftpDelegationToken fails intermittently with JDK7
  

• HDFS-3813. Major improvement reported by Stephen Chu and fixed by Stephen Chu (security , webhdfs)
  Log error message if security and WebHDFS are enabled but principal/keytab are not configured
  

• HDFS-3810. Major sub-task reported by Ivan Kelly and fixed by Ivan Kelly
  Implement format() for BKJM
  

• HDFS-3809. Major sub-task reported by Ivan Kelly and fixed by Ivan Kelly (namenode)
  Make BKJM use protobufs for all serialization with ZK
  

• HDFS-3804. Major bug reported by Trevor Robinson and fixed by Trevor Robinson (test)
  TestHftpFileSystem fails intermittently with JDK7
  

• HDFS-3789. Major sub-task reported by Ivan Kelly and fixed by Ivan Kelly (ha , namenode)
  JournalManager#format() should be able to throw IOException
  

• HDFS-3753. Major bug reported by Eli Collins and fixed by Colin Patrick McCabe (build , test)
  Tests don't run with native libraries
  

• HDFS-3703. Major improvement reported by nkeywal and fixed by Jing Zhao (datanode , namenode)
  Decrease the datanode failure detection time
  

  This jira adds a new DataNode state called "stale" at the NameNode. DataNodes are marked as stale if it does not send heartbeat message to NameNode within the timeout configured using the configuration parameter "dfs.namenode.stale.datanode.interval" in seconds (default value is 30 seconds). NameNode picks a stale datanode as the last target to read from when returning block locations for reads. This feature is by default turned * off *. To turn on the feature, set the HDFS configuration "dfs.namenode.check.stale.datanode" to true.

• HDFS-3695. Major sub-task reported by Todd Lipcon and fixed by Todd Lipcon (ha , namenode)
  Genericize format() to non-file JournalManagers
  

• HDFS-3682. Minor improvement reported by Eli Collins and fixed by Todd Lipcon (test)
  MiniDFSCluster#init should provide more info when it fails
  

• HDFS-3680. Minor improvement reported by Marcelo Vanzin and fixed by Marcelo Vanzin (namenode)
  Allow customized audit logging in HDFS FSNamesystem
  

• HDFS-3678. Critical bug reported by Todd Lipcon and fixed by Aaron T. Myers (namenode)
  Edit log files are never being purged from 2NN
  

• HDFS-3626. Blocker bug reported by Todd Lipcon and fixed by Todd Lipcon (namenode)
  Creating file with invalid path can corrupt edit log
  

• HDFS-3623. Major sub-task reported by Uma Maheswara Rao G and fixed by Uma Maheswara Rao G (namenode)
  BKJM: zkLatchWaitTimeout hard coded to 6000. Make use of ZKSessionTimeout instead.
  

• HDFS-3616. Major bug reported by Uma Maheswara Rao G and fixed by Jing Zhao (datanode)
  TestWebHdfsWithMultipleNameNodes fails with ConcurrentModificationException in DN shutdown
  

• HDFS-3598. Major new feature reported by Tsz Wo (Nicholas), SZE and fixed by Plamen Jeliazkov (webhdfs)
  WebHDFS: support file concat
  

• HDFS-3573. Minor sub-task reported by Todd Lipcon and fixed by Todd Lipcon (namenode)
  Supply NamespaceInfo when instantiating JournalManagers
  

• HDFS-3571. Major sub-task reported by Todd Lipcon and fixed by Todd Lipcon (ha , namenode)
  Allow EditLogFileInputStream to read from a remote URL
  

• HDFS-3553. Blocker bug reported by Daryn Sharp and fixed by Daryn Sharp
  Hftp proxy tokens are broken
  

• HDFS-3510. Major bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  Improve FSEditLog pre-allocation
  

• HDFS-3507. Critical bug reported by Vinay and fixed by Vinay (ha)
  DFS#isInSafeMode needs to execute only on Active NameNode
  

• HDFS-3483. Major improvement reported by Stephen Chu and fixed by Stephen Fritz
  Better error message when hdfs fsck is run against a ViewFS config
  

• HDFS-3429. Major bug reported by Todd Lipcon and fixed by Todd Lipcon (datanode , performance)
  DataNode reads checksums even if client does not need them
  

• HDFS-3373. Major bug reported by Todd Lipcon and fixed by John George (hdfs-client)
  FileContext HDFS implementation can leak socket caches
  

• HDFS-3224. Minor bug reported by Eli Collins and fixed by Jason Lowe
  Bug in check for DN re-registration with different storage ID
  

• HDFS-3077. Major new feature reported by Todd Lipcon and fixed by Todd Lipcon (ha , namenode)
  Quorum-based protocol for reading and writing edit logs
  

• HDFS-3049. Minor new feature reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (namenode)
  During the normal loading NN startup process, fall back on a different EditLog if we see one that is corrupt
  

• HDFS-2946. Major improvement reported by Aaron T. Myers and fixed by Aaron T. Myers (ha , namenode)
  HA: Put a cap on the number of completed edits files retained by the NN
  

• HDFS-2908. Minor sub-task reported by Suresh Srinivas and fixed by Brandon Li
  Add apache license header for StorageReport.java
  

• HDFS-2656. Major improvement reported by Zhanwei.Wang and fixed by Jing Zhao (webhdfs)
  Implement a pure c client based on webhdfs
  

• HDFS-2264. Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (namenode)
  NamenodeProtocol has the wrong value for clientPrincipal in KerberosInfo annotation
  

• HDFS-1331. Minor bug reported by Allen Wittenauer and fixed by Andy Isaacson (tools)
  dfs -test should work like /bin/test
  

  "test" will not print a warning for non-existent paths when testing for existence

• HDFS-1322. Major bug reported by Ravi Gummadi and fixed by Colin Patrick McCabe
  Document umask in DistributedFileSystem#mkdirs javadocs
  

• HDFS-1245. Major new feature reported by Dmytro Molkov and fixed by Konstantin Shvachko (namenode)
  Plugable block id generation
  

• HADOOP-9289. Blocker bug reported by Daryn Sharp and fixed by Daryn Sharp (fs)
  FsShell rm -f fails for non-matching globs
  

• HADOOP-9278. Major bug reported by Chris Nauroth and fixed by Chris Nauroth (fs)
  HarFileSystem may leak file handle
  

• HADOOP-9276. Minor improvement reported by Arun C Murthy and fixed by Arun C Murthy
  Allow BoundedByteArrayOutputStream to be resettable
  

• HADOOP-9260. Critical bug reported by Jerry Chen and fixed by Chris Nauroth
  Hadoop version may be not correct when starting name node or data node
  

• HADOOP-9255. Critical bug reported by Thomas Graves and fixed by Thomas Graves (scripts)
  relnotes.py missing last jira
  

• HADOOP-9252. Minor bug reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (util)
  StringUtils.humanReadableInt(..) has a race condition
  

• HADOOP-9247. Minor improvement reported by Ivan A. Veselovsky and fixed by Ivan A. Veselovsky
  parametrize Clover "generateXxx" properties to make them re-definable via -D in mvn calls
  

• HADOOP-9231. Major bug reported by Konstantin Boudnik and fixed by Konstantin Boudnik (build)
  Parametrize staging URL for the uniformity of distributionManagement
  

• HADOOP-9221. Major bug reported by Andy Isaacson and fixed by Andy Isaacson
  Convert remaining xdocs to APT
  

• HADOOP-9217. Major test reported by Andrey Klochkov and fixed by Andrey Klochkov (test)
  Print thread dumps when hadoop-common tests fail
  

• HADOOP-9216. Major improvement reported by Tsuyoshi OZAWA and fixed by Tsuyoshi OZAWA (io)
  CompressionCodecFactory#getCodecClasses should trim the result of parsing by Configuration.
  

• HADOOP-9215. Blocker bug reported by Thomas Graves and fixed by Colin Patrick McCabe
  when using cmake-2.6, libhadoop.so doesn't get created (only libhadoop.so.1.0.0)
  

• HADOOP-9212. Major bug reported by Tom White and fixed by Tom White (fs)
  Potential deadlock in FileSystem.Cache/IPC/UGI
  

• HADOOP-9203. Trivial bug reported by Andrew Purtell and fixed by Andrew Purtell (ipc , test)
  RPCCallBenchmark should find a random available port
  

• HADOOP-9193. Minor bug reported by Jason Lowe and fixed by Andy Isaacson (scripts)
  hadoop script can inadvertently expand wildcard arguments when delegating to hdfs script
  

• HADOOP-9192. Major improvement reported by Suresh Srinivas and fixed by Suresh Srinivas (security)
  Move token related request/response messages to common
  

• HADOOP-9190. Major bug reported by Thomas Graves and fixed by Andy Isaacson (documentation)
  packaging docs is broken
  

• HADOOP-9183. Major bug reported by Tom White and fixed by Tom White (ha)
  Potential deadlock in ActiveStandbyElector
  

• HADOOP-9181. Major bug reported by Liang Xie and fixed by Liang Xie
  Set daemon flag for HttpServer's QueuedThreadPool
  

• HADOOP-9178. Minor bug reported by Sandy Ryza and fixed by Sandy Ryza
  src/main/conf is missing hadoop-policy.xml
  

• HADOOP-9173. Major bug reported by Suresh Srinivas and fixed by Suresh Srinivas
  Add security token protobuf definition to common and use it in hdfs
  

• HADOOP-9162. Minor improvement reported by Binglin Chang and fixed by Binglin Chang (native)
  Add utility to check native library availability
  

• HADOOP-9155. Minor bug reported by Binglin Chang and fixed by Binglin Chang
  FsPermission should have different default value, 777 for directory and 666 for file
  

• HADOOP-9153. Major improvement reported by Sandy Ryza and fixed by Sandy Ryza (viewfs)
  Support createNonRecursive in ViewFileSystem
  

• HADOOP-9152. Minor bug reported by Brock Noland and fixed by Brock Noland (fs)
  HDFS can report negative DFS Used on clusters with very small amounts of data
  

• HADOOP-9147. Trivial improvement reported by Jonathan Allen and fixed by Jonathan Allen
  Add missing fields to FIleStatus.toString
  

  Update FileStatus.toString to include missing fields

• HADOOP-9135. Trivial bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (security)
  JniBasedUnixGroupsMappingWithFallback should log at debug rather than info during fallback
  

• HADOOP-9127. Major improvement reported by Daisuke Kobayashi and fixed by Daisuke Kobayashi (documentation)
  Update documentation for ZooKeeper Failover Controller
  

• HADOOP-9119. Minor test reported by Steve Loughran and fixed by Steve Loughran (fs , test)
  Add test to FileSystemContractBaseTest to verify integrity of overwritten files
  

  Patches adds more tests to verify overwritten and more complex operations -write-delete-overwrite. By using differently sized datasets and different data inside, these tests verify that the overwrite really did take place. While HDFS meets all these requirements directly, eventually consistent object stores may not -hence these tests.

• HADOOP-9118. Trivial improvement reported by Steve Loughran and fixed by (test)
  FileSystemContractBaseTest test data for read/write isn't rigorous enough
  

  Resolved as part of HADOOP-9119 -it's test data generator creates more bits in every test byte

• HADOOP-9113. Major bug reported by Karthik Kambatla and fixed by Karthik Kambatla (security , test)
  o.a.h.fs.TestDelegationTokenRenewer is failing intermittently
  

• HADOOP-9106. Major improvement reported by Todd Lipcon and fixed by Robert Parker (ipc)
  Allow configuration of IPC connect timeout
  

  This jira introduces a new configuration parameter "ipc.client.connect.timeout". This configuration defines the Hadoop RPC connection timeout in milliseconds for a client to connect to a server. For details see the description associated with this configuration in core-default.xml.

• HADOOP-9105. Major bug reported by Daryn Sharp and fixed by Daryn Sharp (fs)
  FsShell -moveFromLocal erroneously fails
  

• HADOOP-9103. Major bug reported by yixiaohua and fixed by Todd Lipcon (io)
  UTF8 class does not properly decode Unicode characters outside the basic multilingual plane
  

• HADOOP-9097. Critical bug reported by Tom White and fixed by Thomas Graves (build)
  Maven RAT plugin is not checking all source files
  

• HADOOP-9093. Major improvement reported by Suresh Srinivas and fixed by Suresh Srinivas
  Move all the Exception in PathExceptions to o.a.h.fs package
  

• HADOOP-9090. Minor new feature reported by Mostafa Elhemali and fixed by Mostafa Elhemali (metrics)
  Support on-demand publish of metrics
  

• HADOOP-9072. Major bug reported by Robert Parker and fixed by Robert Parker
  Hadoop-Common-0.23-Build Fails to build in Jenkins
  

• HADOOP-9070. Blocker sub-task reported by Daryn Sharp and fixed by Daryn Sharp (ipc)
  Kerberos SASL server cannot find kerberos key
  

• HADOOP-9067. Minor test reported by Ivan A. Veselovsky and fixed by Ivan A. Veselovsky
  provide test for method org.apache.hadoop.fs.LocalFileSystem.reportChecksumFailure(Path, FSDataInputStream, long, FSDataInputStream, long)
  

• HADOOP-9064. Major bug reported by Karthik Kambatla and fixed by Karthik Kambatla (security)
  Augment DelegationTokenRenewer API to cancel the tokens on calls to removeRenewAction
  

• HADOOP-9054. Major new feature reported by Robert Kanter and fixed by Robert Kanter (security)
  Add AuthenticationHandler that uses Kerberos but allows for an alternate form of authentication for browsers
  

• HADOOP-9049. Major bug reported by Karthik Kambatla and fixed by Karthik Kambatla (security)
  DelegationTokenRenewer needs to be Singleton and FileSystems should register/deregister to/from.
  

• HADOOP-9042. Minor test reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  Add a test for umask in FileSystemContractBaseTest
  

• HADOOP-9041. Critical bug reported by Radim Kolar and fixed by Radim Kolar (fs)
  FileSystem initialization can go into infinite loop
  

• HADOOP-9038. Minor test reported by Ivan A. Veselovsky and fixed by Ivan A. Veselovsky
  provide unit-test coverage of class org.apache.hadoop.fs.LocalDirAllocator.AllocatorPerContext.PathIterator
  

• HADOOP-9035. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (security)
  Generalize setup of LoginContext
  

• HADOOP-9025. Major bug reported by Robert Joseph Evans and fixed by Jonathan Eagles
  org.apache.hadoop.tools.TestCopyListing failing
  

• HADOOP-9022. Major bug reported by Haiyang Jiang and fixed by Jonathan Eagles
  Hadoop distcp tool fails to copy file if -m 0 specified
  

• HADOOP-9021. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (ipc , security)
  Enforce configured SASL method on the server
  

• HADOOP-9020. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (ipc , security)
  Add a SASL PLAIN server
  

• HADOOP-9015. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (ipc)
  Standardize creation of SaslRpcServers
  

• HADOOP-9014. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (ipc)
  Standardize creation of SaslRpcClients
  

• HADOOP-9013. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (fs , security)
  UGI should not hardcode loginUser's authenticationType
  

• HADOOP-9012. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (ipc)
  IPC Client sends wrong connection context
  

• HADOOP-9010. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (fs , security)
  Map UGI authenticationMethod to RPC authMethod
  

• HADOOP-9009. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (fs , security)
  Add SecurityUtil methods to get/set authentication method
  

• HADOOP-9004. Major improvement reported by Stephen Chu and fixed by Stephen Chu (security , test)
  Allow security unit tests to use external KDC
  

• HADOOP-8999. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (ipc)
  SASL negotiation is flawed
  

  The RPC SASL negotiation now always ends with final response. If the SASL mechanism does not have a final response (GSSAPI, PLAIN), then an empty success response is sent to the client. The client will now always expect a final response to definitively know if negotiation is complete/successful.

• HADOOP-8998. Minor improvement reported by Andy Isaacson and fixed by Alejandro Abdelnur
  set Cache-Control no-cache header on all dynamic content
  

• HADOOP-8994. Minor bug reported by Andy Isaacson and fixed by Andy Isaacson (test)
  TestDFSShell creates file named "noFileHere", making further tests hard to understand
  

• HADOOP-8992. Minor improvement reported by Ivan A. Veselovsky and fixed by Ivan A. Veselovsky
  Enhance unit-test coverage of class HarFileSystem
  

• HADOOP-8986. Critical bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans (ipc)
  Server$Call object is never released after it is sent
  

• HADOOP-8985. Minor improvement reported by Binglin Chang and fixed by Binglin Chang (ha , ipc)
  Add namespace declarations in .proto files for languages other than java
  

• HADOOP-8981. Major bug reported by Chris Nauroth and fixed by Xuan Gong (metrics)
  TestMetricsSystemImpl fails on Windows
  

• HADOOP-8962. Critical bug reported by Jason Lowe and fixed by Jason Lowe (fs)
  RawLocalFileSystem.listStatus fails when a child filename contains a colon
  

• HADOOP-8951. Minor improvement reported by Steve Loughran and fixed by Steve Loughran (util)
  RunJar to fail with user-comprehensible error message if jar missing
  

• HADOOP-8948. Major bug reported by Chris Nauroth and fixed by Chris Nauroth (test)
  TestFileUtil.testGetDU fails on Windows due to incorrect assumption of line separator
  

• HADOOP-8932. Major improvement reported by Kihwal Lee and fixed by Kihwal Lee (security)
  JNI-based user-group mapping modules can be too chatty on lookup failures
  

• HADOOP-8931. Trivial improvement reported by Eli Collins and fixed by Eli Collins
  Add Java version to startup message
  

• HADOOP-8930. Major improvement reported by Andrey Klochkov and fixed by Andrey Klochkov (test)
  Cumulative code coverage calculation
  

• HADOOP-8929. Major improvement reported by Todd Lipcon and fixed by Todd Lipcon (metrics)
  Add toString, other improvements for SampleQuantiles
  

• HADOOP-8926. Major improvement reported by Gopal V and fixed by Gopal V (util)
  hadoop.util.PureJavaCrc32 cache hit-ratio is low for static data
  

  Speed up Crc32 by improving the cache hit-ratio of hadoop.util.PureJavaCrc32

• HADOOP-8925. Minor improvement reported by Eli Collins and fixed by Eli Collins (build)
  Remove the packaging
  

• HADOOP-8922. Trivial improvement reported by Damien Hardy and fixed by Damien Hardy (metrics)
  Provide alternate JSONP output for JMXJsonServlet to allow javascript in browser dashboard
  

  Add a JSONP alternative outpout for /jmx HTTP interface to provide a Javascript polling ability in browsers.

• HADOOP-8913. Minor bug reported by Sandy Ryza and fixed by Sandy Ryza (metrics)
  hadoop-metrics2.properties should give units in comment for sampling period
  

• HADOOP-8912. Major bug reported by Raja Aluri and fixed by Raja Aluri (build)
  adding .gitattributes file to prevent CRLF and LF mismatches for source and text files
  

• HADOOP-8911. Major bug reported by Raja Aluri and fixed by Raja Aluri (build)
  CRLF characters in source and text files
  

• HADOOP-8909. Major improvement reported by Chris Nauroth and fixed by Chris Nauroth (build)
  Hadoop Common Maven protoc calls must not depend on external sh script
  

• HADOOP-8906. Critical bug reported by Daryn Sharp and fixed by Daryn Sharp (fs)
  paths with multiple globs are unreliable
  

• HADOOP-8901. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (native)
  GZip and Snappy support may not work without unversioned libraries
  

• HADOOP-8900. Major bug reported by Slavik Krassovsky and fixed by Andy Isaacson
  BuiltInGzipDecompressor throws IOException - stored gzip size doesn't match decompressed size
  

• HADOOP-8894. Major improvement reported by Todd Lipcon and fixed by Todd Lipcon
  GenericTestUtils.waitFor should dump thread stacks on timeout
  

• HADOOP-8889. Major improvement reported by Todd Lipcon and fixed by Todd Lipcon (build , test)
  Upgrade to Surefire 2.12.3
  

• HADOOP-8883. Major bug reported by Robert Kanter and fixed by Robert Kanter
  Anonymous fallback in KerberosAuthenticator is broken
  

• HADOOP-8881. Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  FileBasedKeyStoresFactory initialization logging should be debug not info
  

• HADOOP-8878. Major bug reported by Arpit Gupta and fixed by Arpit Gupta
  uppercase namenode hostname causes hadoop dfs calls with webhdfs filesystem and fsck to fail when security is on
  

• HADOOP-8866. Minor improvement reported by Andrew Wang and fixed by Andrew Wang
  SampleQuantiles#query is O(N^2) instead of O(N)
  

• HADOOP-8860. Major task reported by Tom White and fixed by Tom White (documentation)
  Split MapReduce and YARN sections in documentation navigation
  

• HADOOP-8855. Minor bug reported by Todd Lipcon and fixed by Todd Lipcon (security)
  SSL-based image transfer does not work when Kerberos is disabled
  

• HADOOP-8851. Minor improvement reported by Ivan A. Veselovsky and fixed by Ivan A. Veselovsky (test)
  Use -XX:+HeapDumpOnOutOfMemoryError JVM option in the forked tests
  

• HADOOP-8849. Minor improvement reported by Ivan A. Veselovsky and fixed by Ivan A. Veselovsky
  FileUtil#fullyDelete should grant the target directories +rwx permissions before trying to delete them
  

• HADOOP-8843. Critical bug reported by Robert Joseph Evans and fixed by Jason Lowe
  Old trash directories are never deleted on upgrade from 1.x
  

• HADOOP-8833. Major bug reported by Harsh J and fixed by Harsh J (fs)
  fs -text should make sure to call inputstream.seek(0) before using input stream
  

• HADOOP-8822. Major bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans
  relnotes.py was deleted post mavenization
  

• HADOOP-8819. Major bug reported by Brandon Li and fixed by Brandon Li (fs)
  Should use && instead of & in a few places in FTPFileSystem,FTPInputStream,S3InputStream,ViewFileSystem,ViewFs
  

• HADOOP-8816. Major bug reported by Moritz Moeller and fixed by Moritz Moeller (net)
  HTTP Error 413 full HEAD if using kerberos authentication
  

• HADOOP-8812. Minor improvement reported by Eli Collins and fixed by Eli Collins
  ExitUtil#terminate should print Exception#toString
  

• HADOOP-8811. Critical bug reported by Radim Kolar and fixed by Radim Kolar (native)
  Compile hadoop native library in FreeBSD
  

• HADOOP-8806. Minor improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (build)
  libhadoop.so: dlopen should be better at locating libsnappy.so, etc.
  

• HADOOP-8804. Minor improvement reported by Eli Collins and fixed by Senthil V Kumar
  Improve Web UIs when the wildcard address is used
  

• HADOOP-8795. Minor bug reported by Sean Mackrory and fixed by Sean Mackrory (scripts)
  BASH tab completion doesn't look in PATH, assumes path to executable is specified
  

• HADOOP-8791. Major bug reported by Bertrand Dechoux and fixed by Jing Zhao (documentation)
  rm "Only deletes non empty directory and files."
  

• HADOOP-8789. Minor improvement reported by Andy Isaacson and fixed by Andy Isaacson (test)
  Tests setLevel(Level.OFF) should be Level.ERROR
  

• HADOOP-8786. Major bug reported by Todd Lipcon and fixed by Todd Lipcon
  HttpServer continues to start even if AuthenticationFilter fails to init
  

• HADOOP-8784. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (ipc , security)
  Improve IPC.Client's token use
  

• HADOOP-8783. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (ipc , security)
  Improve RPC.Server's digest auth
  

• HADOOP-8780. Major bug reported by Ahmed Radwan and fixed by Ahmed Radwan
  Update DeprecatedProperties apt file
  

• HADOOP-8756. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (native)
  Fix SEGV when libsnappy is in java.library.path but not LD_LIBRARY_PATH
  

• HADOOP-8755. Major improvement reported by Andrey Klochkov and fixed by Andrey Klochkov (test)
  Print thread dump when tests fail due to timeout
  

• HADOOP-8736. Major improvement reported by Brandon Li and fixed by Brandon Li (ipc)
  Add Builder for building an RPC server
  

• HADOOP-8713. Major bug reported by Trevor Robinson and fixed by Trevor Robinson (test)
  TestRPCCompatibility fails intermittently with JDK7
  

• HADOOP-8712. Minor improvement reported by Robert Parker and fixed by Robert Parker (security)
  Change default hadoop.security.group.mapping
  

  The default group mapping policy has been changed to JniBasedUnixGroupsNetgroupMappingWithFallback. This should maintain the same semantics as the prior default for most users.

• HADOOP-8684. Minor bug reported by Hiroshi Ikeda and fixed by Jing Zhao (io)
  Deadlock between WritableComparator and WritableComparable
  

• HADOOP-8616. Major bug reported by Eli Collins and fixed by Sandy Ryza (viewfs)
  ViewFS configuration requires a trailing slash
  

• HADOOP-8597. Major new feature reported by Harsh J and fixed by Ivan Vladimirov Ivanov (fs)
  FsShell's Text command should be able to read avro data files
  

• HADOOP-8589. Major bug reported by Andrey Klochkov and fixed by Sanjay Radia (fs , test)
  ViewFs tests fail when tests and home dirs are nested
  

• HADOOP-8561. Major improvement reported by Luke Lu and fixed by Yu Gao (security)
  Introduce HADOOP_PROXY_USER for secure impersonation in child hadoop client processes
  

• HADOOP-8427. Major task reported by Eli Collins and fixed by Andy Isaacson (documentation)
  Convert Forrest docs to APT, incremental
  

• HADOOP-8418. Major bug reported by Luke Lu and fixed by Yu Gao (security)
  Fix UGI for IBM JDK running on Windows
  

• HADOOP-7886. Minor improvement reported by Jakob Homan and fixed by SreeHari
  Add toString to FileStatus
  

• HADOOP-7688. Major improvement reported by Tsz Wo (Nicholas), SZE and fixed by Uma Maheswara Rao G
  When a servlet filter throws an exception in init(..), the Jetty server failed silently.
  

• HADOOP-7115. Major bug reported by Arun C Murthy and fixed by Alejandro Abdelnur
  Add a cache for getpwuid_r and getpwgid_r calls
  

• HADOOP-6762. Critical bug reported by sam rash and fixed by sam rash
  exception while doing RPC I/O closes channel
  

• HADOOP-6607. Minor bug reported by Steve Loughran and fixed by Alejandro Abdelnur (io)
  Add different variants of non caching HTTP headers
  

Hadoop 2.0.2-alpha Release Notes

• YARN-137. Major improvement reported by Siddharth Seth and fixed by Siddharth Seth (scheduler)
  Change the default scheduler to the CapacityScheduler
  

  There's some bugs in the FifoScheduler atm - doesn't distribute tasks across nodes and some headroom (available resource) issues. That's not the best experience for users trying out the 2.0 branch. The CS with the default configuration of a single queue behaves the same as the FifoScheduler and doesn't have these issues.

• YARN-108. Critical bug reported by Jason Lowe and fixed by Jason Lowe (nodemanager)
  FSDownload can create cache directories with the wrong permissions
  

  When the cluster is configured with a restrictive umask, e.g.: {{fs.permissions.umask-mode=0077}}, the nodemanager can end up creating directory entries in the public cache with the wrong permissions. The permissions can end up where only the nodemanager user can access files in the public cache, preventing jobs from running properly.

• YARN-106. Major bug reported by Jason Lowe and fixed by Jason Lowe (nodemanager)
  Nodemanager needs to set permissions of local directories
  

  If the nodemanager process is running with a restrictive default umask (e.g.: 0077) then it will create its local directories with permissions that are too restrictive to allow containers from other users to run.

• YARN-88. Major bug reported by Jason Lowe and fixed by Jason Lowe (nodemanager)
  DefaultContainerExecutor can fail to set proper permissions
  

  {{DefaultContainerExecutor}} can fail to set the proper permissions on its local directories if the cluster has been configured with a restrictive umask, e.g.: fs.permissions.umask-mode=0077. The configured umask ends up defeating the permissions requested by {{DefaultContainerExecutor}} when it creates directories.

• YARN-87. Critical bug reported by Jason Lowe and fixed by Jason Lowe (nodemanager)
  NM ResourceLocalizationService does not set permissions of local cache directories
  

  {{ResourceLocalizationService}} creates a file cache and user cache directory when it starts up but doesn't specify the permissions for them when they are created. If the cluster configs are set to limit the default permissions (e.g.: fs.permissions.umask-mode=0077 instead of the default 0022), then the cache directories are created with too-restrictive permissions and no jobs are able to run.

• YARN-83. Major bug reported by Bikas Saha and fixed by Bikas Saha (client)
  Change package of YarnClient to include apache
  

  Currently its org.hadoop.* instead of org.apache.hadoop.*

• YARN-80. Major improvement reported by Todd Lipcon and fixed by Arun C Murthy (capacityscheduler)
  Support delay scheduling for node locality in MR2's capacity scheduler
  

  The capacity scheduler in MR2 doesn't support delay scheduling for achieving node-level locality. So, jobs exhibit poor data locality even if they have good rack locality. Especially on clusters where disk throughput is much better than network capacity, this hurts overall job performance. We should optionally support node-level delay scheduling heuristics similar to what the fair scheduler implements in MR1.

• YARN-79. Major bug reported by Bikas Saha and fixed by Vinod Kumar Vavilapalli (client)
  Calling YarnClientImpl.close throws Exception
  

  The following exception is thrown =========== *org.apache.hadoop.HadoopIllegalArgumentException: Cannot close proxy - is not Closeable or does not provide closeable invocation handler class org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl* *at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:624)* *at org.hadoop.yarn.client.YarnClientImpl.stop(YarnClientImpl.java:102)* at org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.run(UnmanagedAMLauncher.java:336) at org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.testDSShell(TestUnmanagedAMLauncher.java:156) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31) at org.junit.runners.ParentRunner.run(ParentRunner.java:236) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74) ===========

• YARN-75. Major bug reported by Siddharth Seth and fixed by Siddharth Seth
  RMContainer should handle a RELEASE event while RUNNING
  

  An AppMaster can send a container release at any point. Currently this results in an exception, if this is done while the RM considers the container to be RUNNING. The event not being processed correctly also implies that these containers do not show up in the Completed Container List seen by the AM (AMRMProtocol). MR-3902 depends on this set being complete.

• YARN-68. Major bug reported by patrick white and fixed by Daryn Sharp (nodemanager)
  NodeManager will refuse to shutdown indefinitely due to container log aggregation
  

  The nodemanager is able to get into a state where containermanager.logaggregation.AppLogAggregatorImpl will apparently wait indefinitely for log aggregation to complete for an application, even if that application has abnormally terminated and is no longer present. Observed behavior is that an attempt to stop the nodemanager daemon will return but have no effect, the nm log continually displays messages similar to this: [Thread-1]2012-08-21 17:44:07,581 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Waiting for aggregation to complete for application_1345221477405_2733 The only recovery we found to work was to 'kill -9' the nm process. What exactly causes the NM to enter this state is unclear but we do see this behavior reliably when the NM has run a task which failed, for example when debugging oozie distcp actions and having a distcp map task fail, the NM that was running the container will now enter this state where a shutdown on said NM will never complete, 'never' in this case was waiting for 2 hours before killing the nodemanager process.

• YARN-66. Critical bug reported by Thomas Graves and fixed by Thomas Graves (nodemanager)
  aggregated logs permissions not set properly
  

  If the default file permissions are set to something restrictive - like 700, application logs get aggregated and created with those restrictive file permissions which doesn't allow the history server to serve them up. They need to be created with group readable similar to how log aggregation sets up the directory permissions.

• YARN-63. Major bug reported by Jason Lowe and fixed by Jason Lowe (resourcemanager)
  RMNodeImpl is missing valid transitions from the UNHEALTHY state
  

  The ResourceManager isn't properly handling nodes that have been marked UNHEALTHY when they are lost or are decommissioned.

• YARN-60. Blocker sub-task reported by Daryn Sharp and fixed by Vinod Kumar Vavilapalli (nodemanager)
  NMs rejects all container tokens after secret key rolls
  

  The NM's token secret manager will reject all container tokens after the secret key is activated which means the NM will not launch _any_ containers including AMs. The whole yarn cluster becomes inoperable in 1d.

• YARN-58. Critical bug reported by Daryn Sharp and fixed by Jason Lowe (nodemanager)
  NM leaks filesystems
  

  The NM is exhausting its fds because it's not closing fs instances when the app is finished.

• YARN-42. Major bug reported by Devaraj K and fixed by Devaraj K (nodemanager)
  Node Manager throws NPE on startup
  

  NM throws NPE on startup if it doesn't have persmission's on nm local dir's {code:xml} 2012-05-14 16:32:13,468 FATAL org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager org.apache.hadoop.yarn.YarnException: Failed to initialize LocalizationService at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.init(ResourceLocalizationService.java:202) at org.apache.hadoop.yarn.service.CompositeService.init(CompositeService.java:58) at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.init(ContainerManagerImpl.java:183) at org.apache.hadoop.yarn.service.CompositeService.init(CompositeService.java:58) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.init(NodeManager.java:166) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:268) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:284) Caused by: java.io.IOException: mkdir of /mrv2/tmp/nm-local-dir/usercache failed at org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:907) at org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:143) at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:189) at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:706) at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:703) at org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2325) at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:703) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.init(ResourceLocalizationService.java:188) ... 6 more 2012-05-14 16:32:13,472 INFO org.apache.hadoop.yarn.service.CompositeService: Error stopping org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler java.lang.NullPointerException at org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler.stop(NonAggregatingLogHandler.java:82) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:99) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:89) at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.stop(ContainerManagerImpl.java:266) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:99) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:89) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.stop(NodeManager.java:182) at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:122) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) {code}

• YARN-39. Critical sub-task reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli
  RM-NM secret-keys should be randomly generated and rolled every so often
  

  - RM should generate the master-key randomly - The master-key should roll every so often - NM should remember old expired keys so that already doled out container-requests can be satisfied.

• YARN-37. Minor bug reported by Jason Lowe and fixed by Mayank Bansal (resourcemanager)
  TestRMAppTransitions.testAppSubmittedKilled passes for the wrong reason
  

  TestRMAppTransitions#testAppSubmittedKilled causes an invalid event exception but the test doesn't catch the error since the final app state is still killed. Killed for the wrong reason, but the final state is the same.

• YARN-36. Blocker bug reported by Eli Collins and fixed by Radim Kolar
  branch-2.1.0-alpha doesn't build
  

  branch-2.1.0-alpha doesn't build due to the following. Per YARN-1 I updated the mvn version to be 2.1.0-SNAPSHOT, before I hit this issue it didn't compile due to the bogus version. {noformat} hadoop-branch-2.1.0-alpha $ mvn compile [INFO] Scanning for projects... [ERROR] The build could not read 1 project -> [Help 1] [ERROR] [ERROR] The project org.apache.hadoop:hadoop-yarn-project:2.1.0-SNAPSHOT (/home/eli/src/hadoop-branch-2.1.0-alpha/hadoop-yarn-project/pom.xml) has 1 error [ERROR] 'dependencies.dependency.version' for org.hsqldb:hsqldb:jar is missing. @ line 160, column 17 {noformat}

• YARN-31. Major bug reported by Thomas Graves and fixed by Thomas Graves
  TestDelegationTokenRenewer fails on jdk7
  

  TestDelegationTokenRenewer fails when run with jdk7. With JDK7, test methods run in an undefined order. Here it is expecting that testDTRenewal runs first but it no longer is.

• YARN-29. Major bug reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli (client)
  Add a yarn-client module
  

  I see that we are duplicating (some) code for talking to RM via client API. In this light, a yarn-client module will be useful so that clients of all frameworks can use/extend it. And that same module can be the destination for all the YARN's command line tools.

• YARN-27. Major bug reported by Ramya Sunil and fixed by Arun C Murthy
  Failed refreshQueues due to misconfiguration prevents further refreshing of queues
  

  Stumbled upon this problem while refreshing queues with incorrect configuration. The exact scenario was: 1. Added a new queue "newQueue" without defining its capacity. 2. "bin/mapred queue -refreshQueues" fails correctly with "Illegal capacity of -1 for queue root.newQueue" 3. However, after defining the capacity of "newQueue" followed by a second "bin/mapred queue -refreshQueues" throws "org.apache.hadoop.metrics2.MetricsException: Metrics source QueueMetrics,q0=root,q1=newQueue already exists!" Also see Hadoop:name=QueueMetrics,q0=root,q1=newQueue,service=ResourceManager metrics being available even though the queue was not added. The expected behavior would be to refresh the queues correctly and allow addition of "newQueue".

• YARN-25. Major bug reported by Thomas Graves and fixed by Robert Joseph Evans
  remove old aggregated logs
  

  Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period. It might make sense for job history server to remove them.

• YARN-22. Minor bug reported by Eli Collins and fixed by Mayank Bansal
  Using URI for yarn.nodemanager log dirs fails
  

  If I use URIs (eg file:///home/eli/hadoop/dirs) for yarn.nodemanager.log-dirs or yarn.nodemanager.remote-app-log-dir the container log servlet fails with an NPE (works if I remove the "file" scheme). Using a URI for yarn.nodemanager.local-dirs works.

• YARN-15. Critical bug reported by Alejandro Abdelnur and fixed by Arun C Murthy (nodemanager)
  YarnConfiguration DEFAULT_YARN_APPLICATION_CLASSPATH should be updated
  

  {code} /** * Default CLASSPATH for YARN applications. A comma-separated list of * CLASSPATH entries */ public static final String[] DEFAULT_YARN_APPLICATION_CLASSPATH = { "$HADOOP_CONF_DIR", "$HADOOP_COMMON_HOME/share/hadoop/common/*", "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*", "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*", "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*", "$YARN_HOME/share/hadoop/mapreduce/*", "$YARN_HOME/share/hadoop/mapreduce/lib/*"}; {code} It should have {{share/yarn/}} and MR should add the {{share/mapreduce/}} (another JIRA?)

• YARN-14. Major bug reported by Jason Lowe and fixed by Jason Lowe (nodemanager)
  Symlinks to peer distributed cache files no longer work
  

  Trying to create a symlink to another file that is specified for the distributed cache will fail to create the link. For example: hadoop jar ... -files "x,y,x#z" will localize the files x and y as x and y, but the z symlink for x will not be created. This is a regression from 1.x behavior.

• YARN-13. Critical bug reported by Todd Lipcon and fixed by
  Merge of yarn reorg into branch-2 copied trunk tree
  

  When the move of yarn from inside MR to the project root was merged into branch-2, it seems like the trunk code base was actually copied into the branch-2 branch, instead of a parallel move occurring. So, the poms in branch-2 show the version as 3.0.0-SNAPSHOT instead of a 2.x snapshot version. This is breaking the branch-2 build.

• YARN-12. Major bug reported by Junping Du and fixed by Junping Du (scheduler)
  Several Findbugs issues with new FairScheduler in YARN
  

  The good feature of FairScheduler is added recently to YARN. As recently PreCommit test from MAPREDUCE-4309, there are several bugs found by Findbugs related to FairScheduler: org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.shutdown() might ignore java.lang.Exception Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.logDisabled; locked 50% of time Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.queueMaxAppsDefault; locked 50% of time Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.userMaxAppsDefault; locked 50% of time The details are in:https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html#DE_MIGHT_IGNORE

• YARN-10. Major improvement reported by Arun C Murthy and fixed by Hitesh Shah
  dist-shell shouldn't have a (test) dependency on hadoop-mapreduce-client-core
  

  dist-shell shouldn't have a (test) dependency on hadoop-mapreduce-client-core, this should be removed.

• YARN-9. Major improvement reported by Arun C Murthy and fixed by Vinod Kumar Vavilapalli
  Rename YARN_HOME to HADOOP_YARN_HOME
  

  We should rename YARN_HOME to HADOOP_YARN_HOME to be consistent with rest of Hadoop sub-projects.

• YARN-1. Major task reported by Arun C Murthy and fixed by Arun C Murthy
  Move YARN out of hadoop-mapreduce
  

  Move YARN out of hadoop-mapreduce-project into hadoop-yarn-project in hadoop trunk

• MAPREDUCE-4691. Critical bug reported by Jason Lowe and fixed by Robert Joseph Evans (jobhistoryserver , mrv2)
  Historyserver can report "Unknown job" after RM says job has completed
  

• MAPREDUCE-4689. Major bug reported by Jason Lowe and fixed by Jason Lowe (client)
  JobClient.getMapTaskReports on failed job results in NPE
  

• MAPREDUCE-4649. Major bug reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli (jobhistoryserver)
  mr-jobhistory-daemon.sh needs to be updated post YARN-1
  

• MAPREDUCE-4647. Major bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans (mrv2)
  We should only unjar jobjar if there is a lib directory in it.
  

• MAPREDUCE-4646. Major bug reported by Jason Lowe and fixed by Jason Lowe (mrv2)
  client does not receive job diagnostics for failed jobs
  

• MAPREDUCE-4642. Major bug reported by Robert Kanter and fixed by Robert Kanter (test)
  MiniMRClientClusterFactory should not use job.setJar()
  

• MAPREDUCE-4641. Major bug reported by Jason Lowe and fixed by Jason Lowe (mrv2)
  Exception in commitJob marks job as successful in job history
  

• MAPREDUCE-4638. Major improvement reported by Arun C Murthy and fixed by Arun C Murthy
  MR AppMaster shouldn't rely on YARN_APPLICATION_CLASSPATH providing MR jars
  

• MAPREDUCE-4635. Major bug reported by Bikas Saha and fixed by Bikas Saha
  MR side of YARN-83. Changing package of YarnClient
  

• MAPREDUCE-4633. Critical bug reported by Thomas Graves and fixed by Thomas Graves (jobhistoryserver)
  history server doesn't set permissions on all subdirs
  

• MAPREDUCE-4629. Major bug reported by Karthik Kambatla and fixed by Karthik Kambatla
  Remove JobHistory.DEBUG_MODE
  

• MAPREDUCE-4614. Major improvement reported by Daryn Sharp and fixed by Daryn Sharp (client , task)
  Simplify debugging a job's tokens
  

• MAPREDUCE-4612. Critical bug reported by Thomas Graves and fixed by Thomas Graves
  job summary file permissions not set when its created
  

• MAPREDUCE-4611. Critical bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans
  MR AM dies badly when Node is decomissioned
  

• MAPREDUCE-4610. Major bug reported by Tom White and fixed by Tom White (mrv2)
  Support deprecated mapreduce.job.counters.limit property in MR2
  

• MAPREDUCE-4608. Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (build)
  hadoop-mapreduce-client is missing some dependencies
  

• MAPREDUCE-4604. Critical bug reported by Ravi Prakash and fixed by Ravi Prakash (mrv2)
  In mapred-default, mapreduce.map.maxattempts & mapreduce.reduce.maxattempts defaults are set to 4 as well as mapreduce.job.maxtaskfailures.per.tracker.
  

• MAPREDUCE-4600. Critical bug reported by Robert Joseph Evans and fixed by Daryn Sharp
  TestTokenCache.java from MRV1 no longer compiles
  

• MAPREDUCE-4580. Major bug reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli
  Change MapReduce to use the yarn-client module
  

• MAPREDUCE-4579. Major bug reported by Thomas Graves and fixed by Thomas Graves
  TestTaskAttempt fails jdk7
  

• MAPREDUCE-4577. Minor bug reported by Alejandro Abdelnur and fixed by Aaron T. Myers (test)
  HDFS-3672 broke TestCombineFileInputFormat.testMissingBlocks() test
  

• MAPREDUCE-4572. Major bug reported by Ahmed Radwan and fixed by Ahmed Radwan (tasktracker , webapps)
  Can not access user logs - Jetty is not configured by default to serve aliases/symlinks
  

• MAPREDUCE-4570. Minor bug reported by Ahmed Radwan and fixed by Ahmed Radwan (mrv2)
  ProcfsBasedProcessTree#constructProcessInfo() prints a warning if procfsDir/<pid>/stat is not found.
  

• MAPREDUCE-4569. Major bug reported by Thomas Graves and fixed by Thomas Graves
  TestHsWebServicesJobsQuery fails on jdk7
  

• MAPREDUCE-4562. Major bug reported by Jarek Jarcec Cecho and fixed by Jarek Jarcec Cecho
  Support for "FileSystemCounter" legacy counter group name for compatibility reasons is creating incorrect counter name
  

• MAPREDUCE-4511. Major improvement reported by Ahmed Radwan and fixed by Ahmed Radwan (mrv1 , mrv2 , performance)
  Add IFile readahead
  

• MAPREDUCE-4504. Major bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans (mrv2)
  SortValidator writes to wrong directory
  

• MAPREDUCE-4503. Major bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans (mrv2)
  Should throw InvalidJobConfException if duplicates found in cacheArchives or cacheFiles
  

• MAPREDUCE-4498. Critical bug reported by Robert Kanter and fixed by Robert Kanter (build , examples)
  Remove hsqldb jar from Hadoop runtime classpath
  

• MAPREDUCE-4496. Major bug reported by Jason Lowe and fixed by Jason Lowe (applicationmaster , mrv2)
  AM logs link is missing user name
  

• MAPREDUCE-4494. Major bug reported by Ahmed Radwan and fixed by Ahmed Radwan (mrv2 , test)
  TestFifoScheduler failing with Metrics source QueueMetrics,q0=default already exists!
  

• MAPREDUCE-4493. Critical bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans (mrv2)
  Distibuted Cache Compatability Issues
  

• MAPREDUCE-4492. Minor bug reported by Nishan Shetty and fixed by Mayank Bansal (mrv2)
  Configuring total queue capacity between 100.5 and 99.5 at perticular level is sucessfull
  

• MAPREDUCE-4484. Major bug reported by Ahmed Radwan and fixed by Ahmed Radwan (mrv2)
  Incorrect IS_MINI_YARN_CLUSTER property name in YarnConfiguration
  

• MAPREDUCE-4483. Major bug reported by John George and fixed by John George
  2.0 build does not work
  

• MAPREDUCE-4470. Major bug reported by Kihwal Lee and fixed by Ilya Katsov (test)
  Fix TestCombineFileInputFormat.testForEmptyFile
  

• MAPREDUCE-4467. Critical bug reported by Andrey Klochkov and fixed by Kihwal Lee (nodemanager)
  IndexCache failures due to missing synchronization
  

• MAPREDUCE-4465. Trivial bug reported by Bo Wang and fixed by Bo Wang
  Update description of yarn.nodemanager.address property
  

• MAPREDUCE-4457. Critical bug reported by Thomas Graves and fixed by Robert Joseph Evans (mrv2)
  mr job invalid transition TA_TOO_MANY_FETCH_FAILURE at FAILED
  

• MAPREDUCE-4456. Major bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans (mrv2)
  LocalDistributedCacheManager can get an ArrayIndexOutOfBounds when creating symlinks
  

• MAPREDUCE-4449. Major bug reported by Ahmed Radwan and fixed by Ahmed Radwan (mrv2)
  Incorrect MR_HISTORY_STORAGE property name in JHAdminConfig
  

• MAPREDUCE-4448. Critical bug reported by Jason Lowe and fixed by Jason Lowe (mrv2 , nodemanager)
  Nodemanager crashes upon application cleanup if aggregation failed to start
  

• MAPREDUCE-4447. Major bug reported by Eli Collins and fixed by Eli Collins (build)
  Remove aop from cruft from the ant build
  

• MAPREDUCE-4444. Blocker bug reported by Nathan Roberts and fixed by Jason Lowe (nodemanager)
  nodemanager fails to start when one of the local-dirs is bad
  

• MAPREDUCE-4441. Blocker bug reported by Karthik Kambatla and fixed by Karthik Kambatla
  Fix build issue caused by MR-3451
  

• MAPREDUCE-4440. Major bug reported by Arun C Murthy and fixed by Arun C Murthy
  Change SchedulerApp & SchedulerNode to be a minimal interface
  

• MAPREDUCE-4437. Critical bug reported by Jason Lowe and fixed by Jason Lowe (applicationmaster , mrv2)
  Race in MR ApplicationMaster can cause reducers to never be scheduled
  

• MAPREDUCE-4432. Trivial bug reported by Gabriel Reid and fixed by
  Confusing warning message when GenericOptionsParser is not used
  

• MAPREDUCE-4427. Major improvement reported by Bikas Saha and fixed by Bikas Saha
  Enable the RM to work with AM's that are not managed by it
  

• MAPREDUCE-4423. Critical bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans (mrv2)
  Potential infinite fetching of map output
  

• MAPREDUCE-4422. Major improvement reported by Arun C Murthy and fixed by Ahmed Radwan (nodemanager)
  YARN_APPLICATION_CLASSPATH needs a documented default value in YarnConfiguration
  

• MAPREDUCE-4419. Major bug reported by Nishan Shetty and fixed by Devaraj K (mrv2)
  ./mapred queue -info <queuename> -showJobs displays all the jobs irrespective of <queuename>
  

• MAPREDUCE-4417. Major new feature reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (mrv2 , security)
  add support for encrypted shuffle
  

• MAPREDUCE-4416. Critical bug reported by Kihwal Lee and fixed by Kihwal Lee (client , mrv2)
  Some tests fail if Clover is enabled
  

• MAPREDUCE-4408. Major improvement reported by Alejandro Abdelnur and fixed by Robert Kanter (mrv1 , mrv2)
  allow jobs to set a JAR that is in the distributed cached
  

• MAPREDUCE-4407. Major bug reported by Ahmed Radwan and fixed by Ahmed Radwan (build , mrv2)
  Add hadoop-yarn-server-tests-<version>-tests.jar to hadoop dist package
  

• MAPREDUCE-4406. Major bug reported by Ahmed Radwan and fixed by Ahmed Radwan (mrv2 , test)
  Users should be able to specify the MiniCluster ResourceManager and JobHistoryServer ports
  

• MAPREDUCE-4402. Major bug reported by Jason Lowe and fixed by Jason Lowe (test)
  TestFileInputFormat fails intermittently
  

• MAPREDUCE-4395. Critical bug reported by Bhallamudi Venkata Siva Kamesh and fixed by Bhallamudi Venkata Siva Kamesh (distributed-cache , job submission , mrv2)
  Possible NPE at ClientDistributedCacheManager#determineTimestamps
  

• MAPREDUCE-4392. Major bug reported by Jason Lowe and fixed by Jason Lowe (mrv2)
  Counters.makeCompactString() changed behavior from 0.20
  

• MAPREDUCE-4387. Major bug reported by Kihwal Lee and fixed by Kihwal Lee (resourcemanager)
  RM gets fatal error and exits during TestRM
  

• MAPREDUCE-4384. Major bug reported by Kihwal Lee and fixed by Kihwal Lee (nodemanager)
  Race conditions in IndexCache
  

• MAPREDUCE-4383. Minor bug reported by Andy Isaacson and fixed by Andy Isaacson (pipes)
  HadoopPipes.cc needs to include unistd.h
  

• MAPREDUCE-4380. Minor bug reported by Devaraj K and fixed by Devaraj K (mrv2 , nodemanager)
  Empty Userlogs directory is getting created under logs directory
  

• MAPREDUCE-4379. Blocker bug reported by Devaraj K and fixed by Devaraj K (mrv2 , nodemanager)
  Node Manager throws java.lang.OutOfMemoryError: Java heap space due to org.apache.hadoop.fs.LocalDirAllocator.contexts
  

• MAPREDUCE-4376. Major bug reported by Jason Lowe and fixed by Kihwal Lee (mrv2 , test)
  TestClusterMRNotification times out
  

• MAPREDUCE-4375. Major improvement reported by Robert Joseph Evans and fixed by Robert Joseph Evans (applicationmaster)
  Show Configuration Tracability in MR UI
  

• MAPREDUCE-4372. Major bug reported by Devaraj K and fixed by Devaraj K (mrv2 , resourcemanager)
  Deadlock in Resource Manager between SchedulerEventDispatcher.EventProcessor and Shutdown hook manager
  

• MAPREDUCE-4361. Major bug reported by Jason Lowe and fixed by Jason Lowe (mrv2)
  Fix detailed metrics for protobuf-based RPC on 0.23
  

• MAPREDUCE-4355. Major new feature reported by Karthik Kambatla and fixed by Karthik Kambatla (mrv1 , mrv2)
  Add RunningJob.getJobStatus()
  

• MAPREDUCE-4341. Major bug reported by Thomas Graves and fixed by Karthik Kambatla (capacity-sched , mrv2)
  add types to capacity scheduler properties documentation
  

• MAPREDUCE-4336. Major bug reported by Siddharth Seth and fixed by Ahmed Radwan (mrv2)
  Distributed Shell fails when used with the CapacityScheduler
  

• MAPREDUCE-4320. Major bug reported by Thomas Graves and fixed by Thomas Graves (contrib/gridmix)
  gridmix mainClass wrong in pom.xml
  

• MAPREDUCE-4313. Blocker bug reported by Eli Collins and fixed by Robert Joseph Evans (build , test)
  TestTokenCache doesn't compile due TokenCache.getDelegationToken compilation error
  

• MAPREDUCE-4311. Major bug reported by Thomas Graves and fixed by Karthik Kambatla (capacity-sched , mrv2)
  Capacity scheduler.xml does not accept decimal values for capacity and maximum-capacity settings
  

• MAPREDUCE-4307. Major bug reported by Ahmed Radwan and fixed by Ahmed Radwan (mrv2)
  TeraInputFormat calls FileSystem.getDefaultBlockSize() without a Path - Failure when using ViewFileSystem
  

• MAPREDUCE-4306. Major bug reported by Ahmed Radwan and fixed by Ahmed Radwan (mrv2)
  Problem running Distributed Shell applications as a user other than the one started the daemons
  

• MAPREDUCE-4302. Critical bug reported by Daryn Sharp and fixed by Daryn Sharp (nodemanager)
  NM goes down if error encountered during log aggregation
  

• MAPREDUCE-4301. Major improvement reported by Robert Joseph Evans and fixed by Robert Joseph Evans (applicationmaster)
  Dedupe some strings in MRAM for memory savings
  

• MAPREDUCE-4300. Major bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans (applicationmaster)
  OOM in AM can turn it into a zombie.
  

• MAPREDUCE-4299. Major bug reported by Tom White and fixed by Tom White (mrv2)
  Terasort hangs with MR2 FifoScheduler
  

• MAPREDUCE-4297. Major bug reported by Ravi Prakash and fixed by Ravi Prakash (contrib/gridmix)
  Usersmap file in gridmix should not fail on empty lines
  

• MAPREDUCE-4295. Critical bug reported by Thomas Graves and fixed by Thomas Graves (mrv2 , resourcemanager)
  RM crashes due to DNS issue
  

• MAPREDUCE-4290. Major bug reported by Nishan Shetty and fixed by Devaraj K (mrv2)
  JobStatus.getState() API is giving ambiguous values
  

• MAPREDUCE-4283. Major improvement reported by Jason Lowe and fixed by Jason Lowe (jobhistoryserver , mrv2)
  Display tail of aggregated logs by default
  

• MAPREDUCE-4276. Major bug reported by Ahmed Radwan and fixed by Ahmed Radwan (mrv2)
  Allow setting yarn.nodemanager.delete.debug-delay-sec property to "-1" for easier container debugging.
  

• MAPREDUCE-4270. Major bug reported by Brock Noland and fixed by Thomas Graves (mrv2)
  data_join test classes are in the wrong packge
  

• MAPREDUCE-4269. Major bug reported by Jonathan Eagles and fixed by Jonathan Eagles (mrv2)
  documentation: Gridmix has javadoc warnings in StressJobFactory
  

• MAPREDUCE-4267. Critical bug reported by Thomas Graves and fixed by Thomas Graves (mrv2)
  mavenize pipes
  

• MAPREDUCE-4264. Blocker bug reported by Thomas Graves and fixed by Thomas Graves (mrv2)
  Got ClassCastException when using mapreduce.history.server.delegationtoken.required=true
  

• MAPREDUCE-4262. Minor bug reported by Devaraj K and fixed by Devaraj K (mrv2 , nodemanager)
  NM gives wrong log message saying "Connected to ResourceManager" before trying to connect
  

• MAPREDUCE-4252. Major bug reported by Tom White and fixed by Tom White (mrv2)
  MR2 job never completes with 1 pending task
  

• MAPREDUCE-4250. Major bug reported by Patrick Hunt and fixed by Patrick Hunt (nodemanager)
  hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster
  

• MAPREDUCE-4238. Critical bug reported by Thomas Graves and fixed by Thomas Graves (mrv2)
  mavenize data_join
  

• MAPREDUCE-4237. Major bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans
  TestNodeStatusUpdater can fail if localhost has a domain associated with it
  

• MAPREDUCE-4233. Critical bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans
  NPE can happen in RMNMNodeInfo.
  

• MAPREDUCE-4228. Major bug reported by Jason Lowe and fixed by Jason Lowe (applicationmaster , mrv2)
  mapreduce.job.reduce.slowstart.completedmaps is not working properly to delay the scheduling of the reduce tasks
  

• MAPREDUCE-4226. Major bug reported by Tom White and fixed by Tom White (mrv2)
  ConcurrentModificationException in FileSystemCounterGroup
  

• MAPREDUCE-4224. Major bug reported by Devaraj K and fixed by Devaraj K (mrv2 , scheduler , test)
  TestFifoScheduler throws org.apache.hadoop.metrics2.MetricsException
  

• MAPREDUCE-4220. Minor bug reported by Jonathan Eagles and fixed by Jonathan Eagles (mrv2)
  RM apps page starttime/endtime sorts are incorrect
  

• MAPREDUCE-4215. Major bug reported by Jonathan Eagles and fixed by Jonathan Eagles (mrv2)
  RM app page shows 500 error on appid parse error
  

• MAPREDUCE-4212. Major test reported by Daryn Sharp and fixed by Daryn Sharp (test)
  TestJobClientGetJob sometimes fails
  

• MAPREDUCE-4211. Minor bug reported by Jonathan Eagles and fixed by Jonathan Eagles (mrv2)
  Error conditions (missing appid, appid not found) are masked in the RM app page
  

• MAPREDUCE-4210. Major improvement reported by Daryn Sharp and fixed by Daryn Sharp (webapps)
  Expose listener address for WebApp
  

• MAPREDUCE-4209. Major bug reported by Radim Kolar and fixed by (build)
  junit dependency in hadoop-mapreduce-client is missing scope test
  

• MAPREDUCE-4206. Minor bug reported by Jonathan Eagles and fixed by Jonathan Eagles (mrv2)
  Sorting by Last Health-Update on the RM nodes page sorts does not work correctly
  

• MAPREDUCE-4205. Major improvement reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (mrv2)
  retrofit all JVM shutdown hooks to use ShutdownHookManager
  

• MAPREDUCE-4197. Major bug reported by Ravi Prakash and fixed by Ravi Prakash
  Include the hsqldb jar in the hadoop-mapreduce tar file
  

• MAPREDUCE-4194. Major bug reported by Jonathan Eagles and fixed by Jonathan Eagles (mrv2)
  ConcurrentModificationError in DirectoryCollection
  

• MAPREDUCE-4190. Major improvement reported by Thomas Graves and fixed by Thomas Graves (mrv2 , webapps)
  Improve web UI for task attempts userlog link
  

• MAPREDUCE-4189. Critical bug reported by Devaraj K and fixed by Devaraj K (mrv2)
  TestContainerManagerSecurity is failing
  

• MAPREDUCE-4169. Minor bug reported by Jonathan Eagles and fixed by Jonathan Eagles (mrv2)
  Container Logs appear in unsorted order
  

• MAPREDUCE-4165. Trivial bug reported by Jonathan Eagles and fixed by Jonathan Eagles (mrv2)
  Committing is misspelled as commiting in task logs
  

• MAPREDUCE-4163. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (mrv2)
  consistently set the bind address
  

• MAPREDUCE-4162. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (client , mrv2)
  Correctly set token service
  

• MAPREDUCE-4161. Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (client , mrv2)
  create sockets consistently
  

• MAPREDUCE-4160. Major bug reported by Thomas Graves and fixed by Thomas Graves (test)
  some mrv1 ant tests fail with timeout - due to 4156
  

• MAPREDUCE-4159. Major bug reported by Nishan Shetty and fixed by Devaraj K (mrv2)
  Job is running in Uber mode after setting "mapreduce.job.ubertask.maxreduces" to zero
  

• MAPREDUCE-4157. Major improvement reported by Jason Lowe and fixed by Jason Lowe (mrv2)
  ResourceManager should not kill apps that are well behaved
  

• MAPREDUCE-4156. Major bug reported by Thomas Graves and fixed by Thomas Graves (build)
  ant build fails compiling JobInProgress
  

• MAPREDUCE-4152. Major bug reported by Thomas Graves and fixed by Thomas Graves (mrv2)
  map task left hanging after AM dies trying to connect to RM
  

• MAPREDUCE-4151. Major improvement reported by Jason Lowe and fixed by Jason Lowe (mrv2 , webapps)
  RM scheduler web page should filter apps to those that are relevant to scheduling
  

• MAPREDUCE-4148. Major bug reported by Tom White and fixed by Tom White (mrv2)
  MapReduce should not have a compile-time dependency on HDFS
  

• MAPREDUCE-4146. Major improvement reported by Tom White and fixed by Ahmed Radwan
  Support limits on task status string length and number of block locations in branch-2
  

• MAPREDUCE-4144. Critical bug reported by Jason Lowe and fixed by Jason Lowe (mrv2)
  ResourceManager NPE while handling NODE_UPDATE
  

• MAPREDUCE-4140. Major bug reported by Patrick Hunt and fixed by Patrick Hunt (client , mrv2)
  mapreduce classes incorrectly importing "clover.org.apache.*" classes
  

• MAPREDUCE-4139. Major bug reported by Jason Lowe and fixed by Jason Lowe (mrv2)
  Potential ResourceManager deadlock when SchedulerEventDispatcher is stopped
  

• MAPREDUCE-4134. Major task reported by Ravi Prakash and fixed by Ravi Prakash (mrv2)
  Remove references of mapred.child.ulimit etc. since they are not being used any more
  

• MAPREDUCE-4133. Major bug reported by John George and fixed by John George
  MR over viewfs is broken
  

• MAPREDUCE-4129. Major bug reported by Ahmed Radwan and fixed by Ahmed Radwan (mrv2)
  Lots of unneeded counters log messages
  

• MAPREDUCE-4128. Major bug reported by Bikas Saha and fixed by Bikas Saha (mrv2)
  AM Recovery expects all attempts of a completed task to also be completed.
  

• MAPREDUCE-4117. Critical bug reported by Devaraj K and fixed by Devaraj K (client , mrv2)
  mapred job -status throws NullPointerException
  

• MAPREDUCE-4102. Major bug reported by Thomas Graves and fixed by Bhallamudi Venkata Siva Kamesh (webapps)
  job counters not available in Jobhistory webui for killed jobs
  

• MAPREDUCE-4099. Critical bug reported by Jason Lowe and fixed by Jason Lowe (mrv2)
  ApplicationMaster may fail to remove staging directory
  

• MAPREDUCE-4097. Major bug reported by Alejandro Abdelnur and fixed by Roman Shaposhnik (build)
  tools testcases fail because missing mrapp-generated-classpath file in classpath
  

• MAPREDUCE-4092. Blocker bug reported by Jonathan Eagles and fixed by Jonathan Eagles (mrv2)
  commitJob Exception does not fail job (regression in 0.23 vs 0.20)
  

• MAPREDUCE-4091. Critical bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (build , test)
  tools testcases failing because of MAPREDUCE-4082
  

• MAPREDUCE-4089. Blocker bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans (mrv2)
  Hung Tasks never time out.
  

• MAPREDUCE-4082. Critical bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (build)
  hadoop-mapreduce-client-app's mrapp-generated-classpath file should not be in the module JAR
  

• MAPREDUCE-4079. Blocker improvement reported by Robert Joseph Evans and fixed by Robert Joseph Evans (mr-am , mrv2)
  Allow MR AppMaster to limit ephemeral port range.
  

• MAPREDUCE-4074. Major bug reported by Devaraj K and fixed by xieguiming
  Client continuously retries to RM When RM goes down before launching Application Master
  

• MAPREDUCE-4073. Critical bug reported by Siddharth Seth and fixed by Siddharth Seth (mrv2 , scheduler)
  CS assigns multiple off-switch containers when using multi-level-queues
  

• MAPREDUCE-4072. Major bug reported by Anupam Seth and fixed by Anupam Seth (mrv2)
  User set java.library.path seems to overwrite default creating problems native lib loading
  

  -Djava.library.path in mapred.child.java.opts can cause issues with native libraries. LD_LIBRARY_PATH through mapred.child.env should be used instead.

• MAPREDUCE-4068. Blocker bug reported by Ahmed Radwan and fixed by Robert Kanter (mrv2)
  Jars in lib subdirectory of the submittable JAR are not added to the classpath
  

• MAPREDUCE-4062. Major bug reported by Thomas Graves and fixed by Thomas Graves (mrv2)
  AM Launcher thread can hang forever
  

• MAPREDUCE-4060. Major bug reported by Jason Lowe and fixed by Jason Lowe (build)
  Multiple SLF4J binding warning
  

• MAPREDUCE-4059. Major improvement reported by Robert Joseph Evans and fixed by Robert Joseph Evans (mrv2)
  The history server should have a separate pluggable storage/query interface
  

• MAPREDUCE-4053. Major bug reported by Alejandro Abdelnur and fixed by Robert Joseph Evans (mrv2)
  Counters group names deprecation is wrong, iterating over group names deprecated names don't show up
  

• MAPREDUCE-4051. Major task reported by Ravi Prakash and fixed by Ravi Prakash
  Remove the empty hadoop-mapreduce-project/assembly/all.xml file
  

• MAPREDUCE-4050. Major bug reported by Bhallamudi Venkata Siva Kamesh and fixed by Bhallamudi Venkata Siva Kamesh (mrv2)
  Invalid node link
  

• MAPREDUCE-4048. Major bug reported by Devaraj K and fixed by Devaraj K (mrv2)
  NullPointerException exception while accessing the Application Master UI
  

• MAPREDUCE-4040. Minor bug reported by Bhallamudi Venkata Siva Kamesh and fixed by Bhallamudi Venkata Siva Kamesh (jobhistoryserver , mrv2)
  History links should use hostname rather than IP address.
  

• MAPREDUCE-4031. Critical bug reported by Devaraj K and fixed by Devaraj K (mrv2 , nodemanager)
  Node Manager hangs on shut down
  

• MAPREDUCE-4024. Major bug reported by Thomas Graves and fixed by Thomas Graves (mrv2)
  RM webservices can't query on finalStatus
  

• MAPREDUCE-4020. Major bug reported by Jason Lowe and fixed by Anupam Seth (mrv2 , webapps)
  Web services returns incorrect JSON for deep queue tree
  

• MAPREDUCE-4017. Trivial improvement reported by Koji Noguchi and fixed by Thomas Graves (jobhistoryserver , jobtracker)
  Add jobname to jobsummary log
  

  The Job Summary log may contain commas in values that are escaped by a '\' character. This was true before, but is more likely to be exposed now.

• MAPREDUCE-4012. Minor bug reported by Koji Noguchi and fixed by Thomas Graves
  Hadoop Job setup error leaves no useful info to users (when LinuxTaskController is used)
  

• MAPREDUCE-4010. Critical bug reported by Jason Lowe and fixed by Alejandro Abdelnur (mrv2)
  TestWritableJobConf fails on trunk
  

• MAPREDUCE-4002. Major bug reported by Bhallamudi Venkata Siva Kamesh and fixed by Bhallamudi Venkata Siva Kamesh (examples)
  MultiFileWordCount job fails if the input path is not from default file system
  

• MAPREDUCE-3999. Major bug reported by Ravi Prakash and fixed by Ravi Prakash (mrv2 , webapps)
  Tracking link gives an error if the AppMaster hasn't started yet
  

• MAPREDUCE-3993. Major bug reported by Todd Lipcon and fixed by Karthik Kambatla (mrv1 , mrv2)
  Graceful handling of codec errors during decompression
  

• MAPREDUCE-3992. Major bug reported by Todd Lipcon and fixed by Todd Lipcon (mrv1)
  Reduce fetcher doesn't verify HTTP status code of response
  

• MAPREDUCE-3988. Major bug reported by Vinod Kumar Vavilapalli and fixed by Eric Payne (mrv2)
  mapreduce.job.local.dir doesn't point to a single directory on a node.
  

• MAPREDUCE-3983. Major test reported by Robert Joseph Evans and fixed by Ravi Prakash (mrv1)
  TestTTResourceReporting can fail, and should just be deleted
  

• MAPREDUCE-3972. Major sub-task reported by Robert Joseph Evans and fixed by Robert Joseph Evans (mrv2)
  Locking and exception issues in JobHistory Server.
  

• MAPREDUCE-3947. Minor bug reported by Todd Lipcon and fixed by Devaraj K
  yarn.app.mapreduce.am.resource.mb not documented
  

• MAPREDUCE-3942. Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli (mrv2 , security)
  Randomize master key generation for ApplicationTokenSecretManager and roll it every so often
  

• MAPREDUCE-3940. Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli (mrv2 , security)
  ContainerTokens should have an expiry interval
  

  ContainerTokens now have an expiry interval so that stale tokens cannot be used for launching containers.

• MAPREDUCE-3932. Critical bug reported by Vinod Kumar Vavilapalli and fixed by Robert Joseph Evans (mr-am , mrv2)
  MR tasks failing and crashing the AM when available-resources/headRoom becomes zero
  

• MAPREDUCE-3927. Critical bug reported by MengWang and fixed by Bhallamudi Venkata Siva Kamesh (mrv2)
  Shuffle hang when set map.failures.percent
  

• MAPREDUCE-3907. Minor improvement reported by Eugene Koontz and fixed by Eugene Koontz (documentation)
  Document entries mapred-default.xml for the jobhistory server.
  

• MAPREDUCE-3906. Trivial improvement reported by Eugene Koontz and fixed by Eugene Koontz (documentation)
  Fix inconsistency in documentation regarding mapreduce.jobhistory.principal
  

• MAPREDUCE-3893. Critical bug reported by Thomas Graves and fixed by Thomas Graves (mrv2)
  allow capacity scheduler configs maximum-applications and maximum-am-resource-percent configurable on a per queue basis
  

• MAPREDUCE-3889. Critical bug reported by Thomas Graves and fixed by Devaraj K (mrv2)
  job client tries to use /tasklog interface, but that doesn't exist anymore
  

• MAPREDUCE-3873. Minor bug reported by Nishan Shetty and fixed by xieguiming (mrv2 , nodemanager)
  Nodemanager is not getting decommisioned if the absolute ip is given in exclude file.
  

  Fixed NodeManagers' decommissioning at RM to accept IP addresses also.

• MAPREDUCE-3871. Major improvement reported by Tom White and fixed by Tom White (distributed-cache)
  Allow symlinking in LocalJobRunner DistributedCache
  

• MAPREDUCE-3870. Major bug reported by Bhallamudi Venkata Siva Kamesh and fixed by Bhallamudi Venkata Siva Kamesh (mrv2)
  Invalid App Metrics
  

• MAPREDUCE-3850. Major improvement reported by Daryn Sharp and fixed by Daryn Sharp (security)
  Avoid redundant calls for tokens in TokenCache
  

• MAPREDUCE-3842. Critical improvement reported by Alejandro Abdelnur and fixed by Thomas Graves (mrv2 , webapps)
  stop webpages from automatic refreshing
  

• MAPREDUCE-3812. Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Harsh J (mrv2 , performance)
  Lower default allocation sizes, fix allocation configurations and document them
  

  Removes two sets of previously available config properties: 1. ( yarn.scheduler.fifo.minimum-allocation-mb and yarn.scheduler.fifo.maximum-allocation-mb ) and, 2. ( yarn.scheduler.capacity.minimum-allocation-mb and yarn.scheduler.capacity.maximum-allocation-mb ) In favor of two new, generically named properties: 1. yarn.scheduler.minimum-allocation-mb - This acts as the floor value of memory resource requests for containers. 2. yarn.scheduler.maximum-allocation-mb - This acts as the ceiling value of memory resource requests for containers. Both these properties need to be set at the ResourceManager (RM) to take effect, as the RM is where the scheduler resides. Also changes the default minimum and maximums to 128 MB and 10 GB respectively.

• MAPREDUCE-3782. Critical bug reported by Arpit Gupta and fixed by Jason Lowe (mrv2)
  teragen terasort jobs fail when using webhdfs://
  

• MAPREDUCE-3773. Major new feature reported by Owen O'Malley and fixed by Owen O'Malley (jobtracker)
  Add queue metrics with buckets for job run times
  

• MAPREDUCE-3728. Critical bug reported by Roman Shaposhnik and fixed by Giridharan Kesavan (mrv2 , nodemanager)
  ShuffleHandler can't access results when configured in a secure mode
  

• MAPREDUCE-3682. Major bug reported by David Capwell and fixed by Ravi Prakash (mrv2)
  Tracker URL says AM tasks run on localhost
  

• MAPREDUCE-3672. Major bug reported by Vinod Kumar Vavilapalli and fixed by Anupam Seth (mr-am , mrv2)
  Killed maps shouldn't be counted towards JobCounter.NUM_FAILED_MAPS
  

• MAPREDUCE-3659. Major improvement reported by Daryn Sharp and fixed by Daryn Sharp (security)
  Host-based token support
  

• MAPREDUCE-3650. Blocker bug reported by Thomas Graves and fixed by Ravi Prakash (mrv2)
  testGetTokensForHftpFS() fails
  

• MAPREDUCE-3621. Major bug reported by Thomas Graves and fixed by Ravi Prakash (mrv2)
  TestDBJob and TestDataDrivenDBInputFormat ant tests fail
  

• MAPREDUCE-3613. Critical sub-task reported by Thomas Graves and fixed by Thomas Graves (mrv2)
  web service calls header contains 2 content types
  

• MAPREDUCE-3543. Critical bug reported by Mahadev konar and fixed by Thomas Graves (mrv2)
  Mavenize Gridmix.
  

  Note that to apply this you should first run the script - ./MAPREDUCE-3543v3.sh svn, then apply the patch. If this is merged to more then trunk, the version inside of hadoop-tools/hadoop-gridmix/pom.xml will need to be udpated accordingly.

• MAPREDUCE-3506. Minor bug reported by Ratandeep Ratti and fixed by Jason Lowe (client , mrv2)
  Calling getPriority on JobInfo after parsing a history log with JobHistoryParser throws a NullPointerException
  

• MAPREDUCE-3493. Minor bug reported by Ahmed Radwan and fixed by (mrv2)
  Add the default mapreduce.shuffle.port property to mapred-default.xml
  

• MAPREDUCE-3451. Major new feature reported by Patrick Wendell and fixed by Patrick Wendell (mrv2 , scheduler)
  Port Fair Scheduler to MR2
  

• MAPREDUCE-3350. Critical bug reported by Vinod Kumar Vavilapalli and fixed by Jonathan Eagles (mrv2 , webapps)
  Per-app RM page should have the list of application-attempts like on the app JHS page
  

• MAPREDUCE-3348. Major bug reported by Devaraj K and fixed by Devaraj K (mrv2)
  mapred job -status fails to give info even if the job is present in History
  

  Fixed a bug in MR client to redirect to JobHistoryServer correctly when RM forgets the app.

• MAPREDUCE-3289. Major improvement reported by Todd Lipcon and fixed by Todd Lipcon (mrv2 , nodemanager , performance)
  Make use of fadvise in the NM's shuffle handler
  

• MAPREDUCE-3082. Major bug reported by Rajit Saha and fixed by John George (harchive)
  archive command take wrong path for input file with current directory
  

• MAPREDUCE-2786. Minor improvement reported by Plamen Jeliazkov and fixed by Plamen Jeliazkov (benchmarks)
  TestDFSIO should also test compression reading/writing from command-line.
  

• MAPREDUCE-2739. Minor bug reported by Ahmed Radwan and fixed by Bo Wang (mrv2)
  MR-279: Update installation docs (remove YarnClientFactory)
  

• MAPREDUCE-2374. Major bug reported by Todd Lipcon and fixed by Andy Isaacson
  "Text File Busy" errors launching MR tasks
  

• MAPREDUCE-2289. Major bug reported by Todd Lipcon and fixed by Ahmed Radwan (job submission)
  Permissions race can make getStagingDir fail on local filesystem
  

• MAPREDUCE-2220. Minor bug reported by Rui KUBO and fixed by Rui KUBO (documentation)
  Fix new API FileOutputFormat-related typos in mapred-default.xml
  

• MAPREDUCE-987. Minor new feature reported by Philip Zeyliger and fixed by Ahmed Radwan (build , test)
  Exposing MiniDFS and MiniMR clusters as a single process command-line
  

• HDFS-3972. Critical bug reported by Todd Lipcon and fixed by Todd Lipcon (name-node)
  Trash emptier fails in secure HA cluster
  

• HDFS-3928. Major bug reported by Eli Collins and fixed by Eli Collins (test)
  MiniDFSCluster should reset the first ExitException on shutdown
  

• HDFS-3902. Minor bug reported by Andy Isaacson and fixed by Andy Isaacson
  TestDatanodeBlockScanner#testBlockCorruptionPolicy is broken
  

• HDFS-3895. Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (build)
  hadoop-client must include commons-cli
  

• HDFS-3890. Critical bug reported by Thomas Graves and fixed by Thomas Graves
  filecontext mkdirs doesn't apply umask as expected
  

• HDFS-3888. Minor bug reported by Jing Zhao and fixed by Jing Zhao
  BlockPlacementPolicyDefault code cleanup
  

• HDFS-3887. Trivial improvement reported by Jing Zhao and fixed by Jing Zhao (name-node)
  Remove redundant chooseTarget methods in BlockPlacementPolicy.java
  

• HDFS-3879. Minor bug reported by Eli Collins and fixed by Eli Collins (name-node)
  Fix findbugs warning in TransferFsImage on branch-2
  

• HDFS-3873. Major bug reported by Daryn Sharp and fixed by Daryn Sharp (hdfs client)
  Hftp assumes security is disabled if token fetch fails
  

• HDFS-3871. Minor improvement reported by Arun C Murthy and fixed by Arun C Murthy (hdfs client)
  Change NameNodeProxies to use HADOOP-8748
  

• HDFS-3866. Minor improvement reported by Ryan Hennig and fixed by Plamen Jeliazkov (build)
  HttpFS POM should have property where to download tomcat from
  

• HDFS-3864. Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (name-node)
  NN does not update internal file mtime for OP_CLOSE when reading from the edit log
  

• HDFS-3861. Blocker bug reported by Kihwal Lee and fixed by Kihwal Lee (hdfs client)
  Deadlock in DFSClient
  

• HDFS-3860. Major bug reported by Jing Zhao and fixed by Jing Zhao
  HeartbeatManager#Monitor may wrongly hold the writelock of namesystem
  

• HDFS-3856. Blocker bug reported by Thomas Graves and fixed by Eli Collins (test)
  TestHDFSServerPorts failure is causing surefire fork failure
  

• HDFS-3853. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (name-node)
  Port MiniDFSCluster enableManagedDfsDirsRedundancy option to branch-2
  

• HDFS-3852. Major bug reported by Aaron T. Myers and fixed by Daryn Sharp (hdfs client , security)
  TestHftpDelegationToken is broken after HADOOP-8225
  

• HDFS-3849. Critical bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (name-node)
  When re-loading the FSImage, we should clear the existing genStamp and leases.
  

• HDFS-3844. Trivial improvement reported by Jing Zhao and fixed by Jing Zhao
  Add @Override where necessary and remove unnecessary {@inheritdoc} and imports
  

• HDFS-3841. Major bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans
  Port HDFS-3835 to branch-0.23
  

• HDFS-3837. Major bug reported by Eli Collins and fixed by Eli Collins (data-node)
  Fix DataNode.recoverBlock findbugs warning
  

• HDFS-3835. Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (name-node , security)
  Long-lived 2NN cannot perform a checkpoint if security is enabled and the NN restarts with outstanding delegation tokens
  

• HDFS-3833. Major bug reported by Brandon Li and fixed by Brandon Li (test)
  TestDFSShell fails on Windows due to file concurrent read write
  

• HDFS-3832. Major bug reported by Suresh Srinivas and fixed by Suresh Srinivas (data-node , name-node)
  Remove protocol methods related to DistributedUpgrade
  

• HDFS-3830. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (libhdfs)
  test_libhdfs_threaded: use forceNewInstance
  

• HDFS-3819. Minor improvement reported by Jing Zhao and fixed by Jing Zhao
  Should check whether invalidate work percentage default value is not greater than 1.0f
  

• HDFS-3816. Major bug reported by Jing Zhao and fixed by Jing Zhao (name-node)
  Invalidate work percentage default value should be 0.32f instead of 32
  

• HDFS-3808. Critical bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (fuse-dfs)
  fuse_dfs: postpone libhdfs intialization until after fork
  

• HDFS-3803. Minor bug reported by Andrew Purtell and fixed by (data-node)
  BlockPoolSliceScanner new work period notice is very chatty at INFO level
  

• HDFS-3802. Trivial improvement reported by Jing Zhao and fixed by Jing Zhao
  StartupOption.name in HdfsServerConstants should be final
  

• HDFS-3796. Minor improvement reported by Todd Lipcon and fixed by Todd Lipcon (test)
  Speed up edit log tests by avoiding fsync()
  

• HDFS-3794. Major bug reported by Ravi Prakash and fixed by Ravi Prakash (webhdfs)
  WebHDFS Open used with Offset returns the original (and incorrect) Content Length in the HTTP Header.
  

• HDFS-3790. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (fuse-dfs)
  test_fuse_dfs.c doesn't compile on centos 5
  

• HDFS-3788. Critical bug reported by Eli Collins and fixed by Tsz Wo (Nicholas), SZE (webhdfs)
  distcp can't copy large files using webhdfs due to missing Content-Length header
  

• HDFS-3765. Major improvement reported by Vinay and fixed by Vinay (ha)
  Namenode INITIALIZESHAREDEDITS should be able to initialize all shared storages
  

• HDFS-3760. Minor bug reported by Andy Isaacson and fixed by Andy Isaacson (hdfs client)
  primitiveCreate is a write, not a read
  

• HDFS-3758. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (fuse-dfs)
  TestFuseDFS test failing
  

• HDFS-3756. Critical bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  DelegationTokenFetcher creates 2 HTTP connections, the second one not properly configured
  

• HDFS-3755. Major bug reported by Todd Lipcon and fixed by Todd Lipcon (name-node)
  Creating an already-open-for-write file with overwrite=true fails
  

• HDFS-3754. Major bug reported by Eli Collins and fixed by Eli Collins (data-node)
  BlockSender doesn't shutdown ReadaheadPool threads
  

• HDFS-3738. Minor bug reported by Aaron T. Myers and fixed by Aaron T. Myers (test)
  TestDFSClientRetries#testFailuresArePerOperation sets incorrect timeout config
  

• HDFS-3733. Major bug reported by Andy Isaacson and fixed by Andy Isaacson (webhdfs)
  Audit logs should include WebHDFS access
  

• HDFS-3732. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (fuse-dfs)
  fuse_dfs: incorrect configuration value checked for connection expiry timer period
  

• HDFS-3731. Blocker bug reported by Suresh Srinivas and fixed by Kihwal Lee (data-node)
  2.0 release upgrade must handle blocks being written from 1.0
  

• HDFS-3724. Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur
  add InterfaceAudience annotations to HttpFS classes and making inner enum static
  

• HDFS-3723. Major improvement reported by E. Sammer and fixed by Jing Zhao (scripts , tools)
  All commands should support meaningful --help
  

• HDFS-3721. Critical bug reported by Todd Lipcon and fixed by Aaron T. Myers (data-node , hdfs client)
  hsync support broke wire compatibility
  

• HDFS-3720. Major bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (libhdfs)
  hdfs.h must get packaged
  

• HDFS-3718. Critical bug reported by Kihwal Lee and fixed by Kihwal Lee (data-node)
  Datanode won't shutdown because of runaway DataBlockScanner thread
  

• HDFS-3715. Major bug reported by Eli Collins and fixed by Andrew Wang (test)
  Fix TestFileCreation#testFileCreationNamenodeRestart
  

• HDFS-3711. Major improvement reported by Andrew Wang and fixed by Andrew Wang
  Manually convert remaining tests to JUnit4
  

• HDFS-3710. Minor bug reported by Andy Isaacson and fixed by Andy Isaacson (libhdfs)
  libhdfs misuses O_RDONLY/WRONLY/RDWR
  

• HDFS-3709. Major test reported by Eli Collins and fixed by Eli Collins (test)
  TestStartup tests still binding to the ephemeral port
  

• HDFS-3707. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  TestFSInputChecker: improper use of skip
  

• HDFS-3697. Minor improvement reported by Todd Lipcon and fixed by Todd Lipcon (data-node , performance)
  Enable fadvise readahead by default
  

  The datanode now performs 4MB readahead by default when reading data from its disks, if the native libraries are present. This has been shown to improve performance in many workloads. The feature may be disabled by setting dfs.datanode.readahead.bytes to "0".

• HDFS-3696. Critical bug reported by Kihwal Lee and fixed by Tsz Wo (Nicholas), SZE
  Create files with WebHdfsFileSystem goes OOM when file size is big
  

• HDFS-3690. Major bug reported by Eli Collins and fixed by Eli Collins
  BlockPlacementPolicyDefault incorrectly casts LOG
  

• HDFS-3688. Major bug reported by Jason Lowe and fixed by Jason Lowe (data-node)
  Namenode loses datanode hostname if datanode re-registers
  

• HDFS-3683. Minor bug reported by Todd Lipcon and fixed by Plamen Jeliazkov (name-node)
  Edit log replay progress indicator shows >100% complete
  

• HDFS-3679. Minor bug reported by Conrad Meyer and fixed by Conrad Meyer (fuse-dfs)
  fuse_dfs notrash option sets usetrash
  

• HDFS-3675. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (libhdfs)
  libhdfs: follow documented return codes
  

• HDFS-3673. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  libhdfs: fix some compiler warnings
  

• HDFS-3672. Major improvement reported by Andrew Wang and fixed by Andrew Wang
  Expose disk-location information for blocks to enable better scheduling
  

• HDFS-3666. Minor improvement reported by Eli Collins and fixed by Eli Collins
  Plumb more exception messages to terminate
  

• HDFS-3665. Major test reported by Eli Collins and fixed by Eli Collins (test)
  Add a test for renaming across file systems via a symlink
  

• HDFS-3664. Major bug reported by Eli Collins and fixed by Colin Patrick McCabe (test)
  BlockManager race when stopping active services
  

• HDFS-3663. Major improvement reported by Eli Collins and fixed by Eli Collins (test)
  MiniDFSCluster should capture the code path that led to the first ExitException
  

• HDFS-3658. Major bug reported by Eli Collins and fixed by Tsz Wo (Nicholas), SZE
  TestDFSClientRetries#testNamenodeRestart failed
  

• HDFS-3650. Major improvement reported by Andrew Wang and fixed by Andrew Wang
  Use MutableQuantiles to provide latency histograms for various operations
  

• HDFS-3646. Critical bug reported by Kihwal Lee and fixed by Kihwal Lee (hdfs client)
  LeaseRenewer can hold reference to inactive DFSClient instances forever
  

• HDFS-3641. Minor improvement reported by Eli Collins and fixed by Eli Collins
  Move server Util time methods to common and use now instead of System#currentTimeMillis
  

• HDFS-3637. Major new feature reported by Aaron T. Myers and fixed by Aaron T. Myers (data-node , hdfs client , security)
  Add support for encrypting the DataTransferProtocol
  

• HDFS-3634. Minor test reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (fuse-dfs)
  Add self-contained, mavenized fuse_dfs test
  

• HDFS-3633. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (libhdfs)
  libhdfs: hdfsDelete should pass JNI_FALSE or JNI_TRUE
  

• HDFS-3629. Trivial bug reported by Brandon Li and fixed by Brandon Li (name-node)
  fix the typo in the error message about inconsistent storage layout version
  

• HDFS-3622. Major bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans
  Backport HDFS-3541 to branch-0.23
  

• HDFS-3615. Major bug reported by Eli Collins and fixed by Aaron T. Myers (security)
  Two BlockTokenSecretManager findbugs warnings
  

• HDFS-3613. Trivial improvement reported by Harsh J and fixed by Andrew Wang (name-node)
  GSet prints some INFO level values, which aren't really very useful to all
  

• HDFS-3612. Trivial improvement reported by Harsh J and fixed by Andy Isaacson (name-node)
  Single namenode image directory config warning can be improved
  

• HDFS-3611. Trivial bug reported by Harsh J and fixed by Colin Patrick McCabe (name-node)
  NameNode prints unnecessary WARNs about edit log normally skipping a few bytes
  

• HDFS-3610. Minor improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  fuse_dfs: Provide a way to use the default (configured) NN URI
  

• HDFS-3609. Major bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (libhdfs)
  libhdfs: don't force the URI to look like hdfs://hostname:port
  

• HDFS-3608. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  fuse_dfs: detect changes in UID ticket cache
  

• HDFS-3606. Minor test reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (libhdfs)
  libhdfs: create self-contained unit test
  

• HDFS-3605. Major bug reported by Brahma Reddy Battula and fixed by Todd Lipcon (ha , name-node)
  Block mistakenly marked corrupt during edit log catchup phase of failover
  

• HDFS-3604. Minor improvement reported by Eli Collins and fixed by Eli Collins
  Add dfs.webhdfs.enabled to hdfs-default.xml
  

• HDFS-3603. Major bug reported by Jason Lowe and fixed by Jason Lowe (test)
  Decouple TestHDFSTrash from TestTrash
  

• HDFS-3597. Minor bug reported by Andy Isaacson and fixed by Andy Isaacson
  SNN can fail to start on upgrade
  

• HDFS-3591. Major bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans
  Backport HDFS-3357 to branch-0.23
  

• HDFS-3583. Major improvement reported by Eli Collins and fixed by Andrew Wang (test)
  Convert remaining tests to Junit4
  

• HDFS-3582. Minor improvement reported by Eli Collins and fixed by Eli Collins (test)
  Hook daemon process exit for testing
  

• HDFS-3581. Major bug reported by Eli Collins and fixed by Eli Collins (name-node)
  FSPermissionChecker#checkPermission sticky bit check missing range check
  

• HDFS-3580. Minor bug reported by Andy Isaacson and fixed by Andy Isaacson
  incompatible types; no instance(s) of type variable(s) V exist so that V conforms to boolean compiling HttpFSServer.java with OpenJDK
  

• HDFS-3579. Major bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (libhdfs)
  libhdfs: fix exception handling
  

• HDFS-3577. Blocker bug reported by Alejandro Abdelnur and fixed by Tsz Wo (Nicholas), SZE (hdfs client)
  WebHdfsFileSystem can not read files larger than 24KB
  

• HDFS-3575. Minor bug reported by Brock Noland and fixed by Brock Noland
  HttpFS does not log Exception Stacktraces
  

• HDFS-3574. Minor bug reported by Todd Lipcon and fixed by Todd Lipcon (name-node)
  Fix small race and do some cleanup in GetImageServlet
  

• HDFS-3572. Minor bug reported by Todd Lipcon and fixed by Todd Lipcon (name-node , security)
  Cleanup code which inits SPNEGO in HttpServer
  

• HDFS-3568. Major improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  fuse_dfs: add support for security
  

• HDFS-3559. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  DFSTestUtil: use Builder class to construct DFSTestUtil instances
  

• HDFS-3555. Major bug reported by Jeff Lord and fixed by Andy Isaacson (data-node , hdfs client)
  idle client socket triggers DN ERROR log (should be INFO or DEBUG)
  

• HDFS-3551. Major bug reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (webhdfs)
  WebHDFS CREATE does not use client location for redirection
  

• HDFS-3548. Critical bug reported by Todd Lipcon and fixed by Colin Patrick McCabe (name-node)
  NamenodeFsck.copyBlock fails to create a Block Reader
  

• HDFS-3541. Major bug reported by suja s and fixed by Vinay (data-node)
  Deadlock between recovery, xceiver and packet responder
  

• HDFS-3539. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  libhdfs code cleanups
  

• HDFS-3537. Minor improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (fuse-dfs , libhdfs)
  Move libhdfs and fuse-dfs source to native subdirectories
  

• HDFS-3535. Major new feature reported by Andy Isaacson and fixed by Andy Isaacson (name-node)
  Audit logging should log denied accesses
  

• HDFS-3531. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (name-node)
  EditLogFileOutputStream#preallocate should check for incomplete writes
  

• HDFS-3524. Major bug reported by Eli Collins and fixed by Brandon Li (test)
  TestFileLengthOnClusterRestart failed due to error message change
  

• HDFS-3522. Major bug reported by Brandon Li and fixed by Brandon Li (name-node)
  If NN is in safemode, it should throw SafeModeException when getBlockLocations has zero locations
  

  getBlockLocations(), and hence open() for read, will now throw SafeModeException if the NameNode is still in safe mode and there are no replicas reported yet for one of the blocks in the file.

• HDFS-3520. Major improvement reported by Eli Collins and fixed by Eli Collins (name-node)
  Add transfer rate logging to TransferFsImage
  

• HDFS-3518. Major bug reported by Bikas Saha and fixed by Tsz Wo (Nicholas), SZE (hdfs client)
  Provide API to check HDFS operational state
  

  Add a utility method HdfsUtils.isHealthy(uri) for checking if the given HDFS is healthy.

• HDFS-3517. Minor bug reported by Eli Collins and fixed by Eli Collins (test)
  TestStartup should bind ephemeral ports
  

• HDFS-3516. Major improvement reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (hdfs client)
  Check content-type in WebHdfsFileSystem
  

• HDFS-3514. Major improvement reported by Henry Robinson and fixed by Henry Robinson (test)
  Add missing TestParallelLocalRead
  

• HDFS-3513. Major improvement reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur
  HttpFS should cache filesystems
  

• HDFS-3505. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  DirectoryScanner does not join all threads in shutdown
  

• HDFS-3504. Major improvement reported by Siddharth Seth and fixed by Tsz Wo (Nicholas), SZE
  Configurable retry in DFSClient
  

• HDFS-3502. Major sub-task reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (name-node)
  Change INodeFile and INodeFileUnderConstruction to package private
  

• HDFS-3501. Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (ha , name-node)
  Checkpointing with security enabled will stop working after ticket lifetime expires
  

• HDFS-3491. Major bug reported by Romain Rigaux and fixed by Alejandro Abdelnur
  HttpFs does not set permissions correctly
  

• HDFS-3490. Minor bug reported by Todd Lipcon and fixed by Tsz Wo (Nicholas), SZE (webhdfs)
  DN WebHDFS methods throw NPE if Namenode RPC address param not specified
  

• HDFS-3487. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (tools)
  offlineimageviewer should give byte offset information when it encounters an exception
  

• HDFS-3486. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (security , tools)
  offlineimageviewer can't read fsimage files that contain persistent delegation tokens
  

• HDFS-3485. Minor bug reported by Andy Isaacson and fixed by Andy Isaacson
  DataTransferThrottler will over-throttle when currentTimeMillis jumps
  

• HDFS-3484. Minor bug reported by Aaron T. Myers and fixed by Aaron T. Myers (hdfs client)
  hdfs fsck doesn't work if NN HTTP address is set to 0.0.0.0 even if NN RPC address is configured
  

• HDFS-3481. Major improvement reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur
  Refactor HttpFS handling of JAX-RS query string parameters
  

• HDFS-3480. Major bug reported by Eli Collins and fixed by Vinay (build)
  Multiple SLF4J binding warning
  

• HDFS-3475. Trivial improvement reported by Harsh J and fixed by Harsh J
  Make the replication and invalidation rates configurable
  

  This change adds two new configuration parameters. # {{dfs.namenode.invalidate.work.pct.per.iteration}} for controlling deletion rate of blocks. # {{dfs.namenode.replication.work.multiplier.per.iteration}} for controlling replication rate. This in turn allows controlling the time it takes for decommissioning. Please see hdfs-default.xml for detailed description.

• HDFS-3474. Major sub-task reported by Ivan Kelly and fixed by Ivan Kelly
  Cleanup Exception handling in BookKeeper journal manager
  

• HDFS-3469. Minor bug reported by Vinay and fixed by Vinay (auto-failover)
  start-dfs.sh will start zkfc, but stop-dfs.sh will not stop zkfc similarly.
  

• HDFS-3468. Major sub-task reported by Uma Maheswara Rao G and fixed by Uma Maheswara Rao G
  Make BKJM-ZK session timeout configurable.
  

• HDFS-3466. Major bug reported by Owen O'Malley and fixed by Owen O'Malley (name-node , security)
  The SPNEGO filter for the NameNode should come out of the web keytab file
  

• HDFS-3460. Critical bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur
  HttpFS proxyuser validation with Kerberos ON uses full principal name
  

• HDFS-3454. Minor improvement reported by Eli Collins and fixed by Eli Collins (balancer)
  Balancer unconditionally logs InterruptedException at INFO level on shutdown if security is enabled
  

• HDFS-3452. Blocker sub-task reported by suja s and fixed by Uma Maheswara Rao G
  BKJM:Switch from standby to active fails and NN gets shut down due to delay in clearing of lock
  

• HDFS-3446. Major bug reported by Matthew Jacobs and fixed by Matthew Jacobs (name-node)
  HostsFileReader silently ignores bad includes/excludes
  

  HDFS no longer silently ignores missing or unreadable host files specified by dfs.hosts or dfs.hosts.exclude. In order to specify that no hosts should be included or excluded, administrators should either refrain from setting the relevant config properties, or create an empty file in order to represent an empty list.

• HDFS-3444. Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (hdfs client)
  hdfs groups command doesn't work with security enabled
  

• HDFS-3442. Minor bug reported by suja s and fixed by Andrew Wang
  Incorrect count for Missing Replicas in FSCK report
  

• HDFS-3441. Major sub-task reported by suja s and fixed by Rakesh R
  Race condition between rolling logs at active NN and purging at standby
  

• HDFS-3440. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  should more effectively limit stream memory consumption when reading corrupt edit logs
  

• HDFS-3438. Major improvement reported by Todd Lipcon and fixed by Todd Lipcon (ha)
  BootstrapStandby should not require a rollEdits on active node
  

• HDFS-3436. Major bug reported by Brahma Reddy Battula and fixed by Vinay (data-node)
  adding new datanode to existing pipeline fails in case of Append/Recovery
  

• HDFS-3433. Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (name-node)
  GetImageServlet should allow administrative requestors when security is enabled
  

• HDFS-3428. Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  move DelegationTokenRenewer to common
  

• HDFS-3423. Major sub-task reported by Rakesh R and fixed by Ivan Kelly
  BKJM: NN startup is failing, when tries to recoverUnfinalizedSegments() a bad inProgress_ ZNodes
  

• HDFS-3422. Minor bug reported by Todd Lipcon and fixed by Todd Lipcon (test)
  TestStandbyIsHot timeouts too aggressive
  

• HDFS-3419. Minor improvement reported by Eli Collins and fixed by Eli Collins
  Cleanup LocatedBlock
  

• HDFS-3417. Minor improvement reported by Eli Collins and fixed by Eli Collins (data-node)
  Rename BalancerDatanode#getName to getDisplayName to be consistent with Datanode
  

• HDFS-3416. Minor improvement reported by Eli Collins and fixed by Eli Collins (data-node)
  Cleanup DatanodeID and DatanodeRegistration constructors used by testing
  

• HDFS-3415. Major bug reported by Brahma Reddy Battula and fixed by Brandon Li (name-node)
  During NameNode starting up, it may pick wrong storage directory inspector when the layout versions of the storage directories are different
  

• HDFS-3414. Minor bug reported by Aaron T. Myers and fixed by Aaron T. Myers (balancer)
  Balancer does not find NameNode if rpc-address or servicerpc-address are not set in client configs
  

• HDFS-3413. Critical bug reported by Todd Lipcon and fixed by Aaron T. Myers (ha , test)
  TestFailureToReadEdits timing out
  

• HDFS-3408. Minor sub-task reported by Rakesh R and fixed by Rakesh R (name-node)
  BKJM : Namenode format fails, if there is no BK root
  

• HDFS-3404. Major improvement reported by Aaron T. Myers and fixed by Aaron T. Myers
  Make putImage in GetImageServlet infer remote address to fetch from request
  

• HDFS-3401. Major improvement reported by Eli Collins and fixed by Eli Collins (data-node , test)
  Cleanup DatanodeDescriptor creation in the tests
  

• HDFS-3400. Major improvement reported by Aaron T. Myers and fixed by Aaron T. Myers (data-node , scripts)
  DNs should be able start with jsvc even if security is disabled
  

• HDFS-3398. Minor bug reported by Brahma Reddy Battula and fixed by amith (hdfs client)
  Client will not retry when primaryDN is down once it's just got pipeline
  

• HDFS-3394. Minor improvement reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (name-node)
  Do not use generic in INodeFile.getLastBlock()
  

• HDFS-3391. Critical bug reported by Arun C Murthy and fixed by Todd Lipcon
  TestPipelinesFailover#testLeaseRecoveryAfterFailover is failing
  

• HDFS-3390. Minor improvement reported by Aaron T. Myers and fixed by Aaron T. Myers (hdfs client)
  DFSAdmin should print full stack traces of errors when DEBUG logging is enabled
  

• HDFS-3389. Major sub-task reported by Uma Maheswara Rao G and fixed by Uma Maheswara Rao G (name-node)
  Document the BKJM usage in Namenode HA.
  

• HDFS-3385. Major bug reported by Brahma Reddy Battula and fixed by Tsz Wo (Nicholas), SZE (name-node)
  ClassCastException when trying to append a file
  

• HDFS-3372. Minor improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (tools)
  offlineEditsViewer should be able to read a binary edits file with recovery mode
  

• HDFS-3369. Minor sub-task reported by John George and fixed by John George (name-node)
  change variable names referring to inode in blockmanagement to more appropriate
  

• HDFS-3368. Major bug reported by Konstantin Shvachko and fixed by Konstantin Shvachko (name-node)
  Missing blocks due to bad DataNodes coming up and down.
  

• HDFS-3359. Critical bug reported by Todd Lipcon and fixed by Todd Lipcon (hdfs client)
  DFSClient.close should close cached sockets
  

• HDFS-3341. Minor improvement reported by Todd Lipcon and fixed by Todd Lipcon
  Change minimum RPC versions to 2.0.0-SNAPSHOT instead of 2.0.0
  

• HDFS-3335. Major improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  check for edit log corruption at the end of the log
  

• HDFS-3334. Major bug reported by Daryn Sharp and fixed by Daryn Sharp (hdfs client)
  ByteRangeInputStream leaks streams
  

• HDFS-3331. Major bug reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (name-node)
  setBalancerBandwidth do not checkSuperuserPrivilege
  

• HDFS-3321. Major bug reported by Ravi Prakash and fixed by Ravi Prakash
  Error message for insufficient data nodes to come out of safemode is wrong.
  

• HDFS-3318. Blocker bug reported by Daryn Sharp and fixed by Daryn Sharp (hdfs client)
  Hftp hangs on transfers >2GB
  

• HDFS-3312. Blocker bug reported by Daryn Sharp and fixed by Daryn Sharp (hdfs client)
  Hftp selects wrong token service
  

• HDFS-3308. Critical bug reported by Daryn Sharp and fixed by Daryn Sharp (webhdfs)
  hftp/webhdfs can't get tokens if authority has no port
  

• HDFS-3306. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  fuse_dfs: don't lock release operations
  

• HDFS-3291. Major test reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur
  add test that covers HttpFS working w/ a non-HDFS Hadoop filesystem
  

• HDFS-3276. Minor improvement reported by Vinithra Varadharajan and fixed by Todd Lipcon (ha , name-node)
  initializeSharedEdits should have a -nonInteractive flag
  

• HDFS-3266. Minor bug reported by Aaron T. Myers and fixed by madhukara phatak
  DFSTestUtil#waitCorruptReplicas doesn't sleep between checks
  

• HDFS-3258. Major test reported by Eli Collins and fixed by Junping Du (test)
  Test for HADOOP-8144 (pseudoSortByDistance in NetworkTopology for first rack local node)
  

• HDFS-3243. Major bug reported by Todd Lipcon and fixed by Henry Robinson (hdfs client , test)
  TestParallelRead timing out on jenkins
  

• HDFS-3235. Minor bug reported by Henry Robinson and fixed by Henry Robinson
  MiniDFSClusterManager doesn't correctly support -format option
  

• HDFS-3230. Minor improvement reported by Eli Collins and fixed by Eli Collins (test)
  Cleanup DatanodeID creation in the tests
  

• HDFS-3194. Major bug reported by suja s and fixed by Andy Isaacson (data-node)
  DataNode block scanner is running too frequently
  

• HDFS-3190. Minor sub-task reported by Todd Lipcon and fixed by Todd Lipcon (name-node)
  Simple refactors in existing NN code to assist QuorumJournalManager extension
  

• HDFS-3177. Major bug reported by Kihwal Lee and fixed by Kihwal Lee (data-node , hdfs client)
  Allow DFSClient to find out and use the CRC type being used for a file.
  

• HDFS-3176. Major bug reported by Kihwal Lee and fixed by Kihwal Lee (hdfs client)
  JsonUtil should not parse the MD5MD5CRC32FileChecksum bytes on its own.
  

• HDFS-3170. Major improvement reported by Todd Lipcon and fixed by Matthew Jacobs (data-node)
  Add more useful metrics for write latency
  

• HDFS-3168. Major sub-task reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (name-node)
  Clean up FSNamesystem and BlockManager
  

• HDFS-3166. Critical bug reported by Daryn Sharp and fixed by Daryn Sharp (hdfs client)
  Hftp connections do not have a timeout
  

• HDFS-3157. Major bug reported by J.Andreina and fixed by Ashish Singhi (name-node)
  Error in deleting block is keep on coming from DN even after the block report and directory scanning has happened
  

• HDFS-3150. Major new feature reported by Eli Collins and fixed by Eli Collins (data-node , hdfs client)
  Add option for clients to contact DNs via hostname
  

• HDFS-3136. Major bug reported by Jason Lowe and fixed by Jason Lowe (build)
  Multiple SLF4J binding warning
  

• HDFS-3134. Major improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (name-node)
  Harden edit log loader against malformed or malicious input
  

• HDFS-3113. Major new feature reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur
  httpfs does not support delegation tokens
  

• HDFS-3110. Major improvement reported by Henry Robinson and fixed by Henry Robinson (libhdfs , performance)
  libhdfs implementation of direct read API
  

  libhdfs is enhanced to read directly into user-supplied buffers when possible, reducing the number of memory copies.

• HDFS-3067. Major bug reported by Henry Robinson and fixed by Henry Robinson (hdfs client)
  NPE in DFSInputStream.readBuffer if read is repeated on corrupted block
  

• HDFS-3058. Major sub-task reported by Ivan Kelly and fixed by Ivan Kelly
  HA: Bring BookKeeperJournalManager up to date with HA changes
  

• HDFS-3054. Major bug reported by patrick white and fixed by Colin Patrick McCabe (tools)
  distcp -skipcrccheck has no effect
  

• HDFS-3048. Major bug reported by Eli Collins and fixed by Andy Isaacson (name-node)
  Small race in BlockManager#close
  

• HDFS-3042. Major new feature reported by Todd Lipcon and fixed by Todd Lipcon (auto-failover , ha)
  Automatic failover support for NN HA
  

• HDFS-3040. Trivial improvement reported by Aaron T. Myers and fixed by madhukara phatak (test)
  TestMulitipleNNDataBlockScanner is misspelled
  

• HDFS-3037. Minor bug reported by Aaron T. Myers and fixed by Aaron T. Myers (test)
  TestMulitipleNNDataBlockScanner#testBlockScannerAfterRestart is racy
  

• HDFS-3031. Major bug reported by Stephen Chu and fixed by Todd Lipcon (ha)
  HA: Error (failed to close file) when uploading large file + kill active NN + manual failover
  

• HDFS-3002. Trivial improvement reported by Suresh Srinivas and fixed by Suresh Srinivas (test)
  TestNameNodeMetrics need not wait for metrics update with new metrics framework
  

• HDFS-2988. Minor improvement reported by Todd Lipcon and fixed by Miomir Boljanovic (name-node)
  Improve error message when storage directory lock fails
  

• HDFS-2982. Critical bug reported by Todd Lipcon and fixed by Colin Patrick McCabe (name-node)
  Startup performance suffers when there are many edit log segments
  

• HDFS-2978. Major new feature reported by Aaron T. Myers and fixed by Aaron T. Myers (name-node)
  The NameNode should expose name dir statuses via JMX
  

• HDFS-2966. Minor bug reported by Steve Loughran and fixed by Steve Loughran (test)
  TestNameNodeMetrics tests can fail under load
  

• HDFS-2963. Minor bug reported by J.Andreina and fixed by Andrew Wang
  Console Output is confusing while executing metasave (dfsadmin command)
  

• HDFS-2914. Major bug reported by Hari Mankude and fixed by Vinay (ha , name-node)
  HA: Standby should not enter safemode when resources are low
  

• HDFS-2885. Major improvement reported by Eli Collins and fixed by Tsz Wo (Nicholas), SZE (name-node)
  Remove "federation" from the nameservice config options
  

• HDFS-2834. Major improvement reported by Henry Robinson and fixed by Henry Robinson (hdfs client , performance)
  ByteBuffer-based read API for DFSInputStream
  

• HDFS-2800. Major bug reported by Aaron T. Myers and fixed by Todd Lipcon (ha , test)
  HA: TestStandbyCheckpoints.testCheckpointCancellation is racy
  

• HDFS-2797. Major bug reported by Aaron T. Myers and fixed by Colin Patrick McCabe (ha , name-node)
  Fix misuses of InputStream#skip in the edit log code
  

• HDFS-2793. Major new feature reported by Aaron T. Myers and fixed by Todd Lipcon (name-node)
  Add an admin command to trigger an edit log roll
  

  Introduced a new command, "hdfs dfsadmin -rollEdits" which requests that the active NameNode roll its edit log. This can be useful for administrators manually backing up log segments.

• HDFS-2759. Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (ha , name-node)
  Pre-allocate HDFS edit log files after writing version number
  

• HDFS-2757. Major bug reported by Jean-Daniel Cryans and fixed by Jean-Daniel Cryans
  Cannot read a local block that's being written to when using the local read short circuit
  

• HDFS-2727. Minor improvement reported by Sho Shimauchi and fixed by Colin Patrick McCabe (libhdfs)
  libhdfs should get the default block size from the server
  

  libhdfs now uses the server block size configuration rather than the deprecated dfs.block.size client configuration.

• HDFS-2717. Major sub-task reported by Ivan Kelly and fixed by Ivan Kelly
  BookKeeper Journal output stream doesn't check addComplete rc
  

• HDFS-2686. Major improvement reported by Todd Lipcon and fixed by Suresh Srinivas (data-node , name-node)
  Remove DistributedUpgrade related code
  

  This jira removes functionality that has not been used/applicable since release 0.17. The incompatibility introduced by this change will not affect any HDFS users.

• HDFS-2652. Major improvement reported by Daryn Sharp and fixed by Daryn Sharp
  Port token service changes from 205
  

• HDFS-2619. Major bug reported by Owen O'Malley and fixed by Owen O'Malley (build)
  Remove my personal email address from the libhdfs build file.
  

• HDFS-2617. Major improvement reported by Jakob Homan and fixed by Jakob Homan (security)
  Replaced Kerberized SSL for image transfer and fsck with SPNEGO-based solution
  

  Due to the requirement that KSSL use weak encryption types for Kerberos tickets, HTTP authentication to the NameNode will now use SPNEGO by default. This will require users of previous branch-1 releases with security enabled to modify their configurations and create new Kerberos principals in order to use SPNEGO. The old behavior of using KSSL can optionally be enabled by setting the configuration option "hadoop.security.use-weak-http-crypto" to "true".

• HDFS-2421. Major improvement reported by Hairong Kuang and fixed by Jing Zhao (name-node)
  Improve the concurrency of SerialNumberMap in NameNode
  

• HDFS-2391. Major improvement reported by Rajit Saha and fixed by Harsh J (balancer)
  Newly set BalancerBandwidth value is not displayed anywhere
  

• HDFS-2330. Major sub-task reported by Uma Maheswara Rao G and fixed by Uma Maheswara Rao G (name-node)
  In NNStorage.java, IOExceptions of stream closures can mask root exceptions.
  

• HDFS-2285. Major bug reported by Konstantin Shvachko and fixed by Konstantin Shvachko (name-node)
  BackupNode should reject requests trying to modify namespace
  

• HDFS-2025. Minor bug reported by sravankorumilli and fixed by Ashish Singhi (data-node)
  Go Back to File View link is not working in tail.jsp
  

• HDFS-1490. Minor bug reported by Dmytro Molkov and fixed by Vinay (name-node)
  TransferFSImage should timeout
  

• HDFS-1249. Minor bug reported by matsusaka kentaro and fixed by Colin Patrick McCabe (fuse-dfs)
  with fuse-dfs, chown which only has owner (or only group) argument fails with Input/output error.
  

• HDFS-1153. Minor bug reported by Ravi Phulari and fixed by Ravi Phulari (data-node)
  dfsnodelist.jsp should handle invalid input parameters
  

• HDFS-1013. Minor improvement reported by Todd Lipcon and fixed by Eugene Koontz
  Miscellaneous improvements to HTML markup for web UIs
  

• HDFS-799. Major improvement reported by Christian Kunz and fixed by Colin Patrick McCabe
  libhdfs must call DetachCurrentThread when a thread is destroyed
  

• HDFS-766. Minor bug reported by Ravi Phulari and fixed by Jon Zuanich
  Error message not clear for set space quota out of boundary values.
  

• HDFS-744. Major new feature reported by Hairong Kuang and fixed by Lars Hofhansl (data-node , hdfs client)
  Support hsync in HDFS
  

• HDFS-711. Major bug reported by freestyler and fixed by Colin Patrick McCabe (documentation)
  hdfsUtime does not handle atime = 0 or mtime = 0 correctly
  

• HDFS-470. Minor bug reported by Pete Wyckoff and fixed by Colin Patrick McCabe
  libhdfs should handle 0-length reads from FSInputStream correctly
  

• HADOOP-8801. Major bug reported by Eli Collins and fixed by Eli Collins
  ExitUtil#terminate should capture the exception stack trace
  

• HADOOP-8794. Major bug reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli
  Modifiy bin/hadoop to point to HADOOP_YARN_HOME
  

• HADOOP-8781. Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (scripts)
  hadoop-config.sh should add JAVA_LIBRARY_PATH to LD_LIBRARY_PATH
  

• HADOOP-8775. Major bug reported by Sandy Ryza and fixed by Sandy Ryza
  MR2 distcp permits non-positive value to -bandwidth option which causes job never to complete
  

• HADOOP-8770. Blocker bug reported by Todd Lipcon and fixed by Eli Collins (trash)
  NN should not RPC to self to find trash defaults (causes deadlock)
  

• HADOOP-8766. Major bug reported by Eli Collins and fixed by Colin Patrick McCabe (test)
  FileContextMainOperationsBaseTest should randomize the root dir
  

• HADOOP-8764. Major bug reported by Trevor Robinson and fixed by Trevor Robinson (build)
  CMake: HADOOP-8737 broke ARM build
  

• HADOOP-8754. Minor improvement reported by Brandon Li and fixed by Brandon Li (ipc)
  Deprecate all the RPC.getServer() variants
  

• HADOOP-8749. Major bug reported by Ahmed Radwan and fixed by Ahmed Radwan (conf)
  HADOOP-8031 changed the way in which relative xincludes are handled in Configuration.
  

• HADOOP-8748. Minor improvement reported by Arun C Murthy and fixed by Arun C Murthy (io)
  Move dfsclient retry to a util class
  

• HADOOP-8747. Major bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (native)
  Syntax error on cmake version 2.6 patch 2 in JNIFlags.cmake
  

• HADOOP-8737. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (native)
  cmake: always use JAVA_HOME to find libjvm.so, jni.h, jni_md.h
  

• HADOOP-8727. Major bug reported by Harsh J and fixed by Harsh J (conf)
  Gracefully deprecate dfs.umaskmode in 2.x onwards
  

• HADOOP-8726. Major bug reported by Benoy Antony and fixed by Daryn Sharp (security)
  The Secrets in Credentials are not available to MR tasks
  

• HADOOP-8725. Blocker bug reported by Daryn Sharp and fixed by Daryn Sharp (security)
  MR is broken when security is off
  

• HADOOP-8722. Minor bug reported by Eli Collins and fixed by Colin Patrick McCabe (documentation)
  Update BUILDING.txt with latest snappy info
  

• HADOOP-8721. Critical bug reported by suja s and fixed by Vinay (auto-failover , ha)
  ZKFC should not retry 45 times when attempting a graceful fence during a failover
  

• HADOOP-8720. Trivial bug reported by Vlad Rozov and fixed by Vlad Rozov (test)
  TestLocalFileSystem should use test root subdirectory
  

• HADOOP-8710. Major improvement reported by Eli Collins and fixed by Eli Collins (fs)
  Remove ability for users to easily run the trash emptier
  

  The trash emptier may no longer be run using "hadoop org.apache.hadoop.fs.Trash". The trash emptier runs on the NameNode (if configured). Old trash checkpoints may be deleted using "hadoop fs -expunge".

• HADOOP-8709. Critical bug reported by Jason Lowe and fixed by Jason Lowe (fs)
  globStatus changed behavior from 0.20/1.x
  

• HADOOP-8703. Major bug reported by Dave Thompson and fixed by Dave Thompson
  distcpV2: turn CRC checking off for 0 byte size
  

  distcp skips CRC on 0 byte files.

• HADOOP-8700. Minor improvement reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (util)
  Move the checksum type constants to an enum
  

• HADOOP-8699. Critical bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (test)
  some common testcases create core-site.xml in test-classes making other testcases to fail
  

• HADOOP-8697. Major bug reported by Trevor Robinson and fixed by Trevor Robinson (test)
  TestWritableName fails intermittently with JDK7
  

• HADOOP-8695. Major bug reported by Trevor Robinson and fixed by Trevor Robinson (test)
  TestPathData fails intermittently with JDK7
  

• HADOOP-8693. Major bug reported by Trevor Robinson and fixed by Trevor Robinson (test)
  TestSecurityUtil fails intermittently with JDK7
  

• HADOOP-8692. Major bug reported by Trevor Robinson and fixed by Trevor Robinson (test)
  TestLocalDirAllocator fails intermittently with JDK7
  

• HADOOP-8689. Major improvement reported by Eli Collins and fixed by Eli Collins (fs)
  Make trash a server side configuration option
  

  If fs.trash.interval is configured on the server then the client's value for this configuration is ignored.

• HADOOP-8686. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (native)
  Fix warnings in native code
  

• HADOOP-8660. Major bug reported by Eli Collins and fixed by Alejandro Abdelnur
  TestPseudoAuthenticator failing with NPE
  

• HADOOP-8659. Major bug reported by Trevor Robinson and fixed by Colin Patrick McCabe (native)
  Native libraries must build with soft-float ABI for Oracle JVM on ARM
  

• HADOOP-8655. Major bug reported by Arun A K and fixed by (util)
  In TextInputFormat, while specifying textinputformat.record.delimiter the character/character sequences in data file similar to starting character/starting character sequence in delimiter were found missing in certain cases in the Map Output
  

• HADOOP-8654. Major bug reported by Gelesh and fixed by (util)
  TextInputFormat delimiter bug:- Input Text portion ends with & Delimiter starts with same char/char sequence
  

• HADOOP-8648. Major bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  libhadoop: native CRC32 validation crashes when io.bytes.per.checksum=1
  

• HADOOP-8644. Critical new feature reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  AuthenticatedURL should be able to use SSLFactory
  

• HADOOP-8637. Critical bug reported by Daryn Sharp and fixed by Daryn Sharp (fs)
  FilterFileSystem#setWriteChecksum is broken
  

• HADOOP-8635. Critical improvement reported by Daryn Sharp and fixed by Daryn Sharp (fs)
  Cannot cancel paths registered deleteOnExit
  

• HADOOP-8634. Critical bug reported by Daryn Sharp and fixed by Daryn Sharp (fs)
  Ensure FileSystem#close doesn't squawk for deleteOnExit paths
  

• HADOOP-8633. Critical bug reported by Daryn Sharp and fixed by Daryn Sharp (fs)
  Interrupted FsShell copies may leave tmp files
  

• HADOOP-8632. Major bug reported by Costin Leau and fixed by Costin Leau (conf)
  Configuration leaking class-loaders
  

• HADOOP-8627. Critical bug reported by Daryn Sharp and fixed by Daryn Sharp (fs)
  FS deleteOnExit may delete the wrong path
  

• HADOOP-8626. Major bug reported by Jonathan Natkins and fixed by Jonathan Natkins (security)
  Typo in default setting for hadoop.security.group.mapping.ldap.search.filter.user
  

• HADOOP-8624. Minor improvement reported by Todd Lipcon and fixed by Todd Lipcon (ipc)
  ProtobufRpcEngine should log all RPCs if TRACE logging is enabled
  

• HADOOP-8623. Minor improvement reported by Steven Willis and fixed by Steven Willis (scripts)
  hadoop jar command should respect HADOOP_OPTS
  

• HADOOP-8620. Minor improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (build)
  Add -Drequire.fuse and -Drequire.snappy
  

• HADOOP-8614. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  IOUtils#skipFully hangs forever on EOF
  

• HADOOP-8613. Critical bug reported by Daryn Sharp and fixed by Daryn Sharp
  AbstractDelegationTokenIdentifier#getUser() should set token auth type
  

• HADOOP-8611. Major bug reported by Kihwal Lee and fixed by Robert Parker (security)
  Allow fall-back to the shell-based implementation when JNI-based users-group mapping fails
  

• HADOOP-8609. Major improvement reported by Todd Lipcon and fixed by Jon Zuanich
  IPC server logs a useless message when shutting down socket
  

• HADOOP-8606. Major bug reported by Daryn Sharp and fixed by Daryn Sharp (fs)
  FileSystem.get may return the wrong filesystem
  

• HADOOP-8599. Major bug reported by Andrey Klochkov and fixed by Andrey Klochkov (fs)
  Non empty response from FileSystem.getFileBlockLocations when asking for data beyond the end of file
  

• HADOOP-8587. Minor bug reported by Eli Collins and fixed by Eli Collins (fs)
  HarFileSystem access of harMetaCache isn't threadsafe
  

• HADOOP-8586. Major bug reported by Eli Collins and fixed by Eli Collins
  Fixup a bunch of SPNEGO misspellings
  

• HADOOP-8585. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  Fix initialization circularity between UserGroupInformation and HadoopConfiguration
  

• HADOOP-8581. Major new feature reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  add support for HTTPS to the web UIs
  

• HADOOP-8573. Major bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans (conf)
  Configuration tries to read from an inputstream resource multiple times.
  

• HADOOP-8566. Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (io)
  AvroReflectSerializer.accept(Class) throws a NPE if the class has no package (primitive types and arrays)
  

• HADOOP-8563. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (build)
  don't package hadoop-pipes examples/bin
  

• HADOOP-8551. Major bug reported by Robert Joseph Evans and fixed by John George (fs)
  fs -mkdir creates parent directories without the -p option
  

  FsShell's "mkdir" no longer implicitly creates all non-existent parent directories. The command adopts the posix compliant behavior of requiring the "-p" flag to auto-create parent directories.

• HADOOP-8550. Major bug reported by Robert Joseph Evans and fixed by John George (fs)
  hadoop fs -touchz automatically created parent directories
  

• HADOOP-8547. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  Package hadoop-pipes examples/bin directory (again)
  

• HADOOP-8543. Major bug reported by Radim Kolar and fixed by Radim Kolar (build)
  Invalid pom.xml files on 0.23 branch
  

• HADOOP-8541. Major improvement reported by Andrew Wang and fixed by Andrew Wang (metrics)
  Better high-percentile latency metrics
  

• HADOOP-8538. Major bug reported by Trevor Robinson and fixed by Trevor Robinson (native)
  CMake builds fail on ARM
  

• HADOOP-8537. Major bug reported by Todd Lipcon and fixed by Todd Lipcon (io)
  Two TFile tests failing recently
  

• HADOOP-8535. Major improvement reported by Jonathan Eagles and fixed by Jonathan Eagles (build)
  Cut hadoop build times in half (upgrade maven-compiler-plugin to 2.5.1)
  

• HADOOP-8533. Major improvement reported by Suresh Srinivas and fixed by Brandon Li (ipc)
  Remove Parallel Call in IPC
  

  Merged the change to branch-2

• HADOOP-8531. Trivial improvement reported by Harsh J and fixed by madhukara phatak (io)
  SequenceFile Writer can throw out a better error if a serializer or deserializer isn't available
  

• HADOOP-8525. Trivial improvement reported by Robert Joseph Evans and fixed by Robert Joseph Evans
  Provide Improved Traceability for Configuration
  

• HADOOP-8524. Trivial improvement reported by Harsh J and fixed by Harsh J (conf)
  Allow users to get source of a Configuration parameter
  

• HADOOP-8512. Minor bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  AuthenticatedURL should reset the Token when the server returns other than OK on authentication
  

• HADOOP-8509. Minor bug reported by Matteo Bertozzi and fixed by Alejandro Abdelnur (util)
  JarFinder duplicate entry: META-INF/MANIFEST.MF exception
  

• HADOOP-8507. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  Avoid OOM while deserializing DelegationTokenIdentifer
  

• HADOOP-8501. Major bug reported by Radim Kolar and fixed by Radim Kolar (benchmarks)
  Gridmix fails to compile on OpenJDK7u4
  

• HADOOP-8499. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  Lower min.user.id to 500 for the tests
  

• HADOOP-8495. Critical bug reported by Jason Lowe and fixed by Jason Lowe (build)
  Update Netty to avoid leaking file descriptors during shuffle
  

• HADOOP-8488. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  test-patch.sh gives +1 even if the native build fails.
  

• HADOOP-8485. Minor bug reported by Eli Collins and fixed by Eli Collins (documentation)
  Don't hardcode "Apache Hadoop 0.23" in the docs
  

• HADOOP-8481. Trivial bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (documentation)
  update BUILDING.txt to talk about cmake rather than autotools
  

• HADOOP-8480. Trivial bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  The native build should honor -DskipTests
  

• HADOOP-8466. Major bug reported by Bruno Mahé and fixed by Bruno Mahé (build)
  hadoop-client POM incorrectly excludes avro
  

• HADOOP-8465. Major new feature reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  hadoop-auth should support ephemeral authentication
  

• HADOOP-8463. Major improvement reported by Eli Collins and fixed by madhukara phatak (security)
  hadoop.security.auth_to_local needs a key definition and doc
  

• HADOOP-8460. Major bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans (documentation)
  Document proper setting of HADOOP_PID_DIR and HADOOP_SECURE_DN_PID_DIR
  

• HADOOP-8458. Major new feature reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  Add management hook to AuthenticationHandler to enable delegation token operations support
  

• HADOOP-8452. Minor bug reported by Andy Isaacson and fixed by Andy Isaacson
  DN logs backtrace when running under jsvc and /jmx is loaded
  

• HADOOP-8450. Trivial bug reported by Colin Patrick McCabe and fixed by Eli Collins (test)
  Remove src/test/system
  

• HADOOP-8449. Minor bug reported by Joey Echeverria and fixed by Harsh J
  hadoop fs -text fails with compressed sequence files with the codec file extension
  

• HADOOP-8444. Major bug reported by Mariappan Asokan and fixed by madhukara phatak (fs , test)
  Fix the tests FSMainOperationsBaseTest.java and F ileContextMainOperationsBaseTest.java to avoid potential test failure
  

• HADOOP-8438. Major bug reported by Devaraj K and fixed by Devaraj K
  hadoop-validate-setup.sh refers to examples jar file which doesn't exist
  

• HADOOP-8433. Major bug reported by Brahma Reddy Battula and fixed by Brahma Reddy Battula (scripts)
  Don't set HADOOP_LOG_DIR in hadoop-env.sh
  

• HADOOP-8431. Major bug reported by Eli Collins and fixed by Sandy Ryza
  Running distcp wo args throws IllegalArgumentException
  

• HADOOP-8423. Major bug reported by Jason B and fixed by Todd Lipcon (io)
  MapFile.Reader.get() crashes jvm or throws EOFException on Snappy or LZO block-compressed data
  

• HADOOP-8422. Minor bug reported by Eli Collins and fixed by Eli Collins (fs)
  Deprecate FileSystem#getDefault* and getServerDefault methods that don't take a Path argument
  

• HADOOP-8408. Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (viewfs)
  MR doesn't work with a non-default ViewFS mount table and security enabled
  

• HADOOP-8406. Major bug reported by Todd Lipcon and fixed by Todd Lipcon (io)
  CompressionCodecFactory.CODEC_PROVIDERS iteration is thread-unsafe
  

• HADOOP-8403. Major task reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (build)
  bump up POMs version to 2.0.1-SNAPSHOT
  

• HADOOP-8400. Major bug reported by Eli Collins and fixed by Alejandro Abdelnur (security)
  All commands warn "Kerberos krb5 configuration not found" when security is not enabled
  

• HADOOP-8393. Major bug reported by Patrick Hunt and fixed by Patrick Hunt (scripts)
  hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster
  

• HADOOP-8390. Major bug reported by Trevor Robinson and fixed by Trevor Robinson (test)
  TestFileSystemCanonicalization fails with JDK7
  

• HADOOP-8373. Major improvement reported by Daryn Sharp and fixed by Daryn Sharp (ipc)
  Port RPC.getServerAddress to 0.23
  

• HADOOP-8372. Major bug reported by Junping Du and fixed by Junping Du (io , util)
  normalizeHostName() in NetUtils is not working properly in resolving a hostname start with numeric character
  

• HADOOP-8370. Major bug reported by Trevor Robinson and fixed by Trevor Robinson (native)
  Native build failure: javah: class file for org.apache.hadoop.classification.InterfaceAudience not found
  

• HADOOP-8368. Minor improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  Use CMake rather than autotools to build native code
  

• HADOOP-8367. Major improvement reported by Sanjay Radia and fixed by Sanjay Radia
  Improve documentation of declaringClassProtocolName in rpc headers
  

• HADOOP-8362. Trivial improvement reported by Todd Lipcon and fixed by madhukara phatak (conf)
  Improve exception message when Configuration.set() is called with a null key or value
  

• HADOOP-8361. Minor improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  Avoid out-of-memory problems when deserializing strings
  

• HADOOP-8358. Trivial improvement reported by Harsh J and fixed by Harsh J (conf)
  Config-related WARN for dfs.web.ugi can be avoided.
  

• HADOOP-8342. Major bug reported by Randy Clayton and fixed by Alejandro Abdelnur (fs)
  HDFS command fails with exception following merge of HADOOP-8325
  

• HADOOP-8341. Major bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans
  Fix or filter findbugs issues in hadoop-tools
  

• HADOOP-8340. Minor improvement reported by Todd Lipcon and fixed by Todd Lipcon (util)
  SNAPSHOT build versions should compare as less than their eventual final release
  

• HADOOP-8335. Major improvement reported by Daryn Sharp and fixed by Daryn Sharp (util)
  Improve Configuration's address handling
  

• HADOOP-8334. Major bug reported by Daryn Sharp and fixed by Daryn Sharp
  HttpServer sometimes returns incorrect port
  

• HADOOP-8330. Minor bug reported by John George and fixed by John George (test)
  TestSequenceFile.testCreateUsesFsArg() is broken
  

• HADOOP-8329. Major bug reported by Kumar Ravi and fixed by Eli Collins (build)
  Build fails with Java 7
  

• HADOOP-8328. Major bug reported by Tom White and fixed by Tom White (fs)
  Duplicate FileSystem Statistics object for 'file' scheme
  

• HADOOP-8327. Major bug reported by Dave Thompson and fixed by Dave Thompson
  distcpv2 and distcpv1 jars should not coexist
  

  Resolve sporadic distcp issue due to having two DistCp classes (v1 & v2) in the classpath.

• HADOOP-8325. Critical bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (fs)
  Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
  

• HADOOP-8323. Critical improvement reported by Harsh J and fixed by Harsh J (io)
  Revert HADOOP-7940 and improve javadocs and test for Text.clear()
  

• HADOOP-8317. Major bug reported by Radim Kolar and fixed by (build)
  Update maven-assembly-plugin to 2.3 - fix build on FreeBSD
  

• HADOOP-8316. Major bug reported by Eli Collins and fixed by Eli Collins (conf)
  Audit logging should be disabled by default
  

• HADOOP-8305. Major bug reported by John George and fixed by John George (viewfs)
  distcp over viewfs is broken
  

• HADOOP-8288. Major bug reported by Ravi Prakash and fixed by Ravi Prakash
  Remove references of mapred.child.ulimit etc. since they are not being used any more
  

• HADOOP-8287. Major bug reported by Eli Collins and fixed by Eli Collins (conf)
  etc/hadoop is missing hadoop-env.sh
  

• HADOOP-8286. Major improvement reported by Daryn Sharp and fixed by Daryn Sharp (conf)
  Simplify getting a socket address from conf
  

• HADOOP-8283. Major test reported by Daryn Sharp and fixed by Daryn Sharp (test)
  Allow tests to control token service value
  

• HADOOP-8278. Major improvement reported by Tom White and fixed by Tom White (build)
  Make sure components declare correct set of dependencies
  

• HADOOP-8268. Major bug reported by Radim Kolar and fixed by Radim Kolar (build)
  A few pom.xml across Hadoop project may fail XML validation
  

• HADOOP-8244. Major improvement reported by Henry Robinson and fixed by Henry Robinson
  Improve comments on ByteBufferReadable.read
  

• HADOOP-8242. Minor improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  AbstractDelegationTokenIdentifier: add getter methods for owner and realuser
  

• HADOOP-8240. Major improvement reported by Kihwal Lee and fixed by Kihwal Lee (fs)
  Allow users to specify a checksum type on create()
  

• HADOOP-8239. Major improvement reported by Kihwal Lee and fixed by Kihwal Lee (fs)
  Extend MD5MD5CRC32FileChecksum to show the actual checksum type being used
  

• HADOOP-8227. Blocker improvement reported by Robert Joseph Evans and fixed by Robert Joseph Evans
  Allow RPC to limit ephemeral port range.
  

• HADOOP-8225. Blocker bug reported by Mithun Radhakrishnan and fixed by Daryn Sharp (security)
  DistCp fails when invoked by Oozie
  

• HADOOP-8224. Major improvement reported by Eli Collins and fixed by Tomohiko Kinebuchi (conf)
  Don't hardcode hdfs.audit.logger in the scripts
  

• HADOOP-8197. Critical bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (conf)
  Configuration logs WARNs on every use of a deprecated key
  

• HADOOP-8180. Major bug reported by Ravi Prakash and fixed by Ravi Prakash
  Remove hsqldb since its not needed from pom.xml
  

• HADOOP-8179. Minor bug reported by Steve Loughran and fixed by Daryn Sharp (fs)
  risk of NPE in CopyCommands processArguments()
  

• HADOOP-8172. Critical bug reported by Robert Joseph Evans and fixed by Anupam Seth (conf)
  Configuration no longer sets all keys in a deprecated key list.
  

• HADOOP-8168. Major bug reported by Eugene Koontz and fixed by Eugene Koontz (fs)
  empty-string owners or groups causes {{MissingFormatWidthException}} in o.a.h.fs.shell.Ls.ProcessPath()
  

• HADOOP-8167. Blocker bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (conf)
  Configuration deprecation logic breaks backwards compatibility
  

• HADOOP-8144. Minor bug reported by Junping Du and fixed by Junping Du (io)
  pseudoSortByDistance in NetworkTopology doesn't work properly if no local node and first node is local rack node
  

• HADOOP-8135. Major new feature reported by Henry Robinson and fixed by Henry Robinson (fs)
  Add ByteBufferReadable interface to FSDataInputStream
  

• HADOOP-8129. Major bug reported by Ravi Prakash and fixed by Ahmed Radwan (fs , test)
  ViewFileSystemTestSetup setupForViewFileSystem is erring when the user's home directory is somewhere other than /home (eg. /User) etc.
  

• HADOOP-8110. Major bug reported by Tsz Wo (Nicholas), SZE and fixed by Jason Lowe (fs)
  TestViewFsTrash occasionally fails
  

• HADOOP-8104. Major bug reported by Colin Patrick McCabe and fixed by Alejandro Abdelnur
  Inconsistent Jackson versions
  

• HADOOP-8088. Major bug reported by Kihwal Lee and fixed by Kihwal Lee (security)
  User-group mapping cache incorrectly does negative caching on transient failures
  

• HADOOP-8075. Major improvement reported by Eli Collins and fixed by Hızır Sefa İrken (native)
  Lower native-hadoop library log from info to debug
  

• HADOOP-8060. Major bug reported by Kihwal Lee and fixed by Kihwal Lee (fs , util)
  Add a capability to discover and set checksum types per file.
  

• HADOOP-8031. Major bug reported by Elias Ross and fixed by Elias Ross (conf)
  Configuration class fails to find embedded .jar resources; should use URL.openStream()
  

• HADOOP-8014. Major bug reported by Daryn Sharp and fixed by John George (fs)
  ViewFileSystem does not correctly implement getDefaultBlockSize, getDefaultReplication, getContentSummary
  

• HADOOP-8005. Major bug reported by Joe Crobak and fixed by Jason Lowe (scripts)
  Multiple SLF4J binding message in .out file for all daemons
  

• HADOOP-7967. Critical bug reported by Daryn Sharp and fixed by Daryn Sharp (fs , security)
  Need generalized multi-token filesystem support
  

• HADOOP-7868. Major bug reported by James Page and fixed by Trevor Robinson (native)
  Hadoop native fails to compile when default linker option is -Wl,--as-needed
  

• HADOOP-7818. Minor bug reported by Eli Collins and fixed by madhukara phatak (util)
  DiskChecker#checkDir should fail if the directory is not executable
  

• HADOOP-7754. Major sub-task reported by Todd Lipcon and fixed by Todd Lipcon (native , performance)
  Expose file descriptors from Hadoop-wrapped local FileSystems
  

• HADOOP-7703. Major bug reported by Devaraj K and fixed by Devaraj K
  WebAppContext should also be stopped and cleared
  

  Improved excpetion handling of shutting down web server. (Devaraj K via Eric Yang)

• HADOOP-7510. Major improvement reported by Daryn Sharp and fixed by Daryn Sharp (security)
  Tokens should use original hostname provided instead of ip
  

• HADOOP-6963. Critical bug reported by Owen O'Malley and fixed by Ravi Prakash (fs)
  Fix FileUtil.getDU. It should not include the size of the directory or follow symbolic links
  

• HADOOP-6802. Major improvement reported by Erik Steffl and fixed by Sho Shimauchi (conf , fs)
  Remove FS_CLIENT_BUFFER_DIR_KEY = "fs.client.buffer.dir" from CommonConfigurationKeys.java (not used, deprecated)
  

• HADOOP-3886. Minor bug reported by brien colwell and fixed by Jingguo Yao (documentation)
  Error in javadoc of Reporter, Mapper and Progressable
  

• HADOOP-3450. Minor improvement reported by Ari Rabkin and fixed by Sho Shimauchi (fs)
  Add tests to Local Directory Allocator for asserting their URI-returning capability
  

Hadoop 2.0.1-alpha Release Notes

• HADOOP-8552. Major bug reported by Karthik Kambatla and fixed by Karthik Kambatla (conf , security)
  Conflict: Same security.log.file for multiple users.
  

Hadoop 2.0.0-alpha Release Notes

• MAPREDUCE-4274. Minor improvement reported by Todd Lipcon and fixed by Todd Lipcon (performance , task)
  MapOutputBuffer should use native byte order for kvmeta
  

• MAPREDUCE-4231. Major bug reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (contrib/raid)
  Update RAID to not to use FSInodeInfo
  

• MAPREDUCE-4219. Major improvement reported by Roman Shaposhnik and fixed by Roman Shaposhnik (security)
  make default container-executor.conf.dir be a path relative to the container-executor binary
  

• MAPREDUCE-4202. Major bug reported by Daryn Sharp and fixed by Daryn Sharp (test)
  TestYarnClientProtocolProvider is broken
  

• MAPREDUCE-4193. Major bug reported by Patrick Hunt and fixed by Patrick Hunt (documentation)
  broken doc link for yarn-default.xml in site.xml
  

• MAPREDUCE-4147. Major bug reported by Tom White and fixed by Tom White
  YARN should not have a compile-time dependency on HDFS
  

• MAPREDUCE-4138. Major improvement reported by Tom White and fixed by Tom White
  Reduce memory usage of counters due to non-static nested classes
  

• MAPREDUCE-4113. Major sub-task reported by Devaraj K and fixed by Devaraj K (mrv2 , test)
  Fix tests org.apache.hadoop.mapred.TestClusterMRNotification
  

• MAPREDUCE-4112. Major sub-task reported by Devaraj K and fixed by Devaraj K (mrv2 , test)
  Fix tests org.apache.hadoop.mapred.TestClusterMapReduceTestCase
  

• MAPREDUCE-4111. Major sub-task reported by Devaraj K and fixed by Devaraj K (mrv2 , test)
  Fix tests in org.apache.hadoop.mapred.TestJobName
  

• MAPREDUCE-4110. Major sub-task reported by Devaraj K and fixed by Devaraj K (mrv2 , test)
  Fix tests in org.apache.hadoop.mapred.TestMiniMRClasspath & org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
  

• MAPREDUCE-4108. Major sub-task reported by Devaraj K and fixed by Devaraj K (mrv2)
  Fix tests in org.apache.hadoop.util.TestRunJar
  

• MAPREDUCE-4107. Major sub-task reported by Devaraj K and fixed by Devaraj K (mrv2)
  Fix tests in org.apache.hadoop.ipc.TestSocketFactory
  

• MAPREDUCE-4105. Major bug reported by Ahmed Radwan and fixed by Ahmed Radwan (mrv2)
  Yarn RackResolver ignores rack configurations
  

• MAPREDUCE-4103. Major improvement reported by Todd Lipcon and fixed by Todd Lipcon (documentation)
  Fix HA docs for changes to shell command fencer args
  

• MAPREDUCE-4098. Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (test)
  TestMRApps testSetClasspath fails
  

• MAPREDUCE-4093. Major improvement reported by Devaraj K and fixed by Devaraj K (mrv2)
  Improve RM WebApp start up when proxy address is not set
  

• MAPREDUCE-4081. Blocker bug reported by Jason Lowe and fixed by Jason Lowe (build , mrv2)
  TestMROutputFormat.java does not compile
  

• MAPREDUCE-4076. Blocker bug reported by Devaraj K and fixed by Devaraj K (mrv2)
  Stream job fails with ZipException when use yarn jar command
  

• MAPREDUCE-4066. Minor bug reported by xieguiming and fixed by xieguiming (job submission , mrv2)
  To get "yarn.app.mapreduce.am.staging-dir" value, should set the default value
  

• MAPREDUCE-4057. Major bug reported by Tsz Wo (Nicholas), SZE and fixed by Devaraj K (contrib/raid)
  Compilation error in RAID
  

• MAPREDUCE-4008. Major bug reported by Devaraj K and fixed by Devaraj K (mrv2 , scheduler)
  ResourceManager throws MetricsException on start up saying QueueMetrics MBean already exists
  

• MAPREDUCE-4007. Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (mrv2)
  JobClient getJob(JobID) should return NULL if the job does not exist (for backwards compatibility)
  

• MAPREDUCE-3991. Trivial improvement reported by Harsh J and fixed by Harsh J (documentation)
  Streaming FAQ has some wrong instructions about input files splitting
  

• MAPREDUCE-3989. Major improvement reported by Patrick Hunt and fixed by Patrick Hunt
  cap space usage of default log4j rolling policy (mr specific changes)
  

• MAPREDUCE-3974. Blocker bug reported by Arun C Murthy and fixed by Aaron T. Myers
  TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge
  

• MAPREDUCE-3958. Major bug reported by Bikas Saha and fixed by Bikas Saha (mrv2)
  RM: Remove RMNodeState and replace it with NodeState
  

• MAPREDUCE-3955. Blocker improvement reported by Jitendra Nath Pandey and fixed by Jitendra Nath Pandey (mrv2)
  Replace ProtoOverHadoopRpcEngine with ProtobufRpcEngine.
  

• MAPREDUCE-3952. Major bug reported by Zhenxiao Luo and fixed by Bhallamudi Venkata Siva Kamesh (mrv2)
  In MR2, when Total input paths to process == 1, CombinefileInputFormat.getSplits() returns 0 split.
  

• MAPREDUCE-3935. Major improvement reported by Tom White and fixed by Tom White (client)
  Annotate Counters.Counter and Counters.Group as @Public
  

• MAPREDUCE-3933. Major bug reported by Ahmed Radwan and fixed by Ahmed Radwan (mrv2 , test)
  Failures because MALLOC_ARENA_MAX is not set
  

• MAPREDUCE-3916. Critical bug reported by Roman Shaposhnik and fixed by Devaraj K (mrv2 , resourcemanager , webapps)
  various issues with running yarn proxyserver
  

• MAPREDUCE-3909. Trivial improvement reported by Steve Loughran and fixed by Steve Loughran (mrv2)
  javadoc the Service interfaces
  

• MAPREDUCE-3885. Major improvement reported by Devaraj Das and fixed by Devaraj Das (mrv2)
  Apply the fix similar to HADOOP-8084
  

• MAPREDUCE-3883. Minor improvement reported by Eugene Koontz and fixed by Eugene Koontz (documentation , mrv2)
  Document yarn.nodemanager.delete.debug-delay-sec configuration property
  

• MAPREDUCE-3869. Blocker bug reported by Devaraj K and fixed by Devaraj K (mrv2)
  Distributed shell application fails with NoClassDefFoundError
  

• MAPREDUCE-3867. Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (test)
  MiniMRYarn/MiniYarn uses fixed ports
  

• MAPREDUCE-3818. Blocker bug reported by Vinod Kumar Vavilapalli and fixed by Suresh Srinivas (build , test)
  Trunk MRV1 compilation is broken.
  

  Fixed broken compilation in TestSubmitJob after the patch for HDFS-2895.

• MAPREDUCE-3740. Blocker bug reported by Devaraj K and fixed by Devaraj K (mrv2)
  Mapreduce Trunk compilation fails
  

• MAPREDUCE-3578. Major bug reported by Gilad Wolff and fixed by Tom White (nodemanager)
  starting nodemanager as 'root' gives "Unknown -jvm option"
  

• MAPREDUCE-3545. Major bug reported by Suresh Srinivas and fixed by Suresh Srinivas
  Remove Avro RPC
  

• MAPREDUCE-3431. Minor bug reported by Steve Loughran and fixed by Steve Loughran (resourcemanager)
  NPE in Resource Manager shutdown
  

• MAPREDUCE-3377. Major bug reported by Jane Chen and fixed by Jane Chen
  Compatibility issue with 0.20.203.
  

• MAPREDUCE-3353. Major bug reported by Vinod Kumar Vavilapalli and fixed by Bikas Saha (applicationmaster , mrv2 , resourcemanager)
  Need a RM->AM channel to inform AMs about faulty/unhealthy/lost nodes
  

• MAPREDUCE-3173. Critical bug reported by Devaraj K and fixed by Devaraj K (mrv2)
  MRV2 UI doesn't work properly without internet
  

• MAPREDUCE-2942. Critical bug reported by Vinod Kumar Vavilapalli and fixed by Thomas Graves
  TestNMAuditLogger.testNMAuditLoggerWithIP failing
  

• MAPREDUCE-2934. Major improvement reported by Aaron T. Myers and fixed by Aaron T. Myers (mrv2)
  MR portion of HADOOP-7607 - Simplify the RPC proxy cleanup process
  

• MAPREDUCE-2887. Major improvement reported by Sanjay Radia and fixed by Sanjay Radia
  MR changes to match HADOOP-7524 (multiple RPC protocols)
  

• HDFS-3418. Minor improvement reported by Eli Collins and fixed by Eli Collins
  Rename BlockWithLocationsProto datanodeIDs field to storageIDs
  

• HDFS-3396. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (fuse-dfs)
  FUSE build fails on Ubuntu 12.04
  

• HDFS-3395. Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (name-node)
  NN doesn't start with HA+security enabled and HTTP address set to 0.0.0.0
  

• HDFS-3378. Trivial improvement reported by Eli Collins and fixed by Eli Collins
  Remove DFS_NAMENODE_SECONDARY_HTTPS_PORT_KEY and DEFAULT
  

• HDFS-3376. Critical bug reported by Todd Lipcon and fixed by Todd Lipcon (hdfs client)
  DFSClient fails to make connection to DN if there are many unusable cached sockets
  

• HDFS-3375. Trivial improvement reported by Todd Lipcon and fixed by Todd Lipcon (data-node)
  Put client name in DataXceiver thread name for readBlock and keepalive
  

• HDFS-3365. Minor improvement reported by Todd Lipcon and fixed by Todd Lipcon (hdfs client)
  Enable users to disable socket caching in DFS client configuration
  

• HDFS-3363. Minor sub-task reported by John George and fixed by John George (name-node)
  blockmanagement should stop using INodeFile & INodeFileUC
  

• HDFS-3357. Critical bug reported by Todd Lipcon and fixed by Todd Lipcon (data-node)
  DataXceiver reads from client socket with incorrect/no timeout
  

• HDFS-3351. Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (name-node)
  NameNode#initializeGenericKeys should always set fs.defaultFS regardless of whether HA or Federation is enabled
  

• HDFS-3350. Major bug reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (name-node)
  findbugs warning: INodeFileUnderConstruction doesn't override INodeFile.equals(Object)
  

• HDFS-3339. Minor sub-task reported by John George and fixed by John George (name-node)
  change INode to package private
  

• HDFS-3336. Minor bug reported by Roman Shaposhnik and fixed by Roman Shaposhnik (scripts)
  hdfs launcher script will be better off not special casing namenode command with regards to hadoop.security.logger
  

• HDFS-3332. Major bug reported by amith and fixed by amith (data-node)
  NullPointerException in DN when directoryscanner is trying to report bad blocks
  

• HDFS-3330. Critical bug reported by Todd Lipcon and fixed by Todd Lipcon (name-node)
  If GetImageServlet throws an Error or RTE, response has HTTP "OK" status
  

• HDFS-3328. Minor bug reported by Uma Maheswara Rao G and fixed by Eli Collins (data-node)
  NPE in DataNode.getIpcPort
  

• HDFS-3326. Trivial bug reported by J.Andreina and fixed by Matthew Jacobs (name-node)
  Append enabled log message uses the wrong variable
  

• HDFS-3322. Major sub-task reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (hdfs client)
  Update file context to use HdfsDataInputStream and HdfsDataOutputStream
  

• HDFS-3319. Minor improvement reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (hdfs client)
  DFSOutputStream should not start a thread in constructors
  

• HDFS-3314. Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur
  HttpFS operation for getHomeDirectory is incorrect
  

• HDFS-3309. Major bug reported by Romain Rigaux and fixed by Alejandro Abdelnur
  HttpFS (Hoop) chmod not supporting octal and sticky bit permissions
  

• HDFS-3305. Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (ha , name-node)
  GetImageServlet should consider SBN a valid requestor in a secure HA setup
  

• HDFS-3303. Minor bug reported by Brandon Li and fixed by Brandon Li (name-node)
  RemoteEditLogManifest doesn't need to implements Writable
  

• HDFS-3298. Major sub-task reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (hdfs client)
  Add HdfsDataOutputStream as a public API
  

• HDFS-3294. Trivial improvement reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (data-node , name-node)
  Fix indentation in NamenodeWebHdfsMethods and DatanodeWebHdfsMethods
  

• HDFS-3286. Major bug reported by J.Andreina and fixed by Ashish Singhi (balancer)
  When the threshold value for balancer is 0(zero) ,unexpected output is displayed
  

• HDFS-3284. Minor bug reported by Todd Lipcon and fixed by Todd Lipcon (ha , security)
  bootstrapStandby fails in secure cluster
  

• HDFS-3282. Major sub-task reported by Uma Maheswara Rao G and fixed by Uma Maheswara Rao G (hdfs client)
  Add HdfsDataInputStream as a public API
  

• HDFS-3280. Critical bug reported by Todd Lipcon and fixed by Todd Lipcon (hdfs client)
  DFSOutputStream.sync should not be synchronized
  

• HDFS-3279. Minor improvement reported by Tsz Wo (Nicholas), SZE and fixed by Arpit Gupta (name-node)
  One of the FSEditLog constructors should be moved to TestEditLog
  

• HDFS-3275. Major bug reported by Vinithra Varadharajan and fixed by amith (ha , name-node)
  Format command overwrites contents of non-empty shared edits dir if name dirs are empty without any prompting
  

• HDFS-3268. Critical bug reported by Daryn Sharp and fixed by Daryn Sharp (ha , hdfs client)
  Hdfs mishandles token service & incompatible with HA
  

• HDFS-3263. Major improvement reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur
  HttpFS should read HDFS config from Hadoop site.xml files
  

• HDFS-3260. Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers
  TestDatanodeRegistration should set minimum DN version in addition to minimum NN version
  

• HDFS-3259. Major improvement reported by Aaron T. Myers and fixed by Aaron T. Myers (ha , name-node)
  NameNode#initializeSharedEdits should populate shared edits dir with edit log segments
  

• HDFS-3256. Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers
  HDFS considers blocks under-replicated if topology script is configured with only 1 rack
  

• HDFS-3255. Critical bug reported by Daryn Sharp and fixed by Daryn Sharp (ha , hdfs client)
  HA DFS returns wrong token service
  

• HDFS-3254. Major bug reported by Anupam Seth and fixed by Anupam Seth (fuse-dfs)
  Branch-2 build broken due to wrong version number in fuse-dfs' pom.xml
  

• HDFS-3249. Trivial improvement reported by Todd Lipcon and fixed by Todd Lipcon (name-node)
  Use ToolRunner.confirmPrompt in NameNode
  

• HDFS-3248. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  bootstrapstanby repeated twice in hdfs namenode usage message
  

• HDFS-3247. Minor improvement reported by Todd Lipcon and fixed by Todd Lipcon (ha)
  Improve bootstrapStandby behavior when original NN is not active
  

• HDFS-3244. Major improvement reported by Eli Collins and fixed by Eli Collins
  Remove dead writable code from hdfs/protocol
  

• HDFS-3240. Trivial improvement reported by Todd Lipcon and fixed by Todd Lipcon (data-node)
  Drop log level of "heartbeat: ..." in BPServiceActor to DEBUG
  

• HDFS-3238. Major improvement reported by Eli Collins and fixed by Eli Collins
  ServerCommand and friends don't need to be writables
  

• HDFS-3236. Minor bug reported by Aaron T. Myers and fixed by Aaron T. Myers (ha , name-node)
  NameNode does not initialize generic conf keys when started with -initializeSharedEditsDir
  

• HDFS-3234. Trivial bug reported by Todd Lipcon and fixed by Todd Lipcon (tools)
  Accidentally left log message in GetConf after HDFS-3226
  

• HDFS-3226. Major improvement reported by Todd Lipcon and fixed by Todd Lipcon (tools)
  Allow GetConf tool to print arbitrary keys
  

• HDFS-3222. Major bug reported by Uma Maheswara Rao G and fixed by Uma Maheswara Rao G (hdfs client)
  DFSInputStream#openInfo should not silently get the length as 0 when locations length is zero for last partial block.
  

• HDFS-3214. Blocker bug reported by Todd Lipcon and fixed by Todd Lipcon (data-node)
  InterDatanodeProtocolServerSideTranslatorPB doesn't handle null response from initReplicaRecovery
  

• HDFS-3211. Major sub-task reported by Suresh Srinivas and fixed by Suresh Srinivas (ha , name-node)
  JournalProtocol changes required for introducing epoch and fencing
  

• HDFS-3210. Major bug reported by Eli Collins and fixed by Eli Collins
  JsonUtil#toJsonMap for for a DatanodeInfo should use "ipAddr" instead of "name"
  

• HDFS-3208. Major bug reported by Eli Collins and fixed by Eli Collins (name-node)
  Bogus entries in hosts files are incorrectly displayed in the report
  

• HDFS-3204. Major improvement reported by Suresh Srinivas and fixed by Suresh Srinivas (name-node)
  Minor modification to JournalProtocol.proto to make it generic
  

• HDFS-3202. Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (data-node)
  NamespaceInfo PB translation drops build version
  

• HDFS-3199. Major bug reported by Eli Collins and fixed by Todd Lipcon
  TestValidateConfigurationSettings is failing
  

• HDFS-3187. Minor sub-task reported by Todd Lipcon and fixed by Todd Lipcon (build)
  Upgrade guava to 11.0.2
  

• HDFS-3181. Minor bug reported by Colin Patrick McCabe and fixed by Tsz Wo (Nicholas), SZE (test)
  testHardLeaseRecoveryAfterNameNodeRestart fails when length before restart is 1 byte less than CRC chunk size
  

• HDFS-3179. Major improvement reported by Zhanwei.Wang and fixed by Tsz Wo (Nicholas), SZE (hdfs client)
  Improve the error message: DataStreamer throw an exception, "nodes.length != original.length + 1" on single datanode cluster
  

• HDFS-3172. Trivial improvement reported by Eli Collins and fixed by Eli Collins (name-node)
  dfs.upgrade.permission is dead code
  

• HDFS-3171. Major improvement reported by Eli Collins and fixed by Eli Collins (data-node)
  The DatanodeID "name" field is overloaded
  

• HDFS-3169. Minor improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (test)
  TestFsck should test multiple -move operations in a row
  

• HDFS-3167. Minor new feature reported by Henry Robinson and fixed by Henry Robinson (test)
  CLI-based driver for MiniDFSCluster
  

• HDFS-3164. Major improvement reported by Eli Collins and fixed by Eli Collins (data-node)
  Move DatanodeInfo#hostName to DatanodeID
  

  This change modifies DatanodeID, which is part of the client to server protocol, therefore clients must be upgraded with servers.

• HDFS-3160. Major bug reported by Roman Shaposhnik and fixed by Roman Shaposhnik (scripts)
  httpfs should exec catalina instead of forking it
  

• HDFS-3158. Major improvement reported by Aaron T. Myers and fixed by Aaron T. Myers (name-node)
  LiveNodes member of NameNodeMXBean should list non-DFS used space and capacity per DN
  

• HDFS-3156. Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (test)
  TestDFSHAAdmin is failing post HADOOP-8202
  

• HDFS-3155. Major sub-task reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (data-node)
  Clean up FSDataset implemenation related code.
  

• HDFS-3148. Major new feature reported by Eli Collins and fixed by Eli Collins (hdfs client , performance)
  The client should be able to use multiple local interfaces for data transfer
  

• HDFS-3144. Major improvement reported by Eli Collins and fixed by Eli Collins (data-node)
  Refactor DatanodeID#getName by use
  

  This change modifies DatanodeID, which is part of the client to server protocol, therefore clients must be upgraded with servers.

• HDFS-3143. Major bug reported by Eli Collins and fixed by Arpit Gupta (test)
  TestGetBlocks.testGetBlocks is failing
  

• HDFS-3142. Blocker bug reported by Eli Collins and fixed by Brandon Li (test)
  TestHDFSCLI.testAll is failing
  

• HDFS-3139. Minor improvement reported by Eli Collins and fixed by Eli Collins (data-node)
  Minor Datanode logging improvement
  

• HDFS-3138. Major improvement reported by Eli Collins and fixed by Eli Collins
  Move DatanodeInfo#ipcPort to DatanodeID
  

  This change modifies DatanodeID, which is part of the client to server protocol, therefore clients must be upgraded with servers.

• HDFS-3137. Major improvement reported by Eli Collins and fixed by Eli Collins (name-node)
  Bump LAST_UPGRADABLE_LAYOUT_VERSION to -16
  

  Upgrade from Hadoop versions earlier than 0.18 is not supported as of 2.0. To upgrade from an earlier release, first upgrade to 0.18, and then upgrade again from there.

• HDFS-3132. Minor bug reported by Todd Lipcon and fixed by Todd Lipcon (name-node)
  Findbugs warning on HDFS trunk
  

• HDFS-3130. Major sub-task reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (data-node)
  Move FSDataset implemenation to a package
  

• HDFS-3129. Minor test reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  NetworkTopology: add test that getLeaf should check for invalid topologies
  

• HDFS-3121. Major bug reported by John George and fixed by John George
  hdfs tests for HADOOP-8014
  

• HDFS-3120. Major improvement reported by Eli Collins and fixed by Eli Collins
  Enable hsync and hflush by default
  

• HDFS-3119. Minor bug reported by J.Andreina and fixed by Ashish Singhi (name-node)
  Overreplicated block is not deleted even after the replication factor is reduced after sync follwed by closing that file
  

• HDFS-3111. Trivial task reported by Todd Lipcon and fixed by Uma Maheswara Rao G
  Missing license headers in trunk
  

• HDFS-3109. Major bug reported by Ravi Prakash and fixed by Ravi Prakash
  Remove hsqldb exclusions from pom.xml
  

• HDFS-3105. Major sub-task reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (data-node , hdfs client)
  Add DatanodeStorage information to block recovery
  

• HDFS-3102. Major new feature reported by Todd Lipcon and fixed by Aaron T. Myers (ha , name-node)
  Add CLI tool to initialize the shared-edits dir
  

• HDFS-3101. Major bug reported by Zhanwei.Wang and fixed by Tsz Wo (Nicholas), SZE (webhdfs)
  cannot read empty file using webhdfs
  

• HDFS-3100. Major bug reported by Zhanwei.Wang and fixed by Brandon Li (data-node)
  failed to append data
  

• HDFS-3099. Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (name-node)
  SecondaryNameNode does not properly initialize metrics system
  

• HDFS-3094. Major improvement reported by Arpit Gupta and fixed by Arpit Gupta
  add -nonInteractive and -force option to namenode -format command
  

  The 'namenode -format' command now supports the flags '-nonInteractive' and '-force' to improve usefulness without user input.

• HDFS-3093. Critical bug reported by Todd Lipcon and fixed by Todd Lipcon
  TestAllowFormat is trying to be interactive
  

• HDFS-3091. Major improvement reported by Uma Maheswara Rao G and fixed by Tsz Wo (Nicholas), SZE (data-node , hdfs client , name-node)
  Update the usage limitations of ReplaceDatanodeOnFailure policy in the config description for the smaller clusters.
  

• HDFS-3089. Major sub-task reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (data-node)
  Move FSDatasetInterface and other related classes/interfaces to a package
  

• HDFS-3088. Major sub-task reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (data-node)
  Move FSDatasetInterface inner classes to a package
  

• HDFS-3086. Major sub-task reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (data-node)
  Change Datanode not to send storage list in registration - it will be sent in block report
  

• HDFS-3084. Major improvement reported by Philip Zeyliger and fixed by Todd Lipcon (ha)
  FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port
  

• HDFS-3083. Critical bug reported by Mingjie Lai and fixed by Aaron T. Myers (ha , security)
  Cannot run an MR job with HA and security enabled when second-listed NN active
  

• HDFS-3082. Major sub-task reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (data-node)
  Clean up FSDatasetInterface
  

• HDFS-3071. Major improvement reported by Philip Zeyliger and fixed by Todd Lipcon (ha)
  haadmin failover command does not provide enough detail for when target NN is not ready to be active
  

• HDFS-3070. Major bug reported by Stephen Chu and fixed by Aaron T. Myers (balancer)
  HDFS balancer doesn't ensure that hdfs-site.xml is loaded
  

• HDFS-3066. Major improvement reported by Patrick Hunt and fixed by Patrick Hunt (scripts)
  cap space usage of default log4j rolling policy (hdfs specific changes)
  

• HDFS-3062. Critical bug reported by Mingjie Lai and fixed by Mingjie Lai (ha , security)
  Fail to submit mapred job on a secured-HA-HDFS: logic URI cannot be picked up by job submission.
  

• HDFS-3057. Major bug reported by Roman Shaposhnik and fixed by Roman Shaposhnik (scripts)
  httpfs and hdfs launcher scripts should honor CATALINA_HOME and HADOOP_LIBEXEC_DIR
  

• HDFS-3056. Major improvement reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (data-node)
  Add an interface for DataBlockScanner logging
  

• HDFS-3050. Minor improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (name-node)
  rework OEV to share more code with the NameNode
  

• HDFS-3044. Major improvement reported by Eli Collins and fixed by Colin Patrick McCabe (name-node)
  fsck move should be non-destructive by default
  

  The fsck "move" option is no longer destructive. It copies the accessible blocks of corrupt files to lost and found as before, but no longer deletes the corrupt files after copying the blocks. The original, destructive behavior can be enabled by specifying both the "move" and "delete" options.

• HDFS-3038. Trivial bug reported by Todd Lipcon and fixed by Todd Lipcon
  Add FSEditLog.metrics to findbugs exclude list
  

• HDFS-3036. Trivial improvement reported by Aaron T. Myers and fixed by Aaron T. Myers (name-node)
  Remove unused method DFSUtil#isDefaultNamenodeAddress
  

• HDFS-3032. Major bug reported by Kihwal Lee and fixed by Kihwal Lee (hdfs client)
  Lease renewer tries forever even if renewal is not possible
  

• HDFS-3030. Major improvement reported by Jitendra Nath Pandey and fixed by Jitendra Nath Pandey
  Remove getProtocolVersion and getProtocolSignature from translators
  

• HDFS-3026. Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (ha , name-node)
  HA: Handle failure during HA state transition
  

• HDFS-3024. Minor improvement reported by Todd Lipcon and fixed by Todd Lipcon (name-node)
  Improve performance of stringification in addStoredBlock
  

• HDFS-3021. Major improvement reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (data-node)
  Use generic type to declare FSDatasetInterface
  

• HDFS-3020. Critical bug reported by Todd Lipcon and fixed by Todd Lipcon (name-node)
  Auto-logSync based on edit log buffer size broken
  

• HDFS-3014. Major improvement reported by Sho Shimauchi and fixed by Sho Shimauchi (name-node)
  FSEditLogOp and its subclasses should have toString() method
  

• HDFS-3005. Major bug reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (data-node)
  ConcurrentModificationException in FSDataset$FSVolume.getDfsUsed(..)
  

• HDFS-3004. Major new feature reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (tools)
  Implement Recovery Mode
  

  This is a new feature. It is documented in hdfs_user_guide.xml.

• HDFS-3003. Trivial improvement reported by Brandon Li and fixed by Brandon Li (name-node)
  Remove getHostPortString() from NameNode, replace it with NetUtils.getHostPortString()
  

• HDFS-3000. Major new feature reported by Aaron T. Myers and fixed by Aaron T. Myers (hdfs client)
  Add a public API for setting quotas
  

• HDFS-2995. Major bug reported by Todd Lipcon and fixed by Eli Collins (scripts)
  start-dfs.sh should only start the 2NN for namenodes with dfs.namenode.secondary.http-address configured
  

• HDFS-2983. Major improvement reported by Eli Collins and fixed by Aaron T. Myers
  Relax the build version check to permit rolling upgrades within a release
  

• HDFS-2968. Blocker bug reported by Todd Lipcon and fixed by Todd Lipcon (data-node , name-node)
  Protocol translator for BlockRecoveryCommand broken when multiple blocks need recovery
  

• HDFS-2941. Major new feature reported by Aaron T. Myers and fixed by Aaron T. Myers (hdfs client , name-node)
  Add an administrative command to download a copy of the fsimage from the NN
  

• HDFS-2899. Major sub-task reported by Suresh Srinivas and fixed by Suresh Srinivas
  Service protocol change to support multiple storages added in HDFS-2880
  

• HDFS-2895. Major improvement reported by Suresh Srinivas and fixed by Suresh Srinivas (data-node , name-node)
  Remove Writable wire protocol related code that is no longer necessary
  

• HDFS-2880. Major sub-task reported by Suresh Srinivas and fixed by Suresh Srinivas (data-node , name-node)
  Protocol buffer changes to add support multiple storages
  

• HDFS-2878. Blocker bug reported by Eli Collins and fixed by Todd Lipcon (test)
  TestBlockRecovery does not compile
  

• HDFS-2815. Critical bug reported by Uma Maheswara Rao G and fixed by Uma Maheswara Rao G (name-node)
  Namenode is not coming out of safemode when we perform ( NN crash + restart ) . Also FSCK report shows blocks missed.
  

• HDFS-2801. Major sub-task reported by Jitendra Nath Pandey and fixed by Jitendra Nath Pandey
  Provide a method in client side translators to check for a methods supported in underlying protocol.
  

• HDFS-2799. Major bug reported by Eli Collins and fixed by amith (name-node)
  Trim fs.checkpoint.dir values
  

• HDFS-2768. Major bug reported by Uma Maheswara Rao G and fixed by Uma Maheswara Rao G (name-node)
  BackupNode stop can not close proxy connections because it is not a proxy instance.
  

• HDFS-2765. Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (test)
  TestNameEditsConfigs is incorrectly swallowing IOE
  

• HDFS-2739. Critical bug reported by Sho Shimauchi and fixed by Jitendra Nath Pandey
  SecondaryNameNode doesn't start up
  

• HDFS-2731. Major new feature reported by Aaron T. Myers and fixed by Todd Lipcon (ha)
  HA: Autopopulate standby name dirs if they're empty
  

  The HA NameNode may now be started with the "-bootstrapStandby" flag. This causes it to copy the namespace information and most recent checkpoint from its HA pair, and save it to local storage, allowing an HA setup to be bootstrapped without use of rsync or external tools.

• HDFS-2708. Minor improvement reported by Eli Collins and fixed by Aaron T. Myers (data-node , name-node)
  Stats for the # of blocks per DN
  

• HDFS-2700. Major bug reported by Uma Maheswara Rao G and fixed by Uma Maheswara Rao G
  TestDataNodeMultipleRegistrations is failing in trunk
  

• HDFS-2697. Major sub-task reported by Suresh Srinivas and fixed by Jitendra Nath Pandey
  Move RefreshAuthPolicy, RefreshUserMappings, GetUserMappings protocol to protocol buffers
  

• HDFS-2696. Major bug reported by Petru Dimulescu and fixed by Bruno Mahé (build , fuse-dfs)
  Fix the fuse-fds build
  

• HDFS-2694. Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (name-node)
  Removal of Avro broke non-PB NN services
  

• HDFS-2687. Major sub-task reported by Uma Maheswara Rao G and fixed by Suresh Srinivas (test)
  Tests are failing with ClassCastException, due to new protocol changes
  

• HDFS-2676. Major bug reported by Suresh Srinivas and fixed by Suresh Srinivas
  Remove Avro RPC
  

• HDFS-2669. Major sub-task reported by Sanjay Radia and fixed by Sanjay Radia
  Enable protobuf rpc for ClientNamenodeProtocol
  

• HDFS-2666. Major sub-task reported by Suresh Srinivas and fixed by Suresh Srinivas (test)
  TestBackupNode fails
  

• HDFS-2663. Major sub-task reported by Suresh Srinivas and fixed by Suresh Srinivas
  Optional parameters are not handled correctly
  

• HDFS-2661. Major sub-task reported by Jitendra Nath Pandey and fixed by Jitendra Nath Pandey
  Enable protobuf RPC for DatanodeProtocol
  

• HDFS-2651. Major sub-task reported by Sanjay Radia and fixed by Sanjay Radia
  ClientNameNodeProtocol Translators for Protocol Buffers
  

• HDFS-2650. Minor improvement reported by Hari Mankude and fixed by Hari Mankude
  Replace @inheritDoc with @Override
  

• HDFS-2647. Major sub-task reported by Suresh Srinivas and fixed by Suresh Srinivas (balancer , data-node , hdfs client , name-node)
  Enable protobuf RPC for InterDatanodeProtocol, ClientDatanodeProtocol, JournalProtocol and NamenodeProtocol
  

• HDFS-2642. Major sub-task reported by Jitendra Nath Pandey and fixed by Jitendra Nath Pandey
  Protobuf translators for DatanodeProtocol
  

• HDFS-2636. Major sub-task reported by Suresh Srinivas and fixed by Suresh Srinivas
  Implement protobuf service for ClientDatanodeProtocol
  

• HDFS-2629. Major sub-task reported by Suresh Srinivas and fixed by Suresh Srinivas (data-node)
  Implement protobuf service for InterDatanodeProtocol
  

• HDFS-2618. Major sub-task reported by Suresh Srinivas and fixed by Suresh Srinivas (name-node)
  Implement protobuf service for NamenodeProtocol
  

• HDFS-2597. Major sub-task reported by Sanjay Radia and fixed by Sanjay Radia
  ClientNameNodeProtocol in Protocol Buffers
  

• HDFS-2581. Major sub-task reported by Suresh Srinivas and fixed by Suresh Srinivas (name-node)
  Implement protobuf service for JournalProtocol
  

• HDFS-2532. Critical bug reported by Todd Lipcon and fixed by Uma Maheswara Rao G (test)
  TestDfsOverAvroRpc timing out in trunk
  

• HDFS-2526. Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (hdfs client , name-node)
  (Client)NamenodeProtocolTranslatorR23 do not need to keep a reference to rpcProxyWithoutRetry
  

• HDFS-2520. Major sub-task reported by Suresh Srinivas and fixed by Suresh Srinivas (data-node)
  Protobuf - Add protobuf service for InterDatanodeProtocol
  

• HDFS-2519. Major sub-task reported by Suresh Srinivas and fixed by Suresh Srinivas (data-node , name-node)
  Protobuf - Add protobuf service for DatanodeProtocol
  

• HDFS-2518. Major sub-task reported by Suresh Srinivas and fixed by Suresh Srinivas (name-node)
  Protobuf - Add protobuf service for NamenodeProtocol
  

• HDFS-2517. Major sub-task reported by Suresh Srinivas and fixed by Suresh Srinivas (name-node)
  Protobuf - Add protocol service for JournalProtocol
  

• HDFS-2507. Major improvement reported by Todd Lipcon and fixed by Todd Lipcon (name-node)
  HA: Allow saveNamespace operations to be canceled
  

• HDFS-2505. Minor test reported by Ravi Prakash and fixed by Ravi Prakash (test)
  Add a test to verify getFileChecksum works with ViewFS
  

• HDFS-2499. Major sub-task reported by Suresh Srinivas and fixed by Suresh Srinivas (name-node)
  Fix RPC client creation bug from HDFS-2459
  

• HDFS-2497. Major bug reported by Suresh Srinivas and fixed by Suresh Srinivas
  Fix TestBackupNode failure
  

• HDFS-2496. Major improvement reported by Suresh Srinivas and fixed by Suresh Srinivas
  Separate datatypes for DatanodeProtocol
  

• HDFS-2495. Major sub-task reported by Tomasz Nykiel and fixed by Tomasz Nykiel (name-node)
  Increase granularity of write operations in ReplicationMonitor thus reducing contention for write lock
  

• HDFS-2489. Major sub-task reported by Suresh Srinivas and fixed by Suresh Srinivas
  Move commands Finalize and Register out of DatanodeCommand class.
  

• HDFS-2488. Major sub-task reported by Suresh Srinivas and fixed by Suresh Srinivas (data-node)
  Separate datatypes for InterDatanodeProtocol
  

• HDFS-2481. Major bug reported by Tsz Wo (Nicholas), SZE and fixed by Sanjay Radia
  Unknown protocol: org.apache.hadoop.hdfs.protocol.ClientProtocol
  

• HDFS-2480. Major sub-task reported by Suresh Srinivas and fixed by Suresh Srinivas
  Separate datatypes for NamenodeProtocol
  

• HDFS-2479. Major sub-task reported by Sanjay Radia and fixed by Sanjay Radia
  HDFS Client Data Types in Protocol Buffers
  

• HDFS-2477. Major sub-task reported by Tomasz Nykiel and fixed by Tomasz Nykiel (name-node)
  Optimize computing the diff between a block report and the namenode state.
  

• HDFS-2476. Major sub-task reported by Tomasz Nykiel and fixed by Tomasz Nykiel (name-node)
  More CPU efficient data structure for under-replicated/over-replicated/invalidate blocks
  

• HDFS-2459. Major sub-task reported by Suresh Srinivas and fixed by Suresh Srinivas
  Separate datatypes for Journal protocol
  

• HDFS-2430. Major new feature reported by Aaron T. Myers and fixed by Aaron T. Myers (name-node)
  The number of failed or low-resource volumes the NN can tolerate should be configurable
  

• HDFS-2413. Major improvement reported by Todd Lipcon and fixed by Harsh J (hdfs client)
  Add public APIs for safemode
  

• HDFS-2410. Minor improvement reported by Suresh Srinivas and fixed by Suresh Srinivas (data-node , name-node , test)
  Further clean up hard-coded configuration keys
  

• HDFS-2351. Major improvement reported by Sanjay Radia and fixed by Sanjay Radia
  Change Namenode and Datanode to register each of their protocols seperately
  

• HDFS-2337. Major improvement reported by Aaron T. Myers and fixed by Aaron T. Myers (hdfs client)
  DFSClient shouldn't keep multiple RPC proxy references
  

• HDFS-2334. Major sub-task reported by Ivan Kelly and fixed by Ivan Kelly (name-node)
  Add Closeable to JournalManager
  

• HDFS-2303. Major improvement reported by Roman Shaposhnik and fixed by Mingjie Lai (build , scripts)
  Unbundle jsvc
  

  To run secure Datanodes users must install jsvc for their platform and set JSVC_HOME to point to the location of jsvc in their environment.

• HDFS-2223. Major improvement reported by Todd Lipcon and fixed by Todd Lipcon (name-node)
  Untangle depencencies between NN components
  

• HDFS-2181. Major sub-task reported by Sanjay Radia and fixed by Sanjay Radia
  Separate HDFS Client wire protocol data types
  

• HDFS-2158. Major sub-task reported by Jitendra Nath Pandey and fixed by Jitendra Nath Pandey
  Add JournalSet to manage the set of journals.
  

• HDFS-2038. Critical test reported by Daryn Sharp and fixed by Kihwal Lee (test)
  Update test to handle relative paths with globs
  

• HDFS-2018. Major sub-task reported by Ivan Kelly and fixed by Ivan Kelly
  1073: Move all journal stream management code into one place
  

• HDFS-1765. Major bug reported by Hairong Kuang and fixed by Uma Maheswara Rao G (name-node)
  Block Replication should respect under-replication block priority
  

• HDFS-1623. Major new feature reported by Sanjay Radia and fixed by
  High Availability Framework for HDFS NN
  

• HDFS-1580. Major improvement reported by Ivan Kelly and fixed by Jitendra Nath Pandey (name-node)
  Add interface for generic Write Ahead Logging mechanisms
  

• HDFS-891. Minor bug reported by Steve Loughran and fixed by Harsh J (data-node)
  DataNode no longer needs to check for dfs.network.script
  

• HDFS-860. Minor wish reported by Brian Bockelman and fixed by Brian Bockelman (fuse-dfs)
  fuse-dfs truncate behavior causes issues with scp
  

• HDFS-395. Major sub-task reported by dhruba borthakur and fixed by Tomasz Nykiel (data-node , name-node)
  DFS Scalability: Incremental block reports
  

• HDFS-309. Major improvement reported by Todd Lipcon and fixed by Sho Shimauchi
  FSEditLog should log progress during replay
  

• HDFS-234. Major new feature reported by Luca Telloli and fixed by Ivan Kelly
  Integration with BookKeeper logging system
  

• HDFS-208. Minor improvement reported by Allen Wittenauer and fixed by Uma Maheswara Rao G (name-node)
  name node should warn if only one dir is listed in dfs.name.dir
  

• HADOOP-8619. Major improvement reported by Radim Kolar and fixed by Chris Douglas (io)
  WritableComparator must implement no-arg constructor
  

• HADOOP-8398. Minor improvement reported by Eli Collins and fixed by Eli Collins
  Cleanup BlockLocation
  

• HADOOP-8388. Minor improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  Remove unused BlockLocation serialization
  

• HADOOP-8366. Blocker improvement reported by Sanjay Radia and fixed by Sanjay Radia
  Use ProtoBuf for RpcResponseHeader
  

• HADOOP-8359. Trivial task reported by Harsh J and fixed by Anupam Seth (conf)
  Clear up javadoc warnings in hadoop-common-project
  

• HADOOP-8356. Major improvement reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (fs)
  FileSystem service loading mechanism should print the FileSystem impl it is failing to load
  

• HADOOP-8355. Minor bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  SPNEGO filter throws/logs exception when authentication fails
  

• HADOOP-8353. Major improvement reported by Roman Shaposhnik and fixed by Roman Shaposhnik (scripts)
  hadoop-daemon.sh and yarn-daemon.sh can be misleading on stop
  

• HADOOP-8350. Major improvement reported by Todd Lipcon and fixed by Todd Lipcon (util)
  Improve NetUtils.getInputStream to return a stream which has a tunable timeout
  

• HADOOP-8349. Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (viewfs)
  ViewFS doesn't work when the root of a file system is mounted
  

• HADOOP-8347. Major bug reported by Philip Zeyliger and fixed by Philip Zeyliger (security)
  Hadoop Common logs misspell 'successful'
  

• HADOOP-8343. Major new feature reported by Philip Zeyliger and fixed by Alejandro Abdelnur (util)
  Allow configuration of authorization for JmxJsonServlet and MetricsServlet
  

• HADOOP-8314. Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  HttpServer#hasAdminAccess should return false if authorization is enabled but user is not authenticated
  

• HADOOP-8310. Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (fs)
  FileContext#checkPath should handle URIs with no port
  

• HADOOP-8309. Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  Pseudo & Kerberos AuthenticationHandler should use getType() to create token
  

• HADOOP-8296. Minor bug reported by Thomas Graves and fixed by Devaraj K
  hadoop/yarn daemonlog usage wrong
  

• HADOOP-8285. Major improvement reported by Sanjay Radia and fixed by Sanjay Radia (ipc)
  Use ProtoBuf for RpcPayLoadHeader
  

• HADOOP-8282. Minor bug reported by Devaraj K and fixed by Devaraj K (scripts)
  start-all.sh refers incorrectly start-dfs.sh existence for starting start-yarn.sh
  

• HADOOP-8280. Major improvement reported by Ahmed Radwan and fixed by Ahmed Radwan (test , util)
  Move VersionUtil/TestVersionUtil and GenericTestUtils from HDFS into Common.
  

• HADOOP-8275. Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  Range check DelegationKey length
  

• HADOOP-8270. Minor bug reported by Roman Shaposhnik and fixed by Roman Shaposhnik (scripts)
  hadoop-daemon.sh stop action should return 0 for an already stopped service
  

  The daemon stop action no longer returns failure when stopping an already stopped service.

• HADOOP-8264. Trivial bug reported by Bernd Fondermann and fixed by Bernd Fondermann
  Remove irritating double double quotes in front of hostname
  

• HADOOP-8263. Minor bug reported by Todd Lipcon and fixed by Todd Lipcon (ipc)
  Stringification of IPC calls not useful
  

• HADOOP-8261. Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (fs)
  Har file system doesn't deal with FS URIs with a host but no port
  

• HADOOP-8251. Blocker bug reported by Todd Lipcon and fixed by Todd Lipcon (security)
  SecurityUtil.fetchServiceTicket broken after HADOOP-6941
  

• HADOOP-8243. Critical bug reported by Todd Lipcon and fixed by Todd Lipcon (ha , security)
  Security support broken in CLI (manual) failover controller
  

• HADOOP-8238. Major bug reported by Eli Collins and fixed by Eli Collins
  NetUtils#getHostNameOfIP blows up if given ip:port string w/o port
  

• HADOOP-8236. Major improvement reported by Philip Zeyliger and fixed by Todd Lipcon (ha)
  haadmin should have configurable timeouts for failover commands
  

• HADOOP-8218. Critical bug reported by Todd Lipcon and fixed by Todd Lipcon (ipc , test)
  RPC.closeProxy shouldn't throw error when closing a mock
  

• HADOOP-8214. Major improvement reported by Roman Shaposhnik and fixed by Roman Shaposhnik (scripts)
  make hadoop script recognize a full set of deprecated commands
  

• HADOOP-8211. Major sub-task reported by Eli Collins and fixed by Eli Collins (io , performance)
  Update commons-net version to 3.1
  

• HADOOP-8210. Major sub-task reported by Eli Collins and fixed by Eli Collins (io , performance)
  Common side of HDFS-3148
  

• HADOOP-8206. Major new feature reported by Todd Lipcon and fixed by Todd Lipcon (ha)
  Common portion of ZK-based failover controller
  

• HADOOP-8204. Major bug reported by Tom White and fixed by Todd Lipcon
  TestHealthMonitor fails occasionally
  

• HADOOP-8202. Minor bug reported by Hari Mankude and fixed by Hari Mankude (ipc)
  stopproxy() is not closing the proxies correctly
  

• HADOOP-8200. Minor improvement reported by Eli Collins and fixed by Eli Collins (conf)
  Remove HADOOP_[JOBTRACKER|TASKTRACKER]_OPTS
  

• HADOOP-8199. Major bug reported by Nishan Shetty and fixed by Devaraj K
  Fix issues in start-all.sh and stop-all.sh
  

• HADOOP-8193. Major improvement reported by Todd Lipcon and fixed by Todd Lipcon (ha)
  Refactor FailoverController/HAAdmin code to add an abstract class for "target" services
  

• HADOOP-8191. Major bug reported by Philip Zeyliger and fixed by Todd Lipcon (ha)
  SshFenceByTcpPort uses netcat incorrectly
  

• HADOOP-8189. Major bug reported by Jonathan Natkins and fixed by Jonathan Natkins (security)
  LdapGroupsMapping shouldn't throw away IOException
  

• HADOOP-8185. Major improvement reported by Arpit Gupta and fixed by Arpit Gupta (documentation)
  Update namenode -format documentation and add -nonInteractive and -force
  

• HADOOP-8184. Major improvement reported by Sanjay Radia and fixed by Sanjay Radia (ipc)
  ProtoBuf RPC engine does not need it own reply packet - it can use the IPC layer reply packet.
  

  This change will affect the output of errors for some Hadoop CLI commands. Specifically, the name of the exception class will no longer appear, and instead only the text of the exception message will appear.

• HADOOP-8183. Minor improvement reported by Harsh J and fixed by Harsh J (util)
  Stop using "mapred.used.genericoptionsparser" to avoid unnecessary warnings
  

• HADOOP-8169. Critical bug reported by Thomas Graves and fixed by Thomas Graves (build)
  javadoc generation fails with java.lang.OutOfMemoryError: Java heap space
  

• HADOOP-8164. Major sub-task reported by Suresh Srinivas and fixed by Daryn Sharp (fs)
  Handle paths using back slash as path separator for windows only
  

  This jira only allows providing paths using back slash as separator on Windows. The back slash on *nix system will be used as escape character. The support for paths using back slash as path separator will be removed in HADOOP-8139 in release 23.3.

• HADOOP-8163. Major improvement reported by Todd Lipcon and fixed by Todd Lipcon (ha)
  Improve ActiveStandbyElector to provide hooks for fencing old active
  

• HADOOP-8159. Major bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
  NetworkTopology: getLeaf should check for invalid topologies
  

• HADOOP-8154. Major bug reported by Eli Collins and fixed by Eli Collins (conf)
  DNS#getIPs shouldn't silently return the local host IP for bogus interface names
  

• HADOOP-8152. Major improvement reported by Aaron T. Myers and fixed by Aaron T. Myers (security)
  Expand public APIs for security library classes
  

• HADOOP-8149. Major improvement reported by Patrick Hunt and fixed by Patrick Hunt (conf)
  cap space usage of default log4j rolling policy
  

  Hadoop log files are now rolled by size instead of date (daily) by default. Tools that depend on the log file name format will need to be updated. Users who would like to maintain the previous settings of hadoop.root.logger and hadoop.security.logger can use their current log4j.properties files and update the HADOOP_ROOT_LOGGER and HADOOP_SECURITY_LOGGER environment variables to use DRFA and DRFAS respectively.

• HADOOP-8142. Major task reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (build)
  Update versions from 0.23.2 to 0.23.3
  

• HADOOP-8141. Trivial improvement reported by Todd Lipcon and fixed by Todd Lipcon (security)
  Add method to init krb5 cipher suites
  

• HADOOP-8121. Major new feature reported by Jonathan Natkins and fixed by Jonathan Natkins (security)
  Active Directory Group Mapping Service
  

• HADOOP-8119. Minor bug reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (test)
  Fix javac warnings in TestAuthenticationFilter
  

• HADOOP-8118. Minor improvement reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (metrics)
  Print the stack trace of InstanceAlreadyExistsException in trace level
  

• HADOOP-8117. Trivial improvement reported by Todd Lipcon and fixed by Todd Lipcon (build , test)
  Upgrade test build to Surefire 2.12
  

• HADOOP-8113. Trivial improvement reported by Eugene Koontz and fixed by Eugene Koontz (documentation)
  Correction to BUILDING.txt: HDFS needs ProtocolBuffer, too (not just MapReduce)
  

• HADOOP-8098. Major improvement reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  KerberosAuthenticatorHandler should use _HOST replacement to resolve principal name
  

• HADOOP-8086. Minor improvement reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
  KerberosName silently sets defaultRealm to "" if the Kerberos config is not found, it should log a WARN
  

• HADOOP-8084. Major improvement reported by Devaraj Das and fixed by Devaraj Das (ipc)
  Protobuf RPC engine can be optimized to not do copying for the RPC request/response
  

• HADOOP-8077. Major improvement reported by Todd Lipcon and fixed by Todd Lipcon (ha)
  HA: fencing method should be able to be configured on a per-NN or per-NS basis
  

• HADOOP-8070. Major improvement reported by Todd Lipcon and fixed by Todd Lipcon (benchmarks , ipc)
  Add standalone benchmark of protobuf IPC
  

• HADOOP-8007. Major improvement reported by Aaron T. Myers and fixed by Todd Lipcon (ha)
  HA: use substitution token for fencing argument
  

• HADOOP-7994. Major sub-task reported by Jitendra Nath Pandey and fixed by Jitendra Nath Pandey
  Remove getProtocolVersion and getProtocolSignature from the client side translator and server side implementation
  

• HADOOP-7968. Minor bug reported by Todd Lipcon and fixed by Sho Shimauchi (ipc)
  Errant println left in RPC.getHighestSupportedProtocol
  

• HADOOP-7965. Major sub-task reported by Jitendra Nath Pandey and fixed by Jitendra Nath Pandey (ipc)
  Support for protocol version and signature in PB
  

• HADOOP-7957. Major improvement reported by Jitendra Nath Pandey and fixed by Jitendra Nath Pandey
  Classes deriving GetGroupsBase should be able to override proxy creation.
  

• HADOOP-7940. Major bug reported by Aaron, and fixed by Csaba Miklos (io)
  method clear() in org.apache.hadoop.io.Text does not work
  

• HADOOP-7931. Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (ipc)
  o.a.h.ipc.WritableRpcEngine should have a way to force initialization
  

• HADOOP-7920. Major bug reported by Suresh Srinivas and fixed by Suresh Srinivas (ipc)
  Remove Avro RPC
  

• HADOOP-7913. Major sub-task reported by Sanjay Radia and fixed by Sanjay Radia (ipc)
  Fix bug in ProtoBufRpcEngine -
  

• HADOOP-7900. Major bug reported by Ravi Gummadi and fixed by Ravi Gummadi (fs)
  LocalDirAllocator confChanged() accesses conf.get() twice
  

• HADOOP-7899. Major improvement reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (build)
  Generate proto java files as part of the build
  

• HADOOP-7897. Major bug reported by Suresh Srinivas and fixed by Suresh Srinivas (ipc)
  ProtobufRPCEngine client side exception mechanism is not consistent with WritableRpcEngine
  

• HADOOP-7892. Trivial bug reported by Todd Lipcon and fixed by Todd Lipcon (ipc)
  IPC logs too verbose after "RpcKind" introduction
  

• HADOOP-7888. Major bug reported by Jason Lowe and fixed by Jason Lowe (test)
  TestFailoverProxy fails intermittently on trunk
  

• HADOOP-7876. Major new feature reported by Suresh Srinivas and fixed by Suresh Srinivas (ipc)
  Allow access to BlockKey/DelegationKey endoded key for RPC over protobuf
  

• HADOOP-7875. Major improvement reported by Suresh Srinivas and fixed by Suresh Srinivas (ipc)
  Add helper class to unwrap RemoteException from ServiceException thrown on protobuf based RPC
  

• HADOOP-7862. Major sub-task reported by Sanjay Radia and fixed by Sanjay Radia (ipc)
  Move the support for multiple protocols to lower layer so that Writable, PB and Avro can all use it
  

• HADOOP-7833. Major bug reported by John Lee and fixed by John Lee (ipc)
  Inner classes of org.apache.hadoop.ipc.protobuf.HadoopRpcProtos generates findbugs warnings which results in -1 for findbugs
  

• HADOOP-7806. Major new feature reported by Harsh J and fixed by Harsh J (util)
  Support binding to sub-interfaces
  

• HADOOP-7788. Major new feature reported by Todd Lipcon and fixed by Todd Lipcon (ha)
  HA: Simple HealthMonitor class to watch an HAService
  

• HADOOP-7776. Major sub-task reported by Sanjay Radia and fixed by Sanjay Radia (ipc)
  Make the Ipc-Header in a RPC-Payload an explicit header
  

• HADOOP-7773. Major sub-task reported by Suresh Srinivas and fixed by Suresh Srinivas (ipc)
  Add support for protocol buffer based RPC engine
  

• HADOOP-7729. Major improvement reported by Todd Lipcon and fixed by Todd Lipcon (ipc)
  Send back valid HTTP response if user hits IPC port with HTTP GET
  

• HADOOP-7717. Major improvement reported by Aaron T. Myers and fixed by Aaron T. Myers (ipc)
  Move handling of concurrent client fail-overs to RetryInvocationHandler
  

• HADOOP-7716. Minor improvement reported by Sanjay Radia and fixed by Sanjay Radia
  RPC protocol registration on SS does not log the protocol name (only the class which may be different)
  

• HADOOP-7695. Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (ipc)
  RPC.stopProxy can throw unintended exception while logging error
  

• HADOOP-7693. Major improvement reported by Doug Cutting and fixed by Doug Cutting (ipc)
  fix RPC.Server#addProtocol to work in AvroRpcEngine
  

• HADOOP-7687. Minor improvement reported by Sanjay Radia and fixed by Sanjay Radia
  Make getProtocolSignature public
  

• HADOOP-7635. Major improvement reported by Aaron T. Myers and fixed by Aaron T. Myers (ipc)
  RetryInvocationHandler should release underlying resources on close
  

• HADOOP-7621. Critical bug reported by Alejandro Abdelnur and fixed by Aaron T. Myers (security)
  alfredo config should be in a file not readable by users
  

• HADOOP-7607. Major improvement reported by Aaron T. Myers and fixed by Aaron T. Myers (ipc)
  Simplify the RPC proxy cleanup process
  

• HADOOP-7557. Major sub-task reported by Sanjay Radia and fixed by Sanjay Radia
  Make IPC header be extensible
  

• HADOOP-7549. Major improvement reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (fs)
  Use JDK ServiceLoader mechanism to find FileSystem implementations
  

• HADOOP-7524. Major sub-task reported by Sanjay Radia and fixed by Sanjay Radia (ipc)
  Change RPC to allow multiple protocols including multiple versions of the same protocol
  

• HADOOP-7454. Major new feature reported by Aaron T. Myers and fixed by
  Common side of High Availability Framework (HDFS-1623)
  

• HADOOP-7358. Minor improvement reported by Todd Lipcon and fixed by Todd Lipcon (ipc)
  Improve log levels when exceptions caught in RPC handler
  

• HADOOP-7350. Major improvement reported by Tom White and fixed by Tom White (conf , io)
  Use ServiceLoader to discover compression codec classes
  

• HADOOP-7069. Major improvement reported by Jakob Homan and fixed by (documentation)
  Replace forrest with supported framework
  

• HADOOP-7030. Major new feature reported by Patrick Angeles and fixed by Tom White
  Add TableMapping topology implementation to read host to rack mapping from a file
  

• HADOOP-6941. Major bug reported by Stephen Watt and fixed by Devaraj Das
  Support non-SUN JREs in UserGroupInformation
  

• HADOOP-6924. Major bug reported by Stephen Watt and fixed by Devaraj Das
  Build fails with non-Sun JREs due to different pathing to the operating system architecture shared libraries