Hadoop 0.23.9 Release Notes
These release notes include new developer and user-facing incompatibilities, features, and major improvements.
Changes since Hadoop 0.23.8
- YARN-742.
Major bug reported by Kihwal Lee and fixed by Jason Lowe (nodemanager)
Log aggregation causes a lot of redundant setPermission calls
In one of our clusters, namenode RPC is spending 45% of its time on serving setPermission calls. Further investigation has revealed that most calls are redundantly made on /mapred/logs/<user>/logs. Also mkdirs calls are made before this.
- YARN-478.
Major sub-task reported by Aleksey Gorshkov and fixed by Aleksey Gorshkov
fix coverage org.apache.hadoop.yarn.webapp.log
fix coverage org.apache.hadoop.yarn.webapp.log
one patch for trunk, branch-2, branch-0.23
- YARN-427.
Major sub-task reported by Aleksey Gorshkov and fixed by Aleksey Gorshkov
Coverage fix for org.apache.hadoop.yarn.server.api.*
Coverage fix for org.apache.hadoop.yarn.server.api.*
patch YARN-427-trunk.patch for trunk
patch YARN-427-branch-2.patch for branch-2 and branch-0.23
- YARN-169.
Minor improvement reported by Anthony Rojas and fixed by Anthony Rojas (nodemanager)
Update log4j.appender.EventCounter to use org.apache.hadoop.log.metrics.EventCounter
We should update the log4j.appender.EventCounter in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/container-log4j.properties to use *org.apache.hadoop.log.metrics.EventCounter* rather than *org.apache.hadoop.metrics.jvm.EventCounter* to avoid triggering the following warning:
{code}WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files{code}
- MAPREDUCE-5316.
Major bug reported by Ashwin Shankar and fixed by Ashwin Shankar (client)
job -list-attempt-ids command does not handle illegal task-state
- MAPREDUCE-5315.
Critical bug reported by Mithun Radhakrishnan and fixed by Mithun Radhakrishnan (distcp)
DistCp reports success even on failure.
- MAPREDUCE-5308.
Major bug reported by Nathan Roberts and fixed by Nathan Roberts
Shuffling to memory can get out-of-sync when fetching multiple compressed map outputs
- MAPREDUCE-5268.
Major improvement reported by Jason Lowe and fixed by Karthik Kambatla (jobhistoryserver)
Improve history server startup performance
- MAPREDUCE-5084.
Major test reported by Aleksey Gorshkov and fixed by Aleksey Gorshkov
fix coverage org.apache.hadoop.mapreduce.v2.app.webapp and org.apache.hadoop.mapreduce.v2.hs.webapp
- MAPREDUCE-4019.
Minor bug reported by B Anil Kumar and fixed by Ashwin Shankar (client)
-list-attempt-ids is not working
- HDFS-4878.
Major bug reported by Tao Luo and fixed by Tao Luo (namenode)
On Remove Block, Block is not Removed from neededReplications queue
- HDFS-4867.
Major bug reported by Kihwal Lee and fixed by Plamen Jeliazkov (namenode)
metaSave NPEs when there are invalid blocks in repl queue.
- HDFS-4862.
Major bug reported by Ravi Prakash and fixed by Ravi Prakash
SafeModeInfo.isManual() returns true when resources are low even if it wasn't entered into manually
- HDFS-4832.
Critical bug reported by Ravi Prakash and fixed by Ravi Prakash
Namenode doesn't change the number of missing blocks in safemode when DNs rejoin or leave
This change makes name node keep its internal replication queues and data node state updated in manual safe mode. This allows metrics and UI to present up-to-date information while in safe mode. The behavior during start-up safe mode is unchanged.
- HDFS-4765.
Major bug reported by Andrew Wang and fixed by Andrew Wang (namenode)
Permission check of symlink deletion incorrectly throws UnresolvedLinkException
- HDFS-4205.
Major bug reported by Andy Isaacson and fixed by Jason Lowe (hdfs-client)
fsck fails with symlinks
- HADOOP-9582.
Major bug reported by Ashwin Shankar and fixed by Ashwin Shankar (conf)
Non-existent file to "hadoop fs -conf" doesn't throw error
- HADOOP-9581.
Major bug reported by Ashwin Shankar and fixed by Ashwin Shankar (scripts)
hadoop --config non-existent directory should result in error