CHANGES.txt 9.6 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228
  1. Hadoop Change Log
  2. Trunk (unreleased)
  3. 1. HADOOP-199. Fix reduce progress (broken by HADOOP-182).
  4. (omalley via cutting)
  5. 2. HADOOP-201. Fix 'bin/hadoop dfs -report'. (cutting)
  6. 3. HADOOP-207. Fix JDK 1.4 incompatibility introduced by HADOOP-96.
  7. System.getenv() does not work in JDK 1.4. (Hairong Kuang via cutting)
  8. Release 0.2.0 - 2006-05-05
  9. 1. Fix HADOOP-126. 'bin/hadoop dfs -cp' now correctly copies .crc
  10. files. (Konstantin Shvachko via cutting)
  11. 2. Fix HADOOP-51. Change DFS to support per-file replication counts.
  12. (Konstantin Shvachko via cutting)
  13. 3. Fix HADOOP-131. Add scripts to start/stop dfs and mapred daemons.
  14. Use these in start/stop-all scripts. (Chris Mattmann via cutting)
  15. 4. Stop using ssh options by default that are not yet in widely used
  16. versions of ssh. Folks can still enable their use by uncommenting
  17. a line in conf/hadoop-env.sh. (cutting)
  18. 5. Fix HADOOP-92. Show information about all attempts to run each
  19. task in the web ui. (Mahadev konar via cutting)
  20. 6. Fix HADOOP-128. Improved DFS error handling. (Owen O'Malley via cutting)
  21. 7. Fix HADOOP-129. Replace uses of java.io.File with new class named
  22. Path. This fixes bugs where java.io.File methods were called
  23. directly when FileSystem methods were desired, and reduces the
  24. likelihood of such bugs in the future. It also makes the handling
  25. of pathnames more consistent between local and dfs FileSystems and
  26. between Windows and Unix. java.io.File-based methods are still
  27. available for back-compatibility, but are deprecated and will be
  28. removed once 0.2 is released. (cutting)
  29. 8. Change dfs.data.dir and mapred.local.dir to be comma-separated
  30. lists of directories, no longer be space-separated. This fixes
  31. several bugs on Windows. (cutting)
  32. 9. Fix HADOOP-144. Use mapred task id for dfs client id, to
  33. facilitate debugging. (omalley via cutting)
  34. 10. Fix HADOOP-143. Do not line-wrap stack-traces in web ui.
  35. (omalley via cutting)
  36. 11. Fix HADOOP-118. In DFS, improve clean up of abandoned file
  37. creations. (omalley via cutting)
  38. 12. Fix HADOOP-138. Stop multiple tasks in a single heartbeat, rather
  39. than one per heartbeat. (Stefan via cutting)
  40. 13. Fix HADOOP-139. Remove a potential deadlock in
  41. LocalFileSystem.lock(). (Igor Bolotin via cutting)
  42. 14. Fix HADOOP-134. Don't hang jobs when the tasktracker is
  43. misconfigured to use an un-writable local directory. (omalley via cutting)
  44. 15. Fix HADOOP-115. Correct an error message. (Stack via cutting)
  45. 16. Fix HADOOP-133. Retry pings from child to parent, in case of
  46. (local) communcation problems. Also log exit status, so that one
  47. can distinguish patricide from other deaths. (omalley via cutting)
  48. 17. Fix HADOOP-142. Avoid re-running a task on a host where it has
  49. previously failed. (omalley via cutting)
  50. 18. Fix HADOOP-148. Maintain a task failure count for each
  51. tasktracker and display it in the web ui. (omalley via cutting)
  52. 19. Fix HADOOP-151. Close a potential socket leak, where new IPC
  53. connection pools were created per configuration instance that RPCs
  54. use. Now a global RPC connection pool is used again, as
  55. originally intended. (cutting)
  56. 20. Fix HADOOP-69. Don't throw a NullPointerException when getting
  57. hints for non-existing file split. (Bryan Pendelton via cutting)
  58. 21. Fix HADOOP-157. When a task that writes dfs files (e.g., a reduce
  59. task) failed and was retried, it would fail again and again,
  60. eventually failing the job. The problem was that dfs did not yet
  61. know that the failed task had abandoned the files, and would not
  62. yet let another task create files with the same names. Dfs now
  63. retries when creating a file long enough for locks on abandoned
  64. files to expire. (omalley via cutting)
  65. 22. Fix HADOOP-150. Improved task names that include job
  66. names. (omalley via cutting)
  67. 23. Fix HADOOP-162. Fix ConcurrentModificationException when
  68. releasing file locks. (omalley via cutting)
  69. 24. Fix HADOOP-132. Initial check-in of new Metrics API, including
  70. implementations for writing metric data to a file and for sending
  71. it to Ganglia. (David Bowen via cutting)
  72. 25. Fix HADOOP-160. Remove some uneeded synchronization around
  73. time-consuming operations in the TaskTracker. (omalley via cutting)
  74. 26. Fix HADOOP-166. RPCs failed when passed subclasses of a declared
  75. parameter type. This is fixed by changing ObjectWritable to store
  76. both the declared type and the instance type for Writables. Note
  77. that this incompatibly changes the format of ObjectWritable and
  78. will render unreadable any ObjectWritables stored in files.
  79. Nutch only uses ObjectWritable in intermediate files, so this
  80. should not be a problem for Nutch. (Stefan & cutting)
  81. 27. Fix HADOOP-168. MapReduce RPC protocol methods should all declare
  82. IOException, so that timeouts are handled appropriately.
  83. (omalley via cutting)
  84. 28. Fix HADOOP-169. Don't fail a reduce task if a call to the
  85. jobtracker to locate map outputs fails. (omalley via cutting)
  86. 29. Fix HADOOP-170. Permit FileSystem clients to examine and modify
  87. the replication count of individual files. Also fix a few
  88. replication-related bugs. (Konstantin Shvachko via cutting)
  89. 30. Permit specification of a higher replication levels for job
  90. submission files (job.xml and job.jar). This helps with large
  91. clusters, since these files are read by every node. (cutting)
  92. 31. HADOOP-173. Optimize allocation of tasks with local data. (cutting)
  93. 32. HADOOP-167. Reduce number of Configurations and JobConf's
  94. created. (omalley via cutting)
  95. 33. NUTCH-256. Change FileSystem#createNewFile() to create a .crc
  96. file. The lack of a .crc file was causing warnings. (cutting)
  97. 34. HADOOP-174. Change JobClient to not abort job until it has failed
  98. to contact the job tracker for five attempts, not just one as
  99. before. (omalley via cutting)
  100. 35. HADOOP-177. Change MapReduce web interface to page through tasks.
  101. Previously, when jobs had more than a few thousand tasks they
  102. could crash web browsers. (Mahadev Konar via cutting)
  103. 36. HADOOP-178. In DFS, piggyback blockwork requests from datanodes
  104. on heartbeat responses from namenode. This reduces the volume of
  105. RPC traffic. Also move startup delay in blockwork from datanode
  106. to namenode. This fixes a problem where restarting the namenode
  107. triggered a lot of uneeded replication. (Hairong Kuang via cutting)
  108. 37. HADOOP-183. If the DFS namenode is restarted with different
  109. minimum and/or maximum replication counts, existing files'
  110. replication counts are now automatically adjusted to be within the
  111. newly configured bounds. (Hairong Kuang via cutting)
  112. 38. HADOOP-186. Better error handling in TaskTracker's top-level
  113. loop. Also improve calculation of time to send next heartbeat.
  114. (omalley via cutting)
  115. 39. HADOOP-187. Add two MapReduce examples/benchmarks. One creates
  116. files containing random data. The second sorts the output of the
  117. first. (omalley via cutting)
  118. 40. HADOOP-185. Fix so that, when a task tracker times out making the
  119. RPC asking for a new task to run, the job tracker does not think
  120. that it is actually running the task returned. (omalley via cutting)
  121. 41. HADOOP-190. If a child process hangs after it has reported
  122. completion, its output should not be lost. (Stack via cutting)
  123. 42. HADOOP-184. Re-structure some test code to better support testing
  124. on a cluster. (Mahadev Konar via cutting)
  125. 43. HADOOP-191 Add streaming package, Hadoop's first contrib module.
  126. This permits folks to easily submit MapReduce jobs whose map and
  127. reduce functions are implemented by shell commands. Use
  128. 'bin/hadoop jar build/hadoop-streaming.jar' to get details.
  129. (Michel Tourn via cutting)
  130. 44. HADOOP-189. Fix MapReduce in standalone configuration to
  131. correctly handle job jar files that contain a lib directory with
  132. nested jar files. (cutting)
  133. 45. HADOOP-65. Initial version of record I/O framework that enables
  134. the specification of record types and generates marshalling code
  135. in both Java and C++. Generated Java code implements
  136. WritableComparable, but is not yet otherwise used by
  137. Hadoop. (Milind Bhandarkar via cutting)
  138. 46. HADOOP-193. Add a MapReduce-based FileSystem benchmark.
  139. (Konstantin Shvachko via cutting)
  140. 47. HADOOP-194. Add a MapReduce-based FileSystem checker. This reads
  141. every block in every file in the filesystem. (Konstantin Shvachko
  142. via cutting)
  143. 48. HADOOP-182. Fix so that lost task trackers to not change the
  144. status of reduce tasks or completed jobs. Also fixes the progress
  145. meter so that failed tasks are subtracted. (omalley via cutting)
  146. 49. HADOOP-96. Logging improvements. Log files are now separate from
  147. standard output and standard error files. Logs are now rolled.
  148. Logging of all DFS state changes can be enabled, to facilitate
  149. debugging. (Hairong Kuang via cutting)
  150. Release 0.1.1 - 2006-04-08
  151. 1. Added CHANGES.txt, logging all significant changes to Hadoop. (cutting)
  152. 2. Fix MapReduceBase.close() to throw IOException, as declared in the
  153. Closeable interface. This permits subclasses which override this
  154. method to throw that exception. (cutting)
  155. 3. Fix HADOOP-117. Pathnames were mistakenly transposed in
  156. JobConf.getLocalFile() causing many mapred temporary files to not
  157. be removed. (Raghavendra Prabhu via cutting)
  158. 4. Fix HADOOP-116. Clean up job submission files when jobs complete.
  159. (cutting)
  160. 5. Fix HADOOP-125. Fix handling of absolute paths on Windows (cutting)
  161. Release 0.1.0 - 2006-04-01
  162. 1. The first release of Hadoop.