HADOOP-3759. Provides ability to run memory intensive jobs without affecting other running tasks on the nodes. Contributed by Hemanth Yamijala.

git-svn-id: https://svn.apache.org/repos/asf/hadoop/core/trunk@689064 13f79535-47bb-0310-9956-ffa450edef68

CHANGES.txt

@@ -97,6 +97,9 @@ Trunk (unreleased changes)
     HADOOP-3854. Add support for pluggable servlet filters in the HttpServers.
     (Tsz Wo (Nicholas) Sze via omalley)
 
+    HADOOP-3759. Provides ability to run memory intensive jobs without 
+    affecting other running tasks on the nodes. (Hemanth Yamijala via ddas)
+
   IMPROVEMENTS
 
     HADOOP-3732. Delay intialization of datanode block verification till

conf/hadoop-default.xml

@@ -1284,4 +1284,35 @@ creations/deletions), or "all".</description>
   </description>
 </property>
 
+<property>
+  <name>mapred.tasktracker.tasks.maxmemory</name>
+  <value>-1</value>
+  <description> The maximum amount of virtual memory all tasks running on a
+    tasktracker, including sub-processes they launch, can use. This value is
+    used to compute the amount of free memory available for tasks. Any task
+    scheduled on this tasktracker is guaranteed and constrained to use a 
+    share of this amount. Any task exceeding its share will be killed. 
+    If set to -1, this functionality is disabled, and mapred.task.maxmemory
+    is ignored.
+  </description>
+</property>
+
+<property>
+  <name>mapred.task.maxmemory</name>
+  <value>-1</value>
+  <description> The maximum amount of memory any task of a job will use.
+    A task of this job will be scheduled on a tasktracker, only if the 
+    amount of free memory on the tasktracker is greater than or 
+    equal to this value. If set to -1, tasks are assured a memory limit on
+    the tasktracker equal to 
+    mapred.tasktracker.tasks.maxmemory/number of slots. If the value of 
+    mapred.tasktracker.tasks.maxmemory is set to -1, this value is ignored.
+    
+    Note: If mapred.child.java.opts is specified with an Xmx value, or if 
+    mapred.child.ulimit is specified, then the value of mapred.task.maxmemory
+    must be set to a higher value than these. If not, the task might be 
+    killed even though these limits are not reached.
+  </description>  
+</property>
+
 </configuration>

docs/changes.html

@@ -36,7 +36,7 @@
     function collapse() {
       for (var i = 0; i < document.getElementsByTagName("ul").length; i++) {
         var list = document.getElementsByTagName("ul")[i];
-        if (list.id != 'trunk_(unreleased_changes)_' && list.id != 'release_0.18.0_-_unreleased_') {
+        if (list.id != 'trunk_(unreleased_changes)_' && list.id != 'release_0.18.1_-_unreleased_') {
           list.style.display = "none";
         }
       }
@@ -56,7 +56,7 @@
 </a></h2>
 <ul id="trunk_(unreleased_changes)_">
   <li><a href="javascript:toggleList('trunk_(unreleased_changes)_._incompatible_changes_')">  INCOMPATIBLE CHANGES
-</a>&nbsp;&nbsp;&nbsp;(5)
+</a>&nbsp;&nbsp;&nbsp;(10)
     <ol id="trunk_(unreleased_changes)_._incompatible_changes_">
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3595">HADOOP-3595</a>. Remove deprecated methods for mapred.combine.once
 functionality, which was necessary to providing backwards
@@ -76,10 +76,19 @@ hadoop.hdfs that reflect whether they are client, server, protocol,
 etc. DistributedFileSystem and DFSClient have moved and are now
 considered package private.<br />(Sanjay Radia via omalley)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-2325">HADOOP-2325</a>.  Require Java 6.<br />(cutting)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-372">HADOOP-372</a>.  Add support for multiple input paths with a different
+InputFormat and Mapper for each path.<br />(Chris Smith via tomwhite)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-1700">HADOOP-1700</a>.  Support appending to file in HDFS.<br />(dhruba)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3792">HADOOP-3792</a>. Make FsShell -test consistent with unix semantics, returning
+zero for true and non-zero for false.<br />(Ben Slusky via cdouglas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3664">HADOOP-3664</a>. Remove the deprecated method InputFormat.validateInput,
+which is no longer needed.<br />(tomwhite via omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3549">HADOOP-3549</a>. Give more meaningful errno's in libhdfs. In particular,
+EACCES is returned for permission problems.<br />(Ben Slusky via omalley)</li>
     </ol>
   </li>
   <li><a href="javascript:toggleList('trunk_(unreleased_changes)_._new_features_')">  NEW FEATURES
-</a>&nbsp;&nbsp;&nbsp;(2)
+</a>&nbsp;&nbsp;&nbsp;(14)
     <ol id="trunk_(unreleased_changes)_._new_features_">
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3341">HADOOP-3341</a>. Allow streaming jobs to specify the field separator for map
 and reduce input and output. The new configuration values are:
@@ -92,11 +101,40 @@ All of them default to "\t".<br />(Zheng Shao via omalley)</li>
 Hadoop. You can configure various parameters related to scheduling, such
 as queues and queue properties here. The properties for a queue follow a
 naming convention,such as, hadoop.rm.queue.queue-name.property-name.<br />(Hemanth Yamijala via ddas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3149">HADOOP-3149</a>. Adds a way in which map/reducetasks can create multiple
+outputs.<br />(Alejandro Abdelnur via ddas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3714">HADOOP-3714</a>.  Add a new contrib, bash-tab-completion, which enables
+bash tab completion for the bin/hadoop script. See the README file
+in the contrib directory for the installation.<br />(Chris Smith via enis)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3730">HADOOP-3730</a>. Adds a new JobConf constructor that disables loading
+default configurations.<br />(Alejandro Abdelnur via ddas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3772">HADOOP-3772</a>. Add a new Hadoop Instrumentation api for the JobTracker and
+the TaskTracker, refactor Hadoop Metrics as an implementation of the api.<br />(Ari Rabkin via acmurthy)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-2302">HADOOP-2302</a>. Provides a comparator for numerical sorting of key fields.<br />(ddas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-153">HADOOP-153</a>. Provides a way to skip bad records.<br />(Sharad Agarwal via ddas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-657">HADOOP-657</a>. Free disk space should be modelled and used by the scheduler
+to make scheduling decisions.<br />(Ari Rabkin via omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3719">HADOOP-3719</a>. Initial checkin of Chukwa, which is a data collection and
+analysis framework.<br />(Jerome Boulon, Andy Konwinski, Ari Rabkin,
+and Eric Yang)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3873">HADOOP-3873</a>. Add -filelimit and -sizelimit options to distcp to cap the
+number of files/bytes copied in a particular run to support incremental
+updates and mirroring. (TszWo (Nicholas), SZE via cdouglas)
+</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3585">HADOOP-3585</a>. FailMon package for hardware failure monitoring and
+analysis of anomalies.<br />(Ioannis Koltsidas via dhruba)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-1480">HADOOP-1480</a>. Add counters to the C++ Pipes API.<br />(acmurthy via omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3854">HADOOP-3854</a>. Add support for pluggable servlet filters in the HttpServers.
+(Tsz Wo (Nicholas) Sze via omalley)
+</li>
     </ol>
   </li>
   <li><a href="javascript:toggleList('trunk_(unreleased_changes)_._improvements_')">  IMPROVEMENTS
-</a>&nbsp;&nbsp;&nbsp;(7)
+</a>&nbsp;&nbsp;&nbsp;(40)
     <ol id="trunk_(unreleased_changes)_._improvements_">
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3732">HADOOP-3732</a>. Delay intialization of datanode block verification till
+the verification thread is started.<br />(rangadi)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-1627">HADOOP-1627</a>. Various small improvements to 'dfsadmin -report' output.<br />(rangadi)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3577">HADOOP-3577</a>. Tools to inject blocks into name node and simulated
 data nodes for testing.<br />(Sanjay Radia via hairong)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-2664">HADOOP-2664</a>. Add a lzop compatible codec, so that files compressed by lzop
@@ -108,32 +146,196 @@ Loughran via omalley)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3402">HADOOP-3402</a>. Add terasort example program<br />(omalley)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3660">HADOOP-3660</a>. Add replication factor for injecting blocks in simulated
 datanodes.<br />(Sanjay Radia via cdouglas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3684">HADOOP-3684</a>. Add a cloning function to the contrib/data_join framework
+permitting users to define a more efficient method for cloning values from
+the reduce than serialization/deserialization.<br />(Runping Qi via cdouglas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3478">HADOOP-3478</a>. Improves the handling of map output fetching. Now the
+randomization is by the hosts (and not the map outputs themselves).<br />(Jothi Padmanabhan via ddas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3617">HADOOP-3617</a>. Removed redundant checks of accounting space in MapTask and
+makes the spill thread persistent so as to avoid creating a new one for
+each spill.<br />(Chris Douglas via acmurthy)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3412">HADOOP-3412</a>. Factor the scheduler out of the JobTracker and make
+it pluggable.<br />(Tom White and Brice Arnould via omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3756">HADOOP-3756</a>. Minor. Remove unused dfs.client.buffer.dir from
+hadoop-default.xml.<br />(rangadi)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3327">HADOOP-3327</a>. Treats connection and read timeouts differently in the
+shuffle and the backoff logic is dependent on the type of timeout.<br />(Jothi Padmanabhan via ddas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3747">HADOOP-3747</a>. Adds counter suport for MultipleOutputs.<br />(Alejandro Abdelnur via ddas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3169">HADOOP-3169</a>. LeaseChecker daemon should not be started in DFSClient
+constructor. (TszWo (Nicholas), SZE via hairong)
+</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3824">HADOOP-3824</a>. Move base functionality of StatusHttpServer to a core
+package. (TszWo (Nicholas), SZE via cdouglas)
+</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3646">HADOOP-3646</a>. Add a bzip2 compatible codec, so bzip compressed data
+may be processed by map/reduce.<br />(Abdul Qadeer via cdouglas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3861">HADOOP-3861</a>. MapFile.Reader and Writer should implement Closeable.<br />(tomwhite via omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3791">HADOOP-3791</a>. Introduce generics into ReflectionUtils.<br />(Chris Smith via
+cdouglas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3694">HADOOP-3694</a>. Improve unit test performance by changing
+MiniDFSCluster to listen only on 127.0.0.1.<br />(cutting)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3620">HADOOP-3620</a>. Namenode should synchronously resolve a datanode's network
+location when the datanode registers.<br />(hairong)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3860">HADOOP-3860</a>. NNThroughputBenchmark is extended with rename and delete
+benchmarks.<br />(shv)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3892">HADOOP-3892</a>. Include unix group name in JobConf.<br />(Matei Zaharia via johan)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3875">HADOOP-3875</a>. Change the time period between heartbeats to be relative to
+the end of the heartbeat rpc, rather than the start. This causes better
+behavior if the JobTracker is overloaded.<br />(acmurthy via omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3853">HADOOP-3853</a>. Move multiple input format (<a href="http://issues.apache.org/jira/browse/HADOOP-372">HADOOP-372</a>) extension to
+library package.<br />(tomwhite via johan)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-9">HADOOP-9</a>. Use roulette scheduling for temporary space when the size
+is not known.<br />(Ari Rabkin via omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3202">HADOOP-3202</a>. Use recursive delete rather than FileUtil.fullyDelete.<br />(Amareshwari Sriramadasu via omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3368">HADOOP-3368</a>. Remove common-logging.properties from conf.<br />(Steve Loughran
+via omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3851">HADOOP-3851</a>. Fix spelling mistake in FSNamesystemMetrics.<br />(Steve Loughran
+via omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3780">HADOOP-3780</a>. Remove asynchronous resolution of network topology in the
+JobTracker<br />(Amar Kamat via omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3852">HADOOP-3852</a>. Add ShellCommandExecutor.toString method to make nicer
+error messages.<br />(Steve Loughran via omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3844">HADOOP-3844</a>. Include message of local exception in RPC client failures.<br />(Steve Loughran via omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3935">HADOOP-3935</a>. Split out inner classes from DataNode.java.<br />(johan)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3905">HADOOP-3905</a>. Create generic interfaces for edit log streams.<br />(shv)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3934">HADOOP-3934</a>. Upgrade log4j to 1.2.15.<br />(omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3062">HADOOP-3062</a>. Add metrics to DataNode and TaskTracker to record network
+traffic for HDFS reads/writes and MR shuffling.<br />(cdouglas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3742">HADOOP-3742</a>. Remove HDFS from public java doc and add javadoc-dev for
+generative javadoc for developers.<br />(Sanjay Radia via omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3944">HADOOP-3944</a>. Improve documentation for public TupleWritable class in
+join package.<br />(Chris Douglas via enis)</li>
     </ol>
   </li>
   <li><a href="javascript:toggleList('trunk_(unreleased_changes)_._optimizations_')">  OPTIMIZATIONS
-</a>&nbsp;&nbsp;&nbsp;(1)
+</a>&nbsp;&nbsp;&nbsp;(6)
     <ol id="trunk_(unreleased_changes)_._optimizations_">
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3556">HADOOP-3556</a>. Removed lock contention in MD5Hash by changing the
 singleton MessageDigester by an instance per Thread using
 ThreadLocal.<br />(Iv?n de Prado via omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3328">HADOOP-3328</a>. When client is writing data to DFS, only the last
+datanode in the pipeline needs to verify the checksum. Saves around
+30% CPU on intermediate datanodes.<br />(rangadi)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3863">HADOOP-3863</a>. Use a thread-local string encoder rather than a static one
+that is protected by a lock.<br />(acmurthy via omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3864">HADOOP-3864</a>. Prevent the JobTracker from locking up when a job is being
+initialized.<br />(acmurthy via omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3816">HADOOP-3816</a>. Faster directory listing in KFS.<br />(Sriram Rao via omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-2130">HADOOP-2130</a>. Pipes submit job should have both blocking and non-blocking
+versions.<br />(acmurthy via omalley)</li>
     </ol>
   </li>
   <li><a href="javascript:toggleList('trunk_(unreleased_changes)_._bug_fixes_')">  BUG FIXES
-</a>&nbsp;&nbsp;&nbsp;(2)
+</a>&nbsp;&nbsp;&nbsp;(44)
     <ol id="trunk_(unreleased_changes)_._bug_fixes_">
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3563">HADOOP-3563</a>.  Refactor the distributed upgrade code so that it is
 easier to identify datanode and namenode related code.<br />(dhruba)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3640">HADOOP-3640</a>. Fix the read method in the NativeS3InputStream.<br />(tomwhite via
 omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3711">HADOOP-3711</a>. Fixes the Streaming input parsing to properly find the
+separator.<br />(Amareshwari Sriramadasu via ddas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3725">HADOOP-3725</a>. Prevent TestMiniMRMapDebugScript from swallowing exceptions.<br />(Steve Loughran via cdouglas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3726">HADOOP-3726</a>. Throw exceptions from TestCLI setup and teardown instead of
+swallowing them.<br />(Steve Loughran via cdouglas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3721">HADOOP-3721</a>. Refactor CompositeRecordReader and related mapred.join classes
+to make them clearer.<br />(cdouglas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3720">HADOOP-3720</a>. Re-read the config file when dfsadmin -refreshNodes is invoked
+so dfs.hosts and dfs.hosts.exclude are observed.<br />(lohit vijayarenu via
+cdouglas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3485">HADOOP-3485</a>. Allow writing to files over fuse.<br />(Pete Wyckoff via dhruba)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3723">HADOOP-3723</a>. The flags to the libhdfs.create call can be treated as
+a bitmask.<br />(Pete Wyckoff via dhruba)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3643">HADOOP-3643</a>. Filter out completed tasks when asking for running tasks in
+the JobTracker web/ui.<br />(Amar Kamat via omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3777">HADOOP-3777</a>. Ensure that Lzo compressors/decompressors correctly handle the
+case where native libraries aren't available.<br />(Chris Douglas via acmurthy)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3728">HADOOP-3728</a>. Fix SleepJob so that it doesn't depend on temporary files,
+this ensures we can now run more than one instance of SleepJob
+simultaneously.<br />(Chris Douglas via acmurthy)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3795">HADOOP-3795</a>. Fix saving image files on Namenode with different checkpoint
+stamps.<br />(Lohit Vijayarenu via mahadev)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3624">HADOOP-3624</a>. Improving createeditslog to create tree directory structure.<br />(Lohit Vijayarenu via mahadev)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3778">HADOOP-3778</a>. DFSInputStream.seek() did not retry in case of some errors.<br />(LN via rangadi)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3661">HADOOP-3661</a>. The handling of moving files deleted through fuse-dfs to
+Trash made similar to the behaviour from dfs shell.<br />(Pete Wyckoff via dhruba)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3819">HADOOP-3819</a>. Unset LANG and LC_CTYPE in saveVersion.sh to make it
+compatible with non-English locales.<br />(Rong-En Fan via cdouglas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3848">HADOOP-3848</a>. Cache calls to getSystemDir in the TaskTracker instead of
+calling it for each task start.<br />(acmurthy via omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3131">HADOOP-3131</a>. Fix reduce progress reporting for compressed intermediate
+data.<br />(Matei Zaharia via acmurthy)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3796">HADOOP-3796</a>. fuse-dfs configuration is implemented as file system
+mount options.<br />(Pete Wyckoff via dhruba)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3836">HADOOP-3836</a>. Fix TestMultipleOutputs to correctly clean up.<br />(Alejandro
+Abdelnur via acmurthy)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3805">HADOOP-3805</a>. Improve fuse-dfs write performance.<br />(Pete Wyckoff via zshao)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3846">HADOOP-3846</a>. Fix unit test CreateEditsLog to generate paths correctly.<br />(Lohit Vjayarenu via cdouglas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3904">HADOOP-3904</a>. Fix unit tests using the old dfs package name.
+(TszWo (Nicholas), SZE via johan)
+</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3319">HADOOP-3319</a>. Fix some HOD error messages to go stderr instead of
+stdout.<br />(Vinod Kumar Vavilapalli via omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3907">HADOOP-3907</a>. Move INodeDirectoryWithQuota to its own .java file.
+(Tsz Wo (Nicholas), SZE via hairong)
+</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3919">HADOOP-3919</a>. Fix attribute name in hadoop-default for
+mapred.jobtracker.instrumentation.<br />(Ari Rabkin via omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3903">HADOOP-3903</a>. Change the package name for the servlets to be hdfs instead of
+dfs. (Tsz Wo (Nicholas) Sze via omalley)
+</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3773">HADOOP-3773</a>. Change Pipes to set the default map output key and value
+types correctly.<br />(Koji Noguchi via omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3940">HADOOP-3940</a>. Fix in-memory merge condition to wait when there are no map
+outputs or when the final map outputs are being fetched without contention.<br />(cdouglas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3952">HADOOP-3952</a>. Fix compilation error in TestDataJoin referencing dfs package.<br />(omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3951">HADOOP-3951</a>. Fix package name for FSNamesystem logs and modify other
+hard-coded Logs to use the class name.<br />(cdouglas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3889">HADOOP-3889</a>. Improve error reporting from HftpFileSystem, handling in
+DistCp. (Tsz Wo (Nicholas), SZE via cdouglas)
+</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3946">HADOOP-3946</a>. Fix TestMapRed after hadoop-3664.<br />(tomwhite via omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3949">HADOOP-3949</a>. Remove duplicate jars from Chukwa.<br />(Jerome Boulon via omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3933">HADOOP-3933</a>. DataNode sometimes sends up to io.byte.per.checksum bytes
+more than required to client.<br />(Ning Li via rangadi)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3962">HADOOP-3962</a>. Shell command "fs -count" should support paths with different
+file systems. (Tsz Wo (Nicholas), SZE via mahadev)
+</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3957">HADOOP-3957</a>. Fix javac warnings in DistCp and TestCopyFiles. (Tsz Wo
+(Nicholas), SZE via cdouglas)
+</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3958">HADOOP-3958</a>. Fix TestMapRed to check the success of test-job.<br />(omalley via
+acmurthy)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3985">HADOOP-3985</a>. Fix TestHDFSServerPorts to use random ports.<br />(Hairong Kuang
+via omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3964">HADOOP-3964</a>. Fix javadoc warnings introduced by FailMon.<br />(dhruba)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3785">HADOOP-3785</a>. Fix FileSystem cache to be case-insensitive for scheme and
+authority.<br />(Bill de hOra via cdouglas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3506">HADOOP-3506</a>. Fix a rare NPE caused by error handling in S3.<br />(Tom White via
+cdouglas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3705">HADOOP-3705</a>. Fix mapred.join parser to accept InputFormats named with
+underscore and static, inner classes.<br />(cdouglas)</li>
     </ol>
   </li>
 </ul>
-<h2><a href="javascript:toggleList('release_0.18.0_-_unreleased_')">Release 0.18.0 - Unreleased
+<h2><a href="javascript:toggleList('release_0.18.1_-_unreleased_')">Release 0.18.1 - Unreleased
 </a></h2>
-<ul id="release_0.18.0_-_unreleased_">
-  <li><a href="javascript:toggleList('release_0.18.0_-_unreleased_._incompatible_changes_')">  INCOMPATIBLE CHANGES
+<ul id="release_0.18.1_-_unreleased_">
+  <li><a href="javascript:toggleList('release_0.18.1_-_unreleased_._bug_fixes_')">  BUG FIXES
+</a>&nbsp;&nbsp;&nbsp;(1)
+    <ol id="release_0.18.1_-_unreleased_._bug_fixes_">
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3821">HADOOP-3821</a>. Prevent SequenceFile and IFile from duplicating codecs in
+CodecPool when closed more than once.<br />(Arun Murthy via cdouglas)</li>
+    </ol>
+  </li>
+</ul>
+<h2><a href="javascript:toggleList('older')">Older Releases</a></h2>
+<ul id="older">
+<h3><a href="javascript:toggleList('release_0.18.0_-_2008-08-19_')">Release 0.18.0 - 2008-08-19
+</a></h3>
+<ul id="release_0.18.0_-_2008-08-19_">
+  <li><a href="javascript:toggleList('release_0.18.0_-_2008-08-19_._incompatible_changes_')">  INCOMPATIBLE CHANGES
 </a>&nbsp;&nbsp;&nbsp;(23)
-    <ol id="release_0.18.0_-_unreleased_._incompatible_changes_">
+    <ol id="release_0.18.0_-_2008-08-19_._incompatible_changes_">
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-2703">HADOOP-2703</a>.  The default options to fsck skips checking files
 that are being written to. The output of fsck is incompatible
 with previous release.<br />(lohit vijayarenu via dhruba)</li>
@@ -209,9 +411,9 @@ to SequenceFiles.<br />(cdouglas)</li>
 </li>
     </ol>
   </li>
-  <li><a href="javascript:toggleList('release_0.18.0_-_unreleased_._new_features_')">  NEW FEATURES
+  <li><a href="javascript:toggleList('release_0.18.0_-_2008-08-19_._new_features_')">  NEW FEATURES
 </a>&nbsp;&nbsp;&nbsp;(25)
-    <ol id="release_0.18.0_-_unreleased_._new_features_">
+    <ol id="release_0.18.0_-_2008-08-19_._new_features_">
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3074">HADOOP-3074</a>. Provides a UrlStreamHandler for DFS and other FS,
 relying on FileSystem<br />(taton)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-2585">HADOOP-2585</a>. Name-node imports namespace data from a recent checkpoint
@@ -264,9 +466,11 @@ framework.<br />(tomwhite via omalley)</li>
 in hadoop user guide.<br />(shv)</li>
     </ol>
   </li>
-  <li><a href="javascript:toggleList('release_0.18.0_-_unreleased_._improvements_')">  IMPROVEMENTS
-</a>&nbsp;&nbsp;&nbsp;(46)
-    <ol id="release_0.18.0_-_unreleased_._improvements_">
+  <li><a href="javascript:toggleList('release_0.18.0_-_2008-08-19_._improvements_')">  IMPROVEMENTS
+</a>&nbsp;&nbsp;&nbsp;(47)
+    <ol id="release_0.18.0_-_2008-08-19_._improvements_">
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3677">HADOOP-3677</a>. Simplify generation stamp upgrade by making is a
+local upgrade on datandodes. Deleted distributed upgrade.<br />(rangadi)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-2928">HADOOP-2928</a>. Remove deprecated FileSystem.getContentLength().<br />(Lohit Vjayarenu via rangadi)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3130">HADOOP-3130</a>. Make the connect timeout smaller for getFile.<br />(Amar Ramesh Kamat via ddas)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3160">HADOOP-3160</a>. Remove deprecated exists() from ClientProtocol and
@@ -359,9 +563,9 @@ via the DistributedCache.<br />(Amareshwari Sriramadasu via ddas)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3688">HADOOP-3688</a>. Fix up HDFS docs.<br />(Robert Chansler via hairong)</li>
     </ol>
   </li>
-  <li><a href="javascript:toggleList('release_0.18.0_-_unreleased_._optimizations_')">  OPTIMIZATIONS
+  <li><a href="javascript:toggleList('release_0.18.0_-_2008-08-19_._optimizations_')">  OPTIMIZATIONS
 </a>&nbsp;&nbsp;&nbsp;(10)
-    <ol id="release_0.18.0_-_unreleased_._optimizations_">
+    <ol id="release_0.18.0_-_2008-08-19_._optimizations_">
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3274">HADOOP-3274</a>. The default constructor of BytesWritable creates empty
 byte array. (Tsz Wo (Nicholas), SZE via shv)
 </li>
@@ -382,9 +586,9 @@ InputFormat.validateInput.<br />(tomwhite via omalley)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3552">HADOOP-3552</a>. Add forrest documentation for Hadoop commands.<br />(Sharad Agarwal via cdouglas)</li>
     </ol>
   </li>
-  <li><a href="javascript:toggleList('release_0.18.0_-_unreleased_._bug_fixes_')">  BUG FIXES
-</a>&nbsp;&nbsp;&nbsp;(117)
-    <ol id="release_0.18.0_-_unreleased_._bug_fixes_">
+  <li><a href="javascript:toggleList('release_0.18.0_-_2008-08-19_._bug_fixes_')">  BUG FIXES
+</a>&nbsp;&nbsp;&nbsp;(144)
+    <ol id="release_0.18.0_-_2008-08-19_._bug_fixes_">
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-2905">HADOOP-2905</a>. 'fsck -move' triggers NPE in NameNode.<br />(Lohit Vjayarenu via rangadi)</li>
       <li>Increment ClientProtocol.versionID missed by <a href="http://issues.apache.org/jira/browse/HADOOP-2585">HADOOP-2585</a>.<br />(shv)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3254">HADOOP-3254</a>. Restructure internal namenode methods that process
@@ -460,8 +664,6 @@ write operation.<br />(rangadi)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3296">HADOOP-3296</a>. Fix task cache to work for more than two levels in the cache
 hierarchy. This also adds a new counter to track cache hits at levels
 greater than two.<br />(Amar Kamat via cdouglas)</li>
-      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3370">HADOOP-3370</a>. Ensure that the TaskTracker.runningJobs data-structure is
-correctly cleaned-up on task completion.<br />(Zheng Shao via acmurthy)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3375">HADOOP-3375</a>. Lease paths were sometimes not removed from
 LeaseManager.sortedLeasesByPath. (Tsz Wo (Nicholas), SZE via dhruba)
 </li>
@@ -500,10 +702,13 @@ logging to let failures be reproducible.<br />(cdouglas)</li>
 single spill.<br />(omalley via cdouglas)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3454">HADOOP-3454</a>. Fix Text::find to search only valid byte ranges.<br />(Chad Whipkey
 via cdouglas)</li>
-      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3417">HADOOP-3417</a>. Removes the static configuration variable, commandLineConfig from
-JobClient. Moves the cli parsing from JobShell to GenericOptionsParser.
-Thus removes the class org.apache.hadoop.mapred.JobShell.<br />(Amareshwari Sriramadasu via ddas)</li>
-      <li><a href="http://issues.apache.org/jira/browse/HADOOP-2132">HADOOP-2132</a>. Only RUNNING/PREP jobs can be killed.<br />(Jothi Padmanabhan via ddas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3417">HADOOP-3417</a>. Removes the static configuration variable,
+commandLineConfig from JobClient. Moves the cli parsing from
+JobShell to GenericOptionsParser.  Thus removes the class
+org.apache.hadoop.mapred.JobShell.<br />(Amareshwari Sriramadasu via
+ddas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-2132">HADOOP-2132</a>. Only RUNNING/PREP jobs can be killed.<br />(Jothi Padmanabhan
+via ddas)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3476">HADOOP-3476</a>. Code cleanup in fuse-dfs.<br />(Peter Wyckoff via dhruba)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-2427">HADOOP-2427</a>. Ensure that the cwd of completed tasks is cleaned-up
 correctly on task-completion.<br />(Amareshwari Sri Ramadasu via acmurthy)</li>
@@ -522,14 +727,16 @@ Instead the file is created in the test directory<br />(Mahadev Konar via ddas)<
 in <a href="http://issues.apache.org/jira/browse/HADOOP-3095">HADOOP-3095</a>.<br />(tomwhite)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3135">HADOOP-3135</a>. Get the system directory from the JobTracker instead of from
 the conf.<br />(Subramaniam Krishnan via ddas)</li>
-      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3503">HADOOP-3503</a>. Fix a race condition when client and namenode start simultaneous
-recovery of the same block.  (dhruba &amp; Tsz Wo (Nicholas), SZE)
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3503">HADOOP-3503</a>. Fix a race condition when client and namenode start
+simultaneous recovery of the same block.  (dhruba &amp; Tsz Wo
+(Nicholas), SZE)
 </li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3440">HADOOP-3440</a>. Fixes DistributedCache to not create symlinks for paths which
 don't have fragments even when createSymLink is true.<br />(Abhijit Bagri via ddas)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3463">HADOOP-3463</a>. Hadoop-daemons script should cd to $HADOOP_HOME.<br />(omalley)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3489">HADOOP-3489</a>. Fix NPE in SafeModeMonitor.<br />(Lohit Vijayarenu via shv)</li>
-      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3509">HADOOP-3509</a>. Fix NPE in FSNamesystem.close. (Tsz Wo (Nicholas), SZE via shv)
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3509">HADOOP-3509</a>. Fix NPE in FSNamesystem.close. (Tsz Wo (Nicholas), SZE via
+shv)
 </li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3491">HADOOP-3491</a>. Name-node shutdown causes InterruptedException in
 ResolutionMonitor.<br />(Lohit Vijayarenu via shv)</li>
@@ -573,6 +780,10 @@ the tasktracker would reinitialize and bind to a different port).<br />(Jothi Pa
 system when the NameNode shuts down. (Tsz Wo (Nicholas) Sze via omalley)
 </li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3546">HADOOP-3546</a>. TaskTracker re-initialization gets stuck in cleaning up.<br />(Amareshwari Sriramadasu via ddas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3576">HADOOP-3576</a>. Fix NullPointerException when renaming a directory
+to its subdirectory. (Tse Wo (Nicholas), SZE via hairong)
+</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3320">HADOOP-3320</a>. Fix NullPointerException in NetworkTopology.getDistance().<br />(hairong)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3569">HADOOP-3569</a>. KFS input stream read() now correctly reads 1 byte
 instead of 4.<br />(Sriram Rao via omalley)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3599">HADOOP-3599</a>. Fix JobConf::setCombineOnceOnly to modify the instance rather
@@ -588,18 +799,16 @@ not yet resolved.<br />(Amar Ramesh Kamat via ddas)</li>
 current semantics.<br />(lohit vijayarenu via cdouglas)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3480">HADOOP-3480</a>.  Need to update Eclipse template to reflect current trunk.<br />(Brice Arnould via tomwhite)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3588">HADOOP-3588</a>. Fixed usability issues with archives.<br />(mahadev)</li>
-      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3536">HADOOP-3536</a>. Uncaught exception in DataBlockScanner.
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3635">HADOOP-3635</a>. Uncaught exception in DataBlockScanner.
 (Tsz Wo (Nicholas), SZE via hairong)
 </li>
-      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3539">HADOOP-3539</a>. Exception when closing DFSClient while multiple files are
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3639">HADOOP-3639</a>. Exception when closing DFSClient while multiple files are
 open.<br />(Benjamin Gufler via hairong)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3572">HADOOP-3572</a>. SetQuotas usage interface has some minor bugs.<br />(hairong)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3649">HADOOP-3649</a>. Fix bug in removing blocks from the corrupted block map.<br />(Lohit Vijayarenu via shv)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3604">HADOOP-3604</a>. Work around a JVM synchronization problem observed while
 retrieving the address of direct buffers from compression code by obtaining
 a lock during this call.<br />(Arun C Murthy via cdouglas)</li>
-      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3678">HADOOP-3678</a>. Avoid spurious exceptions logged at DataNode when clients
-read from DFS.<br />(rangadi)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3683">HADOOP-3683</a>. Fix dfs metrics to count file listings rather than files
 listed.<br />(lohit vijayarenu via cdouglas)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3597">HADOOP-3597</a>. Fix SortValidator to use filesystems other than the default as
@@ -614,21 +823,87 @@ classpath jars.<br />(Brice Arnould via nigel)</li>
 sources<br />(cdouglas)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3706">HADOOP-3706</a>. Fix a ClassLoader issue in the mapred.join Parser that
 prevents it from loading user-specified InputFormats.<br />(Jingkei Ly via cdouglas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3718">HADOOP-3718</a>. Fix KFSOutputStream::write(int) to output a byte instead of
+an int, per the OutputStream contract.<br />(Sriram Rao via cdouglas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3647">HADOOP-3647</a>. Add debug logs to help track down a very occassional,
+hard-to-reproduce, bug in shuffle/merge on the reducer.<br />(acmurthy)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3716">HADOOP-3716</a>. Prevent listStatus in KosmosFileSystem from returning
+null for valid, empty directories.<br />(Sriram Rao via cdouglas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3752">HADOOP-3752</a>. Fix audit logging to record rename events.<br />(cdouglas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3737">HADOOP-3737</a>. Fix CompressedWritable to call Deflater::end to release
+compressor memory.<br />(Grant Glouser via cdouglas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3670">HADOOP-3670</a>. Fixes JobTracker to clear out split bytes when no longer
+required.<br />(Amareshwari Sriramadasu via ddas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3755">HADOOP-3755</a>. Update gridmix to work with HOD 0.4<br />(Runping Qi via cdouglas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3743">HADOOP-3743</a>. Fix -libjars, -files, -archives options to work even if
+user code does not implement tools.<br />(Amareshwari Sriramadasu via mahadev)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3774">HADOOP-3774</a>. Fix typos in shell output. (Tsz Wo (Nicholas), SZE via
+cdouglas)
+</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3762">HADOOP-3762</a>. Fixed FileSystem cache to work with the default port.<br />(cutting
+via omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3798">HADOOP-3798</a>. Fix tests compilation.<br />(Mukund Madhugiri via omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3794">HADOOP-3794</a>. Return modification time instead of zero for KosmosFileSystem.<br />(Sriram Rao via cdouglas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3806">HADOOP-3806</a>. Remove debug statement to stdout from QuickSort.<br />(cdouglas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3776">HADOOP-3776</a>. Fix NPE at NameNode when datanode reports a block after it is
+deleted at NameNode.<br />(rangadi)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3537">HADOOP-3537</a>. Disallow adding a datanode to a network topology when its
+network location is not resolved.<br />(hairong)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3571">HADOOP-3571</a>. Fix bug in block removal used in lease recovery.<br />(shv)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3645">HADOOP-3645</a>. MetricsTimeVaryingRate returns wrong value for
+metric_avg_time.<br />(Lohit Vijayarenu via hairong)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3521">HADOOP-3521</a>. Reverted the missing cast to float for sending Counters' values
+to Hadoop metrics which was removed by <a href="http://issues.apache.org/jira/browse/HADOOP-544">HADOOP-544</a>.<br />(acmurthy)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3820">HADOOP-3820</a>. Fixes two problems in the gridmix-env - a syntax error, and a
+wrong definition of USE_REAL_DATASET by default.<br />(Arun Murthy via ddas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3724">HADOOP-3724</a>. Fixes two problems related to storing and recovering lease
+in the fsimage.<br />(dhruba)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3827">HADOOP-3827</a>.  Fixed compression of empty map-outputs.<br />(acmurthy)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3865">HADOOP-3865</a>. Remove reference to FSNamesystem from metrics preventing
+garbage collection.<br />(Lohit Vijayarenu via cdouglas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3884">HADOOP-3884</a>.  Fix so that Eclipse plugin builds against recent
+Eclipse releases.<br />(cutting)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3837">HADOOP-3837</a>. Streaming jobs report progress status.<br />(dhruba)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3897">HADOOP-3897</a>. Fix a NPE in secondary namenode.<br />(Lohit Vijayarenu via
+cdouglas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3901">HADOOP-3901</a>. Fix bin/hadoop to correctly set classpath under cygwin.
+(Tsz Wo (Nicholas) Sze via omalley)
+</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3947">HADOOP-3947</a>. Fix a problem in tasktracker reinitialization.<br />(Amareshwari Sriramadasu via ddas)</li>
     </ol>
   </li>
 </ul>
-<h2><a href="javascript:toggleList('older')">Older Releases</a></h2>
-<ul id="older">
-<h3><a href="javascript:toggleList('release_0.17.2_-_unreleased_')">Release 0.17.2 - Unreleased
+<h3><a href="javascript:toggleList('release_0.17.2_-_2008-08-11_')">Release 0.17.2 - 2008-08-11
 </a></h3>
-<ul id="release_0.17.2_-_unreleased_">
-  <li><a href="javascript:toggleList('release_0.17.2_-_unreleased_._bug_fixes_')">  BUG FIXES
-</a>&nbsp;&nbsp;&nbsp;(3)
-    <ol id="release_0.17.2_-_unreleased_._bug_fixes_">
+<ul id="release_0.17.2_-_2008-08-11_">
+  <li><a href="javascript:toggleList('release_0.17.2_-_2008-08-11_._bug_fixes_')">  BUG FIXES
+</a>&nbsp;&nbsp;&nbsp;(12)
+    <ol id="release_0.17.2_-_2008-08-11_._bug_fixes_">
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3678">HADOOP-3678</a>. Avoid spurious exceptions logged at DataNode when clients
+read from DFS.<br />(rangadi)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3707">HADOOP-3707</a>. NameNode keeps a count of number of blocks scheduled
+to be written to a datanode and uses it to avoid allocating more
+blocks than a datanode can hold.<br />(rangadi)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3760">HADOOP-3760</a>. Fix a bug with HDFS file close() mistakenly introduced
+by <a href="http://issues.apache.org/jira/browse/HADOOP-3681">HADOOP-3681</a>.<br />(Lohit Vijayarenu via rangadi)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3681">HADOOP-3681</a>. DFSClient can get into an infinite loop while closing
 a file if there are some errors.<br />(Lohit Vijayarenu via rangadi)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3002">HADOOP-3002</a>. Hold off block removal while in safe mode.<br />(shv)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3685">HADOOP-3685</a>. Unbalanced replication target.<br />(hairong)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3758">HADOOP-3758</a>. Shutdown datanode on version mismatch instead of retrying
+continuously, preventing excessive logging at the namenode.<br />(lohit vijayarenu via cdouglas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3633">HADOOP-3633</a>. Correct exception handling in DataXceiveServer, and throttle
+the number of xceiver threads in a data-node.<br />(shv)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3370">HADOOP-3370</a>. Ensure that the TaskTracker.runningJobs data-structure is
+correctly cleaned-up on task completion.<br />(Zheng Shao via acmurthy)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3813">HADOOP-3813</a>. Fix task-output clean-up on HDFS to use the recursive
+FileSystem.delete rather than the FileUtil.fullyDelete.<br />(Amareshwari
+Sri Ramadasu via acmurthy)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3859">HADOOP-3859</a>. Allow the maximum number of xceivers in the data node to
+be configurable.<br />(Johan Oskarsson via omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3931">HADOOP-3931</a>. Fix corner case in the map-side sort that causes some values
+to be counted as too large and cause pre-mature spills to disk. Some values
+will also bypass the combiner incorrectly.<br />(cdouglas via omalley)</li>
     </ol>
   </li>
 </ul>
@@ -652,7 +927,7 @@ therefore provides better resource management.<br />(hairong)</li>
     </ol>
   </li>
   <li><a href="javascript:toggleList('release_0.17.1_-_2008-06-23_._bug_fixes_')">  BUG FIXES
-</a>&nbsp;&nbsp;&nbsp;(14)
+</a>&nbsp;&nbsp;&nbsp;(8)
     <ol id="release_0.17.1_-_2008-06-23_._bug_fixes_">
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-2159">HADOOP-2159</a> Namenode stuck in safemode. The counter blockSafe should
 not be decremented for invalid blocks.<br />(hairong)</li>
@@ -671,17 +946,6 @@ value lengths are incorrectly calculated.<br />(cdouglas)</li>
 in the reduce.<br />(Spyros Blanas via cdouglas)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-1979">HADOOP-1979</a>. Speed up fsck by adding a buffered stream.<br />(Lohit
 Vijaya Renu via omalley)</li>
-      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3576">HADOOP-3576</a>. Fix NullPointerException when renaming a directory
-to its subdirectory. (Tse Wo (Nicholas), SZE via hairong)
-</li>
-      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3320">HADOOP-3320</a>. Fix NullPointerException in NetworkTopology.getDistance().<br />(hairong)</li>
-      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3537">HADOOP-3537</a>. Disallow adding a datanode to a network topology when its
-network location is not resolved.<br />(hairong)</li>
-      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3571">HADOOP-3571</a>. Fix bug in block removal used in lease recovery.<br />(shv)</li>
-      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3645">HADOOP-3645</a>. MetricsTimeVaryingRate returns wrong value for
-metric_avg_time.<br />(Lohit Vijayarenu via hairong)</li>
-      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3633">HADOOP-3633</a>. Correct exception handling in DataXceiveServer, and throttle
-the number of xceiver threads in a data-node.<br />(shv)</li>
     </ol>
   </li>
 </ul>

docs/commands_manual.html

@@ -315,7 +315,11 @@ document.write("Last Published: " + document.lastModified);
 <p>
 				  Following are supported by <a href="commands_manual.html#dfsadmin">dfsadmin</a>, 
 				  <a href="commands_manual.html#fs">fs</a>, <a href="commands_manual.html#fsck">fsck</a> and 
-				  <a href="commands_manual.html#job">job</a>.
+				  <a href="commands_manual.html#job">job</a>. 
+				  Applications should implement 
+				  <a href="api/org/apache/hadoop/util/Tool.html">Tool</a> to support
+				  <a href="api/org/apache/hadoop/util/GenericOptionsParser.html">
+				  GenericOptions</a>.
 				</p>
 <table class="ForrestTable" cellspacing="1" cellpadding="4">
 			          
@@ -380,11 +384,11 @@ document.write("Last Published: " + document.lastModified);
 </div>
 		
 		
-<a name="N100FB"></a><a name="User+Commands"></a>
+<a name="N10103"></a><a name="User+Commands"></a>
 <h2 class="h3"> User Commands </h2>
 <div class="section">
 <p>Commands useful for users of a hadoop cluster.</p>
-<a name="N10104"></a><a name="archive"></a>
+<a name="N1010C"></a><a name="archive"></a>
 <h3 class="h4"> archive </h3>
 <p>
 					Creates a hadoop archive. More information can be found at <a href="hadoop_archives.html">Hadoop Archives</a>.
@@ -422,7 +426,7 @@ document.write("Last Published: " + document.lastModified);
 </tr>
 			     
 </table>
-<a name="N1014F"></a><a name="distcp"></a>
+<a name="N10157"></a><a name="distcp"></a>
 <h3 class="h4"> distcp </h3>
 <p>
 					Copy file or directories recursively. More information can be found at <a href="distcp.html">DistCp Guide</a>.
@@ -454,7 +458,7 @@ document.write("Last Published: " + document.lastModified);
 </tr>
 			     
 </table>
-<a name="N1018C"></a><a name="fs"></a>
+<a name="N10194"></a><a name="fs"></a>
 <h3 class="h4"> fs </h3>
 <p>
 					
@@ -468,7 +472,7 @@ document.write("Last Published: " + document.lastModified);
 <p>
 					The various COMMAND_OPTIONS can be found at <a href="hdfs_shell.html">HDFS Shell Guide</a>.
 				</p>
-<a name="N101A8"></a><a name="fsck"></a>
+<a name="N101B0"></a><a name="fsck"></a>
 <h3 class="h4"> fsck </h3>
 <p>
 					Runs a HDFS filesystem checking utility. See <a href="hdfs_user_guide.html#Fsck">Fsck</a> for more info.
@@ -541,7 +545,7 @@ document.write("Last Published: " + document.lastModified);
 </tr>
 					
 </table>
-<a name="N1023C"></a><a name="jar"></a>
+<a name="N10244"></a><a name="jar"></a>
 <h3 class="h4"> jar </h3>
 <p>
 					Runs a jar file. Users can bundle their Map Reduce code in a jar file and execute it using this command.
@@ -561,7 +565,7 @@ document.write("Last Published: " + document.lastModified);
 					<a href="mapred_tutorial.html#Usage">Wordcount example</a>
 				
 </p>
-<a name="N1025A"></a><a name="job"></a>
+<a name="N10262"></a><a name="job"></a>
 <h3 class="h4"> job </h3>
 <p>
 					Command to interact with Map Reduce Jobs.
@@ -648,7 +652,7 @@ document.write("Last Published: " + document.lastModified);
 </tr>
 					
 </table>
-<a name="N102FA"></a><a name="pipes"></a>
+<a name="N10302"></a><a name="pipes"></a>
 <h3 class="h4"> pipes </h3>
 <p>
 					Runs a pipes job.
@@ -753,7 +757,7 @@ document.write("Last Published: " + document.lastModified);
 </tr>
 					
 </table>
-<a name="N103BF"></a><a name="version"></a>
+<a name="N103C7"></a><a name="version"></a>
 <h3 class="h4"> version </h3>
 <p>
 					Prints the version.
@@ -763,7 +767,7 @@ document.write("Last Published: " + document.lastModified);
 <span class="codefrag">Usage: hadoop version</span>
 				
 </p>
-<a name="N103CF"></a><a name="CLASSNAME"></a>
+<a name="N103D7"></a><a name="CLASSNAME"></a>
 <h3 class="h4"> CLASSNAME </h3>
 <p>
 					 hadoop script can be used to invoke any class.
@@ -779,11 +783,11 @@ document.write("Last Published: " + document.lastModified);
 </div>
 		
 		
-<a name="N103E3"></a><a name="Administration+Commands"></a>
+<a name="N103EB"></a><a name="Administration+Commands"></a>
 <h2 class="h3"> Administration Commands </h2>
 <div class="section">
 <p>Commands useful for administrators of a hadoop cluster.</p>
-<a name="N103EC"></a><a name="balancer"></a>
+<a name="N103F4"></a><a name="balancer"></a>
 <h3 class="h4"> balancer </h3>
 <p>
 					Runs a cluster balancing utility. An administrator can simply press Ctrl-C to stop the 
@@ -809,7 +813,7 @@ document.write("Last Published: " + document.lastModified);
 </tr>
 			     
 </table>
-<a name="N1041B"></a><a name="daemonlog"></a>
+<a name="N10423"></a><a name="daemonlog"></a>
 <h3 class="h4"> daemonlog </h3>
 <p>
 					 Get/Set the log level for each daemon.
@@ -846,7 +850,7 @@ document.write("Last Published: " + document.lastModified);
 </tr>
 			     
 </table>
-<a name="N10458"></a><a name="datanode"></a>
+<a name="N10460"></a><a name="datanode"></a>
 <h3 class="h4"> datanode</h3>
 <p>
 					Runs a HDFS datanode.
@@ -872,7 +876,7 @@ document.write("Last Published: " + document.lastModified);
 </tr>
 			     
 </table>
-<a name="N10483"></a><a name="dfsadmin"></a>
+<a name="N1048B"></a><a name="dfsadmin"></a>
 <h3 class="h4"> dfsadmin </h3>
 <p>
 					Runs a HDFS dfsadmin client.
@@ -987,7 +991,7 @@ document.write("Last Published: " + document.lastModified);
 </tr>
 			     
 </table>
-<a name="N10543"></a><a name="jobtracker"></a>
+<a name="N1054B"></a><a name="jobtracker"></a>
 <h3 class="h4"> jobtracker </h3>
 <p>
 					Runs the MapReduce job Tracker node.
@@ -997,7 +1001,7 @@ document.write("Last Published: " + document.lastModified);
 <span class="codefrag">Usage: hadoop jobtracker</span>
 				
 </p>
-<a name="N10553"></a><a name="namenode"></a>
+<a name="N1055B"></a><a name="namenode"></a>
 <h3 class="h4"> namenode </h3>
 <p>
 					Runs the namenode. More info about the upgrade, rollback and finalize is at 
@@ -1055,7 +1059,7 @@ document.write("Last Published: " + document.lastModified);
 </tr>
 			     
 </table>
-<a name="N105BA"></a><a name="secondarynamenode"></a>
+<a name="N105C2"></a><a name="secondarynamenode"></a>
 <h3 class="h4"> secondarynamenode </h3>
 <p>
 					Runs the HDFS secondary namenode. See <a href="hdfs_user_guide.html#Secondary+Namenode">Secondary Namenode</a> 
@@ -1089,7 +1093,7 @@ document.write("Last Published: " + document.lastModified);
 </tr>
 			     
 </table>
-<a name="N105F7"></a><a name="tasktracker"></a>
+<a name="N105FF"></a><a name="tasktracker"></a>
 <h3 class="h4"> tasktracker </h3>
 <p>
 					Runs a MapReduce task Tracker node.

docs/hadoop-default.html

@@ -11,6 +11,14 @@
 <td><a name="hadoop.native.lib">hadoop.native.lib</a></td><td>true</td><td>Should native hadoop libraries, if present, be used.</td>
 </tr>
 <tr>
+<td><a name="hadoop.http.filter.initializers">hadoop.http.filter.initializers</a></td><td></td><td>A comma separated list of class names.  Each class in the list must extend org.apache.hadoop.http.FilterInitializer.
+  
+  The corresponding Filter will be initialized.  Then, the Filter will be applied to all user facing jsp and servlet web pages.  The ordering of the list defines the ordering of the filters.
+
+  The value can be empty.
+  </td>
+</tr>
+<tr>
 <td><a name="hadoop.logfile.size">hadoop.logfile.size</a></td><td>10000000</td><td>The max size of each log file</td>
 </tr>
 <tr>
@@ -76,7 +84,7 @@ creations/deletions), or "all".</td>
   facilitate opening large map files using less memory.</td>
 </tr>
 <tr>
-<td><a name="io.compression.codecs">io.compression.codecs</a></td><td>org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.LzopCodec</td><td>A list of the compression codec classes that can be used 
+<td><a name="io.compression.codecs">io.compression.codecs</a></td><td>org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.LzopCodec,org.apache.hadoop.io.compress.BZip2Codec</td><td>A list of the compression codec classes that can be used 
                for compression/decompression.</td>
 </tr>
 <tr>
@@ -239,11 +247,6 @@ creations/deletions), or "all".</td>
 <td><a name="dfs.permissions.supergroup">dfs.permissions.supergroup</a></td><td>supergroup</td><td>The name of the group of super-users.</td>
 </tr>
 <tr>
-<td><a name="dfs.client.buffer.dir">dfs.client.buffer.dir</a></td><td>${hadoop.tmp.dir}/dfs/tmp</td><td>Determines where on the local filesystem an DFS client
-  should store its blocks before it sends them to the datanode.
-  </td>
-</tr>
-<tr>
 <td><a name="dfs.data.dir">dfs.data.dir</a></td><td>${hadoop.tmp.dir}/dfs/data</td><td>Determines where on the local filesystem an DFS data node
   should store its blocks.  If this is a comma-delimited
   list of directories, then data will be stored in all named
@@ -416,6 +419,10 @@ creations/deletions), or "all".</td>
   </td>
 </tr>
 <tr>
+<td><a name="mapred.tasktracker.instrumentation">mapred.tasktracker.instrumentation</a></td><td>org.apache.hadoop.mapred.TaskTrackerMetricsInst</td><td>Expert: The instrumentation class to associate with each TaskTracker.
+  </td>
+</tr>
+<tr>
 <td><a name="mapred.map.tasks">mapred.map.tasks</a></td><td>2</td><td>The default number of map tasks per job.  Typically set
   to a prime several times greater than number of available hosts.
   Ignored when mapred.job.tracker is "local".  
@@ -428,6 +435,14 @@ creations/deletions), or "all".</td>
   </td>
 </tr>
 <tr>
+<td><a name="mapred.jobtracker.taskScheduler">mapred.jobtracker.taskScheduler</a></td><td>org.apache.hadoop.mapred.JobQueueTaskScheduler</td><td>The class responsible for scheduling the tasks.</td>
+</tr>
+<tr>
+<td><a name="mapred.jobtracker.taskScheduler.maxRunningTasksPerJob">mapred.jobtracker.taskScheduler.maxRunningTasksPerJob</a></td><td></td><td>The maximum number of running tasks for a job before
+  it gets preempted. No limits if undefined.
+  </td>
+</tr>
+<tr>
 <td><a name="mapred.map.max.attempts">mapred.map.max.attempts</a></td><td>4</td><td>Expert: The maximum number of attempts per map task.
   In other words, framework will try to execute a map task these many number
   of times before giving up on it.
@@ -470,6 +485,10 @@ creations/deletions), or "all".</td>
   </td>
 </tr>
 <tr>
+<td><a name="mapred.jobtracker.instrumentation">mapred.jobtracker.instrumentation</a></td><td>org.apache.hadoop.mapred.JobTrackerMetricsInst</td><td>Expert: The instrumentation class to associate with each JobTracker.
+  </td>
+</tr>
+<tr>
 <td><a name="mapred.child.java.opts">mapred.child.java.opts</a></td><td>-Xmx200m</td><td>Java opts for the task tracker child processes.  
   The following symbol, if present, will be interpolated: @taskid@ is replaced 
   by current TaskID. Any other occurrences of '@' will go unchanged.
@@ -752,7 +771,7 @@ creations/deletions), or "all".</td>
   </td>
 </tr>
 <tr>
-<td><a name="topology.script.number.args">topology.script.number.args</a></td><td>20</td><td> The max number of args that the script configured with 
+<td><a name="topology.script.number.args">topology.script.number.args</a></td><td>100</td><td> The max number of args that the script configured with 
     topology.script.file.name should be run with. Each arg is an
     IP address.
   </td>
@@ -763,6 +782,31 @@ creations/deletions), or "all".</td>
     level.
   </td>
 </tr>
+<tr>
+<td><a name="mapred.tasktracker.tasks.maxmemory">mapred.tasktracker.tasks.maxmemory</a></td><td>-1</td><td> The maximum amount of virtual memory all tasks running on a
+    tasktracker, including sub-processes they launch, can use. This value is
+    used to compute the amount of free memory available for tasks. Any task
+    scheduled on this tasktracker is guaranteed and constrained to use a 
+    share of this amount. Any task exceeding its share will be killed. 
+    If set to -1, this functionality is disabled, and mapred.task.maxmemory
+    is ignored.
+  </td>
+</tr>
+<tr>
+<td><a name="mapred.task.maxmemory">mapred.task.maxmemory</a></td><td>-1</td><td> The maximum amount of memory any task of a job will use.
+    A task of this job will be scheduled on a tasktracker, only if the 
+    amount of free memory on the tasktracker is greater than or 
+    equal to this value. If set to -1, tasks are assured a memory limit on
+    the tasktracker equal to 
+    mapred.tasktracker.tasks.maxmemory/number of slots. If the value of 
+    mapred.tasktracker.tasks.maxmemory is set to -1, this value is ignored.
+    
+    Note: If mapred.child.java.opts is specified with an Xmx value, or if 
+    mapred.child.ulimit is specified, then the value of mapred.task.maxmemory
+    must be set to a higher value than these. If not, the task might be 
+    killed even though these limits are not reached.
+  </td>
+</tr>
 </table>
 </body>
 </html>

docs/hdfs_design.html

@@ -588,12 +588,12 @@ document.write("Last Published: " + document.lastModified);
           
 <tr>
             
-<td colspan="1" rowspan="1"> Decommission DataNode <span class="codefrag">datanodename</span> </td><td colspan="1" rowspan="1"> <span class="codefrag">bin/hadoop dfsadmin -decommission datanodename</span> </td>
+<td colspan="1" rowspan="1"> Recommission or decommission DataNode(s) </td><td colspan="1" rowspan="1"> <span class="codefrag">bin/hadoop dfsadmin -refreshNodes</span> </td>
           
 </tr>
         
 </table>
-<a name="N10219"></a><a name="Browser+Interface"></a>
+<a name="N10216"></a><a name="Browser+Interface"></a>
 <h3 class="h4"> Browser Interface </h3>
 <p>
         A typical HDFS install configures a web server to expose the HDFS namespace through a configurable TCP port. This allows a user to navigate the HDFS namespace and view the contents of its files using a web browser.
@@ -601,10 +601,10 @@ document.write("Last Published: " + document.lastModified);
 </div> 
 
     
-<a name="N10224"></a><a name="Space+Reclamation"></a>
+<a name="N10221"></a><a name="Space+Reclamation"></a>
 <h2 class="h3"> Space Reclamation </h2>
 <div class="section">
-<a name="N1022A"></a><a name="File+Deletes+and+Undeletes"></a>
+<a name="N10227"></a><a name="File+Deletes+and+Undeletes"></a>
 <h3 class="h4"> File Deletes and Undeletes </h3>
 <p>
         When a file is deleted by a user or an application, it is not immediately removed from HDFS.  Instead, HDFS first renames it to a file in the <span class="codefrag">/trash</span> directory. The file can be restored quickly as long as it remains in <span class="codefrag">/trash</span>. A file remains in <span class="codefrag">/trash</span> for a configurable amount of time. After the expiry of its life in <span class="codefrag">/trash</span>, the NameNode deletes the file from the HDFS namespace. The deletion of a file causes the blocks associated with the file to be freed. Note that there could be an appreciable time delay between the time a file is deleted by a user and the time of the corresponding increase in free space in HDFS.
@@ -612,7 +612,7 @@ document.write("Last Published: " + document.lastModified);
 <p>
         A user can Undelete a file after deleting it as long as it remains in the <span class="codefrag">/trash</span> directory. If a user wants to undelete a file that he/she has deleted, he/she can navigate the <span class="codefrag">/trash</span> directory and retrieve the file. The <span class="codefrag">/trash</span> directory contains only the latest copy of the file that was deleted. The <span class="codefrag">/trash</span> directory is just like any other directory with one special feature: HDFS applies specified policies to automatically delete files from this directory. The current default policy is to delete files from <span class="codefrag">/trash</span> that are more than 6 hours old. In the future, this policy will be configurable through a well defined interface.
         </p>
-<a name="N10252"></a><a name="Decrease+Replication+Factor"></a>
+<a name="N1024F"></a><a name="Decrease+Replication+Factor"></a>
 <h3 class="h4"> Decrease Replication Factor </h3>
 <p>
         When the replication factor of a file is reduced, the NameNode selects excess replicas that can be deleted. The next Heartbeat transfers this information to the DataNode. The DataNode then removes the corresponding blocks and the corresponding free space appears in the cluster. Once again, there might be a time delay between the completion of the <span class="codefrag">setReplication</span> API call and the appearance of free space in the cluster.
@@ -621,7 +621,7 @@ document.write("Last Published: " + document.lastModified);
 
 
     
-<a name="N10260"></a><a name="References"></a>
+<a name="N1025D"></a><a name="References"></a>
 <h2 class="h3"> References </h2>
 <div class="section">
 <p>

docs/hdfs_user_guide.html

@@ -446,13 +446,26 @@ document.write("Last Published: " + document.lastModified);
 <span class="codefrag">-finalizeUpgrade</span>
    		: removes previous backup of the cluster made during last upgrade.
    	</li>
+    
+<li>
+      
+<span class="codefrag">-refreshNodes</span>
+      : Updates the set of hosts allowed to connect to namenode.
+      Re-reads the config file to update values defined by dfs.hosts and 
+      dfs.host.exclude and reads the entires (hostnames) in those files. 
+      Each entry not defined in dfs.hosts but in dfs.hosts.exclude 
+      is decommissioned. Each entry defined in dfs.hosts and also in 
+      dfs.host.exclude is stopped from decommissioning if it has aleady 
+      been marked for decommission. Entires not present in both the lists 
+      are decommissioned. 
+    </li>
    	
 </ul>
 <p>
    	  For command usage, see <a href="commands_manual.html#dfsadmin">dfsadmin command</a>.
    	</p>
 </div> 
-<a name="N100D2"></a><a name="Secondary+NameNode"></a>
+<a name="N100D8"></a><a name="Secondary+NameNode"></a>
 <h2 class="h3"> Secondary NameNode </h2>
 <div class="section">
 <p>
@@ -537,7 +550,7 @@ document.write("Last Published: " + document.lastModified);
      For command usage, see <a href="commands_manual.html#secondarynamenode"><span class="codefrag">secondarynamenode</span> command</a>.
    </p>
 </div> 
-<a name="N10139"></a><a name="Rebalancer"></a>
+<a name="N1013F"></a><a name="Rebalancer"></a>
 <h2 class="h3"> Rebalancer </h2>
 <div class="section">
 <p>
@@ -585,7 +598,7 @@ document.write("Last Published: " + document.lastModified);
      For command usage, see <a href="commands_manual.html#balancer">balancer command</a>.
    </p>
 </div> 
-<a name="N10164"></a><a name="Rack+Awareness"></a>
+<a name="N1016A"></a><a name="Rack+Awareness"></a>
 <h2 class="h3"> Rack Awareness </h2>
 <div class="section">
 <p>
@@ -604,7 +617,7 @@ document.write("Last Published: " + document.lastModified);
       <a href="http://issues.apache.org/jira/browse/HADOOP-692">HADOOP-692</a>.
     </p>
 </div> 
-<a name="N10179"></a><a name="Safemode"></a>
+<a name="N1017F"></a><a name="Safemode"></a>
 <h2 class="h3"> Safemode </h2>
 <div class="section">
 <p>
@@ -625,7 +638,7 @@ document.write("Last Published: " + document.lastModified);
       <a href="http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/dfs/NameNode.html#setSafeMode(org.apache.hadoop.dfs.FSConstants.SafeModeAction)"><span class="codefrag">setSafeMode()</span></a>.
     </p>
 </div> 
-<a name="N1018B"></a><a name="fsck"></a>
+<a name="N10191"></a><a name="fsck"></a>
 <h2 class="h3"> fsck </h2>
 <div class="section">
 <p>    
@@ -643,7 +656,7 @@ document.write("Last Published: " + document.lastModified);
       <span class="codefrag">fsck</span> can be run on the whole file system or on a subset of files.
      </p>
 </div> 
-<a name="N101AD"></a><a name="Upgrade+and+Rollback"></a>
+<a name="N101B3"></a><a name="Upgrade+and+Rollback"></a>
 <h2 class="h3"> Upgrade and Rollback </h2>
 <div class="section">
 <p>
@@ -702,7 +715,7 @@ document.write("Last Published: " + document.lastModified);
       
 </ul>
 </div> 
-<a name="N101EB"></a><a name="File+Permissions+and+Security"></a>
+<a name="N101F1"></a><a name="File+Permissions+and+Security"></a>
 <h2 class="h3"> File Permissions and Security </h2>
 <div class="section">
 <p>           
@@ -715,7 +728,7 @@ document.write("Last Published: " + document.lastModified);
       <a href="hdfs_permissions_guide.html">Permissions User and Administrator Guide</a>.
      </p>
 </div> 
-<a name="N101F9"></a><a name="Scalability"></a>
+<a name="N101FF"></a><a name="Scalability"></a>
 <h2 class="h3"> Scalability </h2>
 <div class="section">
 <p>
@@ -733,7 +746,7 @@ document.write("Last Published: " + document.lastModified);
       suggested configuration improvements for large Hadoop clusters.
      </p>
 </div> 
-<a name="N1020B"></a><a name="Related+Documentation"></a>
+<a name="N10211"></a><a name="Related+Documentation"></a>
 <h2 class="h3"> Related Documentation </h2>
 <div class="section">
 <p>

docs/mapred_tutorial.html

@@ -310,7 +310,7 @@ document.write("Last Published: " + document.lastModified);
 <a href="#Example%3A+WordCount+v2.0">Example: WordCount v2.0</a>
 <ul class="minitoc">
 <li>
-<a href="#Source+Code-N10DD5">Source Code</a>
+<a href="#Source+Code-N10E0A">Source Code</a>
 </li>
 <li>
 <a href="#Sample+Runs">Sample Runs</a>
@@ -1116,7 +1116,24 @@ document.write("Last Published: " + document.lastModified);
 <br>
         
 </p>
-<a name="N104EC"></a><a name="Walk-through"></a>
+<p> Applications can specify a comma separated list of paths which
+        would be present in the current working directory of the task 
+        using the option <span class="codefrag">-files</span>. The <span class="codefrag">-libjars</span>
+        option allows applications to add jars to the classpaths of the maps
+        and reduces. The <span class="codefrag">-archives</span> allows them to pass archives
+        as arguments that are unzipped/unjarred and a link with name of the
+        jar/zip are created in the current working directory of tasks. More
+        details about the command line options are available at 
+        <a href="commands_manual.html">Commands manual</a>
+</p>
+<p>Running <span class="codefrag">wordcount</span> example with 
+        <span class="codefrag">-libjars</span> and <span class="codefrag">-files</span>:<br>
+        
+<span class="codefrag"> hadoop jar hadoop-examples.jar wordcount -files cachefile.txt 
+        -libjars mylib.jar input output </span> 
+        
+</p>
+<a name="N1050C"></a><a name="Walk-through"></a>
 <h3 class="h4">Walk-through</h3>
 <p>The <span class="codefrag">WordCount</span> application is quite straight-forward.</p>
 <p>The <span class="codefrag">Mapper</span> implementation (lines 14-26), via the 
@@ -1226,7 +1243,7 @@ document.write("Last Published: " + document.lastModified);
 </div>
     
     
-<a name="N105A3"></a><a name="Map%2FReduce+-+User+Interfaces"></a>
+<a name="N105C3"></a><a name="Map%2FReduce+-+User+Interfaces"></a>
 <h2 class="h3">Map/Reduce - User Interfaces</h2>
 <div class="section">
 <p>This section provides a reasonable amount of detail on every user-facing 
@@ -1245,12 +1262,12 @@ document.write("Last Published: " + document.lastModified);
 <p>Finally, we will wrap up by discussing some useful features of the
       framework such as the <span class="codefrag">DistributedCache</span>, 
       <span class="codefrag">IsolationRunner</span> etc.</p>
-<a name="N105DC"></a><a name="Payload"></a>
+<a name="N105FC"></a><a name="Payload"></a>
 <h3 class="h4">Payload</h3>
 <p>Applications typically implement the <span class="codefrag">Mapper</span> and 
         <span class="codefrag">Reducer</span> interfaces to provide the <span class="codefrag">map</span> and 
         <span class="codefrag">reduce</span> methods. These form the core of the job.</p>
-<a name="N105F1"></a><a name="Mapper"></a>
+<a name="N10611"></a><a name="Mapper"></a>
 <h4>Mapper</h4>
 <p>
 <a href="api/org/apache/hadoop/mapred/Mapper.html">
@@ -1306,7 +1323,7 @@ document.write("Last Published: " + document.lastModified);
           <a href="api/org/apache/hadoop/io/compress/CompressionCodec.html">
           CompressionCodec</a> to be used via the <span class="codefrag">JobConf</span>.
           </p>
-<a name="N10667"></a><a name="How+Many+Maps%3F"></a>
+<a name="N10687"></a><a name="How+Many+Maps%3F"></a>
 <h5>How Many Maps?</h5>
 <p>The number of maps is usually driven by the total size of the 
             inputs, that is, the total number of blocks of the input files.</p>
@@ -1319,7 +1336,7 @@ document.write("Last Published: " + document.lastModified);
             <a href="api/org/apache/hadoop/mapred/JobConf.html#setNumMapTasks(int)">
             setNumMapTasks(int)</a> (which only provides a hint to the framework) 
             is used to set it even higher.</p>
-<a name="N1067F"></a><a name="Reducer"></a>
+<a name="N1069F"></a><a name="Reducer"></a>
 <h4>Reducer</h4>
 <p>
 <a href="api/org/apache/hadoop/mapred/Reducer.html">
@@ -1342,18 +1359,18 @@ document.write("Last Published: " + document.lastModified);
 <p>
 <span class="codefrag">Reducer</span> has 3 primary phases: shuffle, sort and reduce.
           </p>
-<a name="N106AF"></a><a name="Shuffle"></a>
+<a name="N106CF"></a><a name="Shuffle"></a>
 <h5>Shuffle</h5>
 <p>Input to the <span class="codefrag">Reducer</span> is the sorted output of the
             mappers. In this phase the framework fetches the relevant partition 
             of the output of all the mappers, via HTTP.</p>
-<a name="N106BC"></a><a name="Sort"></a>
+<a name="N106DC"></a><a name="Sort"></a>
 <h5>Sort</h5>
 <p>The framework groups <span class="codefrag">Reducer</span> inputs by keys (since 
             different mappers may have output the same key) in this stage.</p>
 <p>The shuffle and sort phases occur simultaneously; while 
             map-outputs are being fetched they are merged.</p>
-<a name="N106CB"></a><a name="Secondary+Sort"></a>
+<a name="N106EB"></a><a name="Secondary+Sort"></a>
 <h5>Secondary Sort</h5>
 <p>If equivalence rules for grouping the intermediate keys are 
               required to be different from those for grouping keys before 
@@ -1364,7 +1381,7 @@ document.write("Last Published: " + document.lastModified);
               JobConf.setOutputKeyComparatorClass(Class)</a> can be used to 
               control how intermediate keys are grouped, these can be used in 
               conjunction to simulate <em>secondary sort on values</em>.</p>
-<a name="N106E4"></a><a name="Reduce"></a>
+<a name="N10704"></a><a name="Reduce"></a>
 <h5>Reduce</h5>
 <p>In this phase the 
             <a href="api/org/apache/hadoop/mapred/Reducer.html#reduce(K2, java.util.Iterator, org.apache.hadoop.mapred.OutputCollector, org.apache.hadoop.mapred.Reporter)">
@@ -1380,7 +1397,7 @@ document.write("Last Published: " + document.lastModified);
             progress, set application-level status messages and update 
             <span class="codefrag">Counters</span>, or just indicate that they are alive.</p>
 <p>The output of the <span class="codefrag">Reducer</span> is <em>not sorted</em>.</p>
-<a name="N10712"></a><a name="How+Many+Reduces%3F"></a>
+<a name="N10732"></a><a name="How+Many+Reduces%3F"></a>
 <h5>How Many Reduces?</h5>
 <p>The right number of reduces seems to be <span class="codefrag">0.95</span> or 
             <span class="codefrag">1.75</span> multiplied by (&lt;<em>no. of nodes</em>&gt; * 
@@ -1395,7 +1412,7 @@ document.write("Last Published: " + document.lastModified);
 <p>The scaling factors above are slightly less than whole numbers to 
             reserve a few reduce slots in the framework for speculative-tasks and
             failed tasks.</p>
-<a name="N10737"></a><a name="Reducer+NONE"></a>
+<a name="N10757"></a><a name="Reducer+NONE"></a>
 <h5>Reducer NONE</h5>
 <p>It is legal to set the number of reduce-tasks to <em>zero</em> if 
             no reduction is desired.</p>
@@ -1405,7 +1422,7 @@ document.write("Last Published: " + document.lastModified);
             setOutputPath(Path)</a>. The framework does not sort the 
             map-outputs before writing them out to the <span class="codefrag">FileSystem</span>.
             </p>
-<a name="N10752"></a><a name="Partitioner"></a>
+<a name="N10772"></a><a name="Partitioner"></a>
 <h4>Partitioner</h4>
 <p>
 <a href="api/org/apache/hadoop/mapred/Partitioner.html">
@@ -1419,7 +1436,7 @@ document.write("Last Published: " + document.lastModified);
 <p>
 <a href="api/org/apache/hadoop/mapred/lib/HashPartitioner.html">
           HashPartitioner</a> is the default <span class="codefrag">Partitioner</span>.</p>
-<a name="N10771"></a><a name="Reporter"></a>
+<a name="N10791"></a><a name="Reporter"></a>
 <h4>Reporter</h4>
 <p>
 <a href="api/org/apache/hadoop/mapred/Reporter.html">
@@ -1438,7 +1455,7 @@ document.write("Last Published: " + document.lastModified);
           </p>
 <p>Applications can also update <span class="codefrag">Counters</span> using the 
           <span class="codefrag">Reporter</span>.</p>
-<a name="N1079B"></a><a name="OutputCollector"></a>
+<a name="N107BB"></a><a name="OutputCollector"></a>
 <h4>OutputCollector</h4>
 <p>
 <a href="api/org/apache/hadoop/mapred/OutputCollector.html">
@@ -1449,7 +1466,7 @@ document.write("Last Published: " + document.lastModified);
 <p>Hadoop Map/Reduce comes bundled with a 
         <a href="api/org/apache/hadoop/mapred/lib/package-summary.html">
         library</a> of generally useful mappers, reducers, and partitioners.</p>
-<a name="N107B6"></a><a name="Job+Configuration"></a>
+<a name="N107D6"></a><a name="Job+Configuration"></a>
 <h3 class="h4">Job Configuration</h3>
 <p>
 <a href="api/org/apache/hadoop/mapred/JobConf.html">
@@ -1507,7 +1524,7 @@ document.write("Last Published: " + document.lastModified);
         <a href="api/org/apache/hadoop/conf/Configuration.html#set(java.lang.String, java.lang.String)">set(String, String)</a>/<a href="api/org/apache/hadoop/conf/Configuration.html#get(java.lang.String, java.lang.String)">get(String, String)</a>
         to set/get arbitrary parameters needed by applications. However, use the 
         <span class="codefrag">DistributedCache</span> for large amounts of (read-only) data.</p>
-<a name="N10848"></a><a name="Task+Execution+%26+Environment"></a>
+<a name="N10868"></a><a name="Task+Execution+%26+Environment"></a>
 <h3 class="h4">Task Execution &amp; Environment</h3>
 <p>The <span class="codefrag">TaskTracker</span> executes the <span class="codefrag">Mapper</span>/ 
         <span class="codefrag">Reducer</span>  <em>task</em> as a child process in a separate jvm.
@@ -1550,7 +1567,9 @@ document.write("Last Published: " + document.lastModified);
         
 </p>
 <p>Users/admins can also specify the maximum virtual memory 
-        of the launched child-task using <span class="codefrag">mapred.child.ulimit</span>.
+        of the launched child-task, and any sub-process it launches 
+        recursively, using <span class="codefrag">mapred.child.ulimit</span>. Note that
+        the value set here is a per process limit.
         The value for <span class="codefrag">mapred.child.ulimit</span> should be specified 
         in kilo bytes (KB). And also the value must be greater than
         or equal to the -Xmx passed to JavaVM, else the VM might not start. 
@@ -1561,6 +1580,26 @@ document.write("Last Published: " + document.lastModified);
         <a href="cluster_setup.html#Configuring+the+Environment+of+the+Hadoop+Daemons">
         cluster_setup.html </a>
 </p>
+<p>There are two additional parameters that influence virtual memory
+        limits for tasks run on a tasktracker. The parameter 
+        <span class="codefrag">mapred.tasktracker.maxmemory</span> is set by admins
+        to limit the total memory all tasks that it runs can use together. 
+        Setting this enables the parameter <span class="codefrag">mapred.task.maxmemory</span>
+        that can be used to specify the maximum virtual memory the entire 
+        process tree starting from the launched child-task requires. 
+        This is a cumulative limit of all processes in the process tree. 
+        By specifying this value, users can be assured that the system will 
+        run their tasks only on tasktrackers that have atleast this amount 
+        of free memory available. If at any time during task execution, this 
+        limit is exceeded, the task would be killed by the system. By default, 
+        any task would get a share of 
+        <span class="codefrag">mapred.tasktracker.maxmemory</span>, divided
+        equally among the number of slots. The user can thus verify if the
+        tasks need more memory than this, and specify it in 
+        <span class="codefrag">mapred.task.maxmemory</span>. Specifically, this value must be 
+        greater than any value specified for a maximum heap-size
+        of the child jvm via <span class="codefrag">mapred.child.java.opts</span>, or a ulimit
+        value in <span class="codefrag">mapred.child.ulimit</span>. </p>
 <p>The task tracker has local directory,
         <span class="codefrag"> ${mapred.local.dir}/taskTracker/</span> to create localized
         cache and localized job. It can define multiple local directories 
@@ -1739,7 +1778,7 @@ document.write("Last Published: " + document.lastModified);
         <a href="native_libraries.html#Loading+native+libraries+through+DistributedCache">
         native_libraries.html</a>
 </p>
-<a name="N109E8"></a><a name="Job+Submission+and+Monitoring"></a>
+<a name="N10A1D"></a><a name="Job+Submission+and+Monitoring"></a>
 <h3 class="h4">Job Submission and Monitoring</h3>
 <p>
 <a href="api/org/apache/hadoop/mapred/JobClient.html">
@@ -1800,7 +1839,7 @@ document.write("Last Published: " + document.lastModified);
 <p>Normally the user creates the application, describes various facets 
         of the job via <span class="codefrag">JobConf</span>, and then uses the 
         <span class="codefrag">JobClient</span> to submit the job and monitor its progress.</p>
-<a name="N10A48"></a><a name="Job+Control"></a>
+<a name="N10A7D"></a><a name="Job+Control"></a>
 <h4>Job Control</h4>
 <p>Users may need to chain Map/Reduce jobs to accomplish complex
           tasks which cannot be done via a single Map/Reduce job. This is fairly
@@ -1836,7 +1875,7 @@ document.write("Last Published: " + document.lastModified);
             </li>
           
 </ul>
-<a name="N10A72"></a><a name="Job+Input"></a>
+<a name="N10AA7"></a><a name="Job+Input"></a>
 <h3 class="h4">Job Input</h3>
 <p>
 <a href="api/org/apache/hadoop/mapred/InputFormat.html">
@@ -1884,7 +1923,7 @@ document.write("Last Published: " + document.lastModified);
         appropriate <span class="codefrag">CompressionCodec</span>. However, it must be noted that
         compressed files with the above extensions cannot be <em>split</em> and 
         each compressed file is processed in its entirety by a single mapper.</p>
-<a name="N10ADC"></a><a name="InputSplit"></a>
+<a name="N10B11"></a><a name="InputSplit"></a>
 <h4>InputSplit</h4>
 <p>
 <a href="api/org/apache/hadoop/mapred/InputSplit.html">
@@ -1898,7 +1937,7 @@ document.write("Last Published: " + document.lastModified);
           FileSplit</a> is the default <span class="codefrag">InputSplit</span>. It sets 
           <span class="codefrag">map.input.file</span> to the path of the input file for the
           logical split.</p>
-<a name="N10B01"></a><a name="RecordReader"></a>
+<a name="N10B36"></a><a name="RecordReader"></a>
 <h4>RecordReader</h4>
 <p>
 <a href="api/org/apache/hadoop/mapred/RecordReader.html">
@@ -1910,7 +1949,7 @@ document.write("Last Published: " + document.lastModified);
           for processing. <span class="codefrag">RecordReader</span> thus assumes the 
           responsibility of processing record boundaries and presents the tasks 
           with keys and values.</p>
-<a name="N10B24"></a><a name="Job+Output"></a>
+<a name="N10B59"></a><a name="Job+Output"></a>
 <h3 class="h4">Job Output</h3>
 <p>
 <a href="api/org/apache/hadoop/mapred/OutputFormat.html">
@@ -1935,7 +1974,7 @@ document.write("Last Published: " + document.lastModified);
 <p>
 <span class="codefrag">TextOutputFormat</span> is the default 
         <span class="codefrag">OutputFormat</span>.</p>
-<a name="N10B4D"></a><a name="Task+Side-Effect+Files"></a>
+<a name="N10B82"></a><a name="Task+Side-Effect+Files"></a>
 <h4>Task Side-Effect Files</h4>
 <p>In some applications, component tasks need to create and/or write to
           side-files, which differ from the actual job-output files.</p>
@@ -1974,7 +2013,7 @@ document.write("Last Published: " + document.lastModified);
 <p>The entire discussion holds true for maps of jobs with 
            reducer=NONE (i.e. 0 reduces) since output of the map, in that case, 
            goes directly to HDFS.</p>
-<a name="N10B95"></a><a name="RecordWriter"></a>
+<a name="N10BCA"></a><a name="RecordWriter"></a>
 <h4>RecordWriter</h4>
 <p>
 <a href="api/org/apache/hadoop/mapred/RecordWriter.html">
@@ -1982,9 +2021,9 @@ document.write("Last Published: " + document.lastModified);
           pairs to an output file.</p>
 <p>RecordWriter implementations write the job outputs to the 
           <span class="codefrag">FileSystem</span>.</p>
-<a name="N10BAC"></a><a name="Other+Useful+Features"></a>
+<a name="N10BE1"></a><a name="Other+Useful+Features"></a>
 <h3 class="h4">Other Useful Features</h3>
-<a name="N10BB2"></a><a name="Counters"></a>
+<a name="N10BE7"></a><a name="Counters"></a>
 <h4>Counters</h4>
 <p>
 <span class="codefrag">Counters</span> represent global counters, defined either by 
@@ -2001,7 +2040,7 @@ document.write("Last Published: " + document.lastModified);
           in the <span class="codefrag">map</span> and/or 
           <span class="codefrag">reduce</span> methods. These counters are then globally 
           aggregated by the framework.</p>
-<a name="N10BE1"></a><a name="DistributedCache"></a>
+<a name="N10C16"></a><a name="DistributedCache"></a>
 <h4>DistributedCache</h4>
 <p>
 <a href="api/org/apache/hadoop/filecache/DistributedCache.html">
@@ -2072,7 +2111,7 @@ document.write("Last Published: " + document.lastModified);
           <span class="codefrag">mapred.job.classpath.{files|archives}</span>. Similarly the
           cached files that are symlinked into the working directory of the
           task can be used to distribute native libraries and load them.</p>
-<a name="N10C64"></a><a name="Tool"></a>
+<a name="N10C99"></a><a name="Tool"></a>
 <h4>Tool</h4>
 <p>The <a href="api/org/apache/hadoop/util/Tool.html">Tool</a> 
           interface supports the handling of generic Hadoop command-line options.
@@ -2112,7 +2151,7 @@ document.write("Last Published: " + document.lastModified);
             </span>
           
 </p>
-<a name="N10C96"></a><a name="IsolationRunner"></a>
+<a name="N10CCB"></a><a name="IsolationRunner"></a>
 <h4>IsolationRunner</h4>
 <p>
 <a href="api/org/apache/hadoop/mapred/IsolationRunner.html">
@@ -2136,7 +2175,7 @@ document.write("Last Published: " + document.lastModified);
 <p>
 <span class="codefrag">IsolationRunner</span> will run the failed task in a single 
           jvm, which can be in the debugger, over precisely the same input.</p>
-<a name="N10CC9"></a><a name="Profiling"></a>
+<a name="N10CFE"></a><a name="Profiling"></a>
 <h4>Profiling</h4>
 <p>Profiling is a utility to get a representative (2 or 3) sample
           of built-in java profiler for a sample of maps and reduces. </p>
@@ -2169,7 +2208,7 @@ document.write("Last Published: " + document.lastModified);
           <span class="codefrag">-agentlib:hprof=cpu=samples,heap=sites,force=n,thread=y,verbose=n,file=%s</span>
           
 </p>
-<a name="N10CFD"></a><a name="Debugging"></a>
+<a name="N10D32"></a><a name="Debugging"></a>
 <h4>Debugging</h4>
 <p>Map/Reduce framework provides a facility to run user-provided 
           scripts for debugging. When map/reduce task fails, user can run 
@@ -2180,14 +2219,14 @@ document.write("Last Published: " + document.lastModified);
 <p> In the following sections we discuss how to submit debug script
           along with the job. For submitting debug script, first it has to
           distributed. Then the script has to supplied in Configuration. </p>
-<a name="N10D09"></a><a name="How+to+distribute+script+file%3A"></a>
+<a name="N10D3E"></a><a name="How+to+distribute+script+file%3A"></a>
 <h5> How to distribute script file: </h5>
 <p>
           The user has to use 
           <a href="mapred_tutorial.html#DistributedCache">DistributedCache</a>
           mechanism to <em>distribute</em> and <em>symlink</em> the
           debug script file.</p>
-<a name="N10D1D"></a><a name="How+to+submit+script%3A"></a>
+<a name="N10D52"></a><a name="How+to+submit+script%3A"></a>
 <h5> How to submit script: </h5>
 <p> A quick way to submit debug script is to set values for the 
           properties "mapred.map.task.debug.script" and 
@@ -2211,17 +2250,17 @@ document.write("Last Published: " + document.lastModified);
 <span class="codefrag">$script $stdout $stderr $syslog $jobconf $program </span>  
           
 </p>
-<a name="N10D3F"></a><a name="Default+Behavior%3A"></a>
+<a name="N10D74"></a><a name="Default+Behavior%3A"></a>
 <h5> Default Behavior: </h5>
 <p> For pipes, a default script is run to process core dumps under
           gdb, prints stack trace and gives info about running threads. </p>
-<a name="N10D4A"></a><a name="JobControl"></a>
+<a name="N10D7F"></a><a name="JobControl"></a>
 <h4>JobControl</h4>
 <p>
 <a href="api/org/apache/hadoop/mapred/jobcontrol/package-summary.html">
           JobControl</a> is a utility which encapsulates a set of Map/Reduce jobs
           and their dependencies.</p>
-<a name="N10D57"></a><a name="Data+Compression"></a>
+<a name="N10D8C"></a><a name="Data+Compression"></a>
 <h4>Data Compression</h4>
 <p>Hadoop Map/Reduce provides facilities for the application-writer to
           specify compression for both intermediate map-outputs and the
@@ -2235,7 +2274,7 @@ document.write("Last Published: " + document.lastModified);
           codecs for reasons of both performance (zlib) and non-availability of
           Java libraries (lzo). More details on their usage and availability are
           available <a href="native_libraries.html">here</a>.</p>
-<a name="N10D77"></a><a name="Intermediate+Outputs"></a>
+<a name="N10DAC"></a><a name="Intermediate+Outputs"></a>
 <h5>Intermediate Outputs</h5>
 <p>Applications can control compression of intermediate map-outputs
             via the 
@@ -2244,7 +2283,7 @@ document.write("Last Published: " + document.lastModified);
             <span class="codefrag">CompressionCodec</span> to be used via the
             <a href="api/org/apache/hadoop/mapred/JobConf.html#setMapOutputCompressorClass(java.lang.Class)">
             JobConf.setMapOutputCompressorClass(Class)</a> api.</p>
-<a name="N10D8C"></a><a name="Job+Outputs"></a>
+<a name="N10DC1"></a><a name="Job+Outputs"></a>
 <h5>Job Outputs</h5>
 <p>Applications can control compression of job-outputs via the
             <a href="api/org/apache/hadoop/mapred/FileOutputFormat.html#setCompressOutput(org.apache.hadoop.mapred.JobConf,%20boolean)">
@@ -2264,7 +2303,7 @@ document.write("Last Published: " + document.lastModified);
 </div>
 
     
-<a name="N10DBB"></a><a name="Example%3A+WordCount+v2.0"></a>
+<a name="N10DF0"></a><a name="Example%3A+WordCount+v2.0"></a>
 <h2 class="h3">Example: WordCount v2.0</h2>
 <div class="section">
 <p>Here is a more complete <span class="codefrag">WordCount</span> which uses many of the
@@ -2274,7 +2313,7 @@ document.write("Last Published: " + document.lastModified);
       <a href="quickstart.html#SingleNodeSetup">pseudo-distributed</a> or
       <a href="quickstart.html#Fully-Distributed+Operation">fully-distributed</a> 
       Hadoop installation.</p>
-<a name="N10DD5"></a><a name="Source+Code-N10DD5"></a>
+<a name="N10E0A"></a><a name="Source+Code-N10E0A"></a>
 <h3 class="h4">Source Code</h3>
 <table class="ForrestTable" cellspacing="1" cellpadding="4">
           
@@ -3484,7 +3523,7 @@ document.write("Last Published: " + document.lastModified);
 </tr>
         
 </table>
-<a name="N11537"></a><a name="Sample+Runs"></a>
+<a name="N1156C"></a><a name="Sample+Runs"></a>
 <h3 class="h4">Sample Runs</h3>
 <p>Sample text-files as input:</p>
 <p>
@@ -3652,7 +3691,7 @@ document.write("Last Published: " + document.lastModified);
 <br>
         
 </p>
-<a name="N1160B"></a><a name="Highlights"></a>
+<a name="N11640"></a><a name="Highlights"></a>
 <h3 class="h4">Highlights</h3>
 <p>The second version of <span class="codefrag">WordCount</span> improves upon the 
         previous one by using some features offered by the Map/Reduce framework:

src/docs/src/documentation/content/xdocs/mapred_tutorial.xml

@@ -1082,7 +1082,9 @@
         </p>
         
         <p>Users/admins can also specify the maximum virtual memory 
-        of the launched child-task using <code>mapred.child.ulimit</code>.
+        of the launched child-task, and any sub-process it launches 
+        recursively, using <code>mapred.child.ulimit</code>. Note that
+        the value set here is a per process limit.
         The value for <code>mapred.child.ulimit</code> should be specified 
         in kilo bytes (KB). And also the value must be greater than
         or equal to the -Xmx passed to JavaVM, else the VM might not start. 
@@ -1094,6 +1096,27 @@
         <a href="cluster_setup.html#Configuring+the+Environment+of+the+Hadoop+Daemons">
         cluster_setup.html </a></p>
         
+        <p>There are two additional parameters that influence virtual memory
+        limits for tasks run on a tasktracker. The parameter 
+        <code>mapred.tasktracker.maxmemory</code> is set by admins
+        to limit the total memory all tasks that it runs can use together. 
+        Setting this enables the parameter <code>mapred.task.maxmemory</code>
+        that can be used to specify the maximum virtual memory the entire 
+        process tree starting from the launched child-task requires. 
+        This is a cumulative limit of all processes in the process tree. 
+        By specifying this value, users can be assured that the system will 
+        run their tasks only on tasktrackers that have atleast this amount 
+        of free memory available. If at any time during task execution, this 
+        limit is exceeded, the task would be killed by the system. By default, 
+        any task would get a share of 
+        <code>mapred.tasktracker.maxmemory</code>, divided
+        equally among the number of slots. The user can thus verify if the
+        tasks need more memory than this, and specify it in 
+        <code>mapred.task.maxmemory</code>. Specifically, this value must be 
+        greater than any value specified for a maximum heap-size
+        of the child jvm via <code>mapred.child.java.opts</code>, or a ulimit
+        value in <code>mapred.child.ulimit</code>. </p>
+        
         <p>The task tracker has local directory,
         <code> ${mapred.local.dir}/taskTracker/</code> to create localized
         cache and localized job. It can define multiple local directories 

src/mapred/org/apache/hadoop/mapred/JobConf.java

@@ -104,6 +104,12 @@ public class JobConf extends Configuration {
   
   private static final Log LOG = LogFactory.getLog(JobConf.class);
 
+  /**
+   * A value which if set for memory related configuration options,
+   * indicates that the options are turned off.
+   */
+  public static final long DISABLED_VIRTUAL_MEMORY_LIMIT = -1L;
+  
   /**
    * Construct a map/reduce job configuration.
    */
@@ -1285,6 +1291,67 @@ public class JobConf extends Configuration {
     return get("job.local.dir");
   }
   
+  /**
+   * The maximum amount of virtual memory all tasks running on a
+   * tasktracker, including sub-processes they launch, can use.
+   *  
+   * This value is used to compute the amount of free memory 
+   * available for tasks. Any task scheduled on this tasktracker is 
+   * guaranteed and constrained to use a share of this amount. Any task 
+   * exceeding its share will be killed.
+   * 
+   * If set to {@link #DISABLED_VIRTUAL_MEMORY_LIMIT}, this functionality 
+   * is disabled.
+   * 
+   * @return maximum amount of virtual memory to divide among
+   * @see #getMaxVirtualMemoryForTask()
+   */
+  public long getMaxVirtualMemoryForTasks() {
+    return getLong("mapred.tasktracker.tasks.maxmemory", 
+                      DISABLED_VIRTUAL_MEMORY_LIMIT);
+  }
+  
+  /**
+   * Set the maximum amount of virtual memory all tasks running on a
+   * tasktracker, including sub-processes they launch, can use.
+   * 
+   * @param vmem maximum amount of virtual memory that can be used.
+   * @see #getMaxVirtualMemoryForTasks()
+   */
+  public void setMaxVirtualMemoryForTasks(long vmem) {
+    setLong("mapred.tasktracker.tasks.maxmemory", vmem);
+  }
+  
+  /**
+   * The maximum amount of memory any task of this job will use.
+   * 
+   * A task of this job will be scheduled on a tasktracker, only if the
+   * amount of free memory on the tasktracker is greater than 
+   * or equal to this value.
+   * 
+   * If set to {@link #DISABLED_VIRTUAL_MEMORY_LIMIT}, tasks are assured 
+   * a memory limit on the tasktracker equal to
+   * mapred.tasktracker.tasks.maxmemory/number of slots. If the value of
+   * mapred.tasktracker.tasks.maxmemory is set to -1, this value is 
+   * ignored.
+   * 
+   * @return The maximum amount of memory any task of this job will use.
+   * @see #getMaxVirtualMemoryForTasks()
+   */
+  public long getMaxVirtualMemoryForTask() {
+    return getLong("mapred.task.maxmemory", DISABLED_VIRTUAL_MEMORY_LIMIT);
+  }
+  
+  /**
+   * Set the maximum amount of memory any task of this job can use.
+   * 
+   * @param vmem Maximum amount of memory any task of this job can use.
+   * @see #getMaxVirtualMemoryForTask()
+   */
+  public void setMaxVirtualMemoryForTask(long vmem) {
+    setLong("mapred.task.maxmemory", vmem);
+  }
+    
   /** 
    * Find a jar that contains a class of the same name, if any.
    * It will return a jar file, even if that is not the first thing

src/mapred/org/apache/hadoop/mapred/JobInProgress.java

@@ -133,7 +133,8 @@ class JobInProgress {
   private boolean hasSpeculativeMaps;
   private boolean hasSpeculativeReduces;
   private long inputLength = 0;
-
+  private long maxVirtualMemoryForTask;
+  
   // Per-job counters
   public static enum Counter { 
     NUM_FAILED_MAPS, 
@@ -225,6 +226,7 @@ class JobInProgress {
     this.nonRunningReduces = new LinkedList<TaskInProgress>();    
     this.runningReduces = new LinkedHashSet<TaskInProgress>();
     this.resourceEstimator = new ResourceEstimator(this);
+    this.maxVirtualMemoryForTask = conf.getMaxVirtualMemoryForTask();
   }
 
   /**
@@ -441,6 +443,11 @@ class JobInProgress {
       this.priority = priority;
     }
   }
+
+  // Accessors for resources.
+  public long getMaxVirtualMemoryForTask() {
+    return maxVirtualMemoryForTask;
+  }
   
   long getInputLength() {
     return inputLength;
@@ -1089,9 +1096,10 @@ class JobInProgress {
     
 
     long outSize = resourceEstimator.getEstimatedMapOutputSize();
-    if(tts.getAvailableSpace() < outSize) {
+    long availSpace = tts.getResourceStatus().getAvailableSpace();
+    if(availSpace < outSize) {
       LOG.warn("No room for map task. Node " + node + 
-               " has " + tts.getAvailableSpace() + 
+               " has " + availSpace + 
                " bytes free; but we expect map to take " + outSize);
 
       return -1; //see if a different TIP might work better. 
@@ -1295,9 +1303,10 @@ class JobInProgress {
     }
 
     long outSize = resourceEstimator.getEstimatedReduceInputSize();
-    if(tts.getAvailableSpace() < outSize) {
+    long availSpace = tts.getResourceStatus().getAvailableSpace();
+    if(availSpace < outSize) {
       LOG.warn("No room for reduce task. Node " + taskTracker + " has " +
-               tts.getAvailableSpace() + 
+                availSpace + 
                " bytes free; but we expect reduce input to take " + outSize);
 
       return -1; //see if a different TIP might work better. 

src/mapred/org/apache/hadoop/mapred/JobTracker.java

@@ -1653,7 +1653,15 @@ public class JobTracker implements MRConstants, InterTrackerProtocol,
     TaskInProgress tip = getTip(tipid);
     return (tip == null ? null : tip.getCounters());
   }
-    
+
+  /**
+   * Returns the configured task scheduler for this job tracker.
+   * @return the configured task scheduler
+   */
+  TaskScheduler getTaskScheduler() {
+    return taskScheduler;
+  }
+  
   /**
    * Returns specified TaskInProgress, or null.
    */

src/mapred/org/apache/hadoop/mapred/TaskTracker.java

@@ -177,6 +177,11 @@ public class TaskTracker
   private MapEventsFetcherThread mapEventsFetcher;
   int workerThreads;
   private CleanupQueue directoryCleanupThread;
+  private long maxVirtualMemoryForTasks 
+                                    = JobConf.DISABLED_VIRTUAL_MEMORY_LIMIT;
+  private long defaultMemoryPerTask = JobConf.DISABLED_VIRTUAL_MEMORY_LIMIT;
+  
+  
   /**
    * the minimum interval between jobtracker polls
    */
@@ -432,6 +437,13 @@ public class TaskTracker
                              "Map-events fetcher for all reduce tasks " + "on " + 
                              taskTrackerName);
     mapEventsFetcher.start();
+    maxVirtualMemoryForTasks = fConf.getMaxVirtualMemoryForTasks();
+    if (maxVirtualMemoryForTasks != 
+                JobConf.DISABLED_VIRTUAL_MEMORY_LIMIT) {
+      defaultMemoryPerTask = maxVirtualMemoryForTasks /
+                                    (maxCurrentMapTasks + 
+                                        maxCurrentReduceTasks);
+    }
     this.running = true;
   }
   
@@ -704,7 +716,11 @@ public class TaskTracker
     }
     launchTaskForJob(tip, new JobConf(rjob.jobFile)); 
   }
-    
+
+  private long getDefaultMemoryPerTask() {
+    return defaultMemoryPerTask;
+  }
+
   private void launchTaskForJob(TaskInProgress tip, JobConf jobConf) throws IOException{
     synchronized (tip) {
       try {
@@ -1034,7 +1050,12 @@ public class TaskTracker
     if (askForNewTask) {
       checkLocalDirs(fConf.getLocalDirs());
       askForNewTask = enoughFreeSpace(localMinSpaceStart);
-      status.setAvailableSpace( getFreeSpace() );
+      status.getResourceStatus().setAvailableSpace( getFreeSpace() );
+      long freeVirtualMem = findFreeVirtualMemory();
+      LOG.debug("Setting amount of free virtual memory for the new task: " +
+                    freeVirtualMem);
+      status.getResourceStatus().setFreeVirtualMemory(freeVirtualMem);
+      status.getResourceStatus().setDefaultVirtualMemoryPerTask(getDefaultMemoryPerTask());      
     }
       
     //
@@ -1079,6 +1100,68 @@ public class TaskTracker
     return heartbeatResponse;
   }
 
+  /**
+   * Return the maximum amount of memory available for all tasks on 
+   * this tracker
+   * @return maximum amount of virtual memory
+   */
+  long getMaxVirtualMemoryForTasks() {
+    return maxVirtualMemoryForTasks;
+  }
+  
+  /**
+   * Find the minimum amount of virtual memory that would be
+   * available for a new task.
+   * 
+   * The minimum amount of virtual memory is computed by looking
+   * at the maximum amount of virtual memory that is allowed for
+   * all tasks in the system, as per mapred.tasktracker.tasks.maxmemory,
+   * and the total amount of maximum virtual memory that can be
+   * used by all currently running tasks.
+   * 
+   * @return amount of free virtual memory that can be assured for
+   * new tasks
+   */
+  private synchronized long findFreeVirtualMemory() {
+  
+    if (maxVirtualMemoryForTasks == JobConf.DISABLED_VIRTUAL_MEMORY_LIMIT) {
+      // this will disable picking up tasks based on free memory.
+      return JobConf.DISABLED_VIRTUAL_MEMORY_LIMIT;
+    }
+  
+    long maxMemoryUsed = 0L;
+    for (TaskInProgress tip: runningTasks.values()) {
+      // the following task states are one in which the slot is
+      // still occupied and hence memory of the task should be
+      // accounted in used memory.
+      if ((tip.getRunState() == TaskStatus.State.RUNNING)
+            || (tip.getRunState() == TaskStatus.State.UNASSIGNED)
+            || (tip.getRunState() == TaskStatus.State.COMMIT_PENDING)) {
+        maxMemoryUsed += getMemoryForTask(tip);
+      }
+    }
+  
+    return (maxVirtualMemoryForTasks - maxMemoryUsed);
+  }
+
+  /**
+   * Return the memory allocated for a TIP.
+   * 
+   * If the TIP's job has a configured value for the max memory that is
+   * returned. Else, the default memory that would be assigned for the
+   * task is returned.
+   * @param tip The TaskInProgress
+   * @return the memory allocated for the TIP.
+   */
+  private long getMemoryForTask(TaskInProgress tip) {
+    long memForTask = tip.getJobConf().getMaxVirtualMemoryForTask();
+    if (memForTask == JobConf.DISABLED_VIRTUAL_MEMORY_LIMIT) {
+      memForTask = this.getDefaultMemoryPerTask();
+    }
+    return memForTask;
+  }  
+  
+  
   /**
    * Check if the jobtracker directed a 'reset' of the tasktracker.
    * 

src/mapred/org/apache/hadoop/mapred/TaskTrackerStatus.java

@@ -48,13 +48,98 @@ class TaskTrackerStatus implements Writable {
   volatile long lastSeen;
   private int maxMapTasks;
   private int maxReduceTasks;
-  long availableSpace; //space available on this node
+   
+  /**
+   * Class representing a collection of resources on this tasktracker.
+   */
+  static class ResourceStatus implements Writable {
+    
+    private long freeVirtualMemory;
+    private long defaultVirtualMemoryPerTask;
+    private long availableSpace;
+    
+    ResourceStatus() {
+      freeVirtualMemory = JobConf.DISABLED_VIRTUAL_MEMORY_LIMIT;
+      defaultVirtualMemoryPerTask = JobConf.DISABLED_VIRTUAL_MEMORY_LIMIT;
+      availableSpace = Long.MAX_VALUE;
+    }
+    
+    /**
+     * Set the amount of free virtual memory that is available for running
+     * a new task
+     * @param freeVMem amount of free virtual memory
+     */
+    void setFreeVirtualMemory(long freeVmem) {
+      freeVirtualMemory = freeVmem;
+    }
+
+    /**
+     * Get the amount of free virtual memory that will be available for
+     * running a new task. 
+     * 
+     * If this is {@link JobConf.DISABLED_VIRTUAL_MEMORY_LIMIT}, it should 
+     * be ignored and not used in computation.
+     * 
+     *@return amount of free virtual memory.
+     */
+    long getFreeVirtualMemory() {
+      return freeVirtualMemory;
+    }
+
+    /**
+     * Set the default amount of virtual memory per task.
+     * @param vmem amount of free virtual memory.
+     */
+    void setDefaultVirtualMemoryPerTask(long defaultVmem) {
+      defaultVirtualMemoryPerTask = defaultVmem;
+    }
+    
+    /**
+     * Get the default amount of virtual memory per task.
+     * 
+     * This amount will be returned if a task's job does not specify any
+     * virtual memory itself. If this is 
+     * {@link JobConf.DISABLED_VIRTUAL_MEMORY_LIMIT}, it should be ignored 
+     * and not used in any computation.
+     * 
+     * @return default amount of virtual memory per task. 
+     */    
+    long getDefaultVirtualMemoryPerTask() {
+      return defaultVirtualMemoryPerTask;
+    }
+    
+    void setAvailableSpace(long availSpace) {
+      availableSpace = availSpace;
+    }
     
+    /**
+     * Will return LONG_MAX if space hasn't been measured yet.
+     * @return bytes of available local disk space on this tasktracker.
+     */    
+    long getAvailableSpace() {
+      return availableSpace;
+    }
+    
+    public void write(DataOutput out) throws IOException {
+      WritableUtils.writeVLong(out, freeVirtualMemory);
+      WritableUtils.writeVLong(out, defaultVirtualMemoryPerTask);
+      WritableUtils.writeVLong(out, availableSpace);
+    }
+    
+    public void readFields(DataInput in) throws IOException {
+      freeVirtualMemory = WritableUtils.readVLong(in);;
+      defaultVirtualMemoryPerTask = WritableUtils.readVLong(in);;
+      availableSpace = WritableUtils.readVLong(in);;
+    }
+  }
+  
+  private ResourceStatus resStatus;
+  
   /**
    */
   public TaskTrackerStatus() {
     taskReports = new ArrayList<TaskStatus>();
-    this.availableSpace = Long.MAX_VALUE; //not measured by default.
+    resStatus = new ResourceStatus();
   }
 
   /**
@@ -71,7 +156,7 @@ class TaskTrackerStatus implements Writable {
     this.failures = failures;
     this.maxMapTasks = maxMapTasks;
     this.maxReduceTasks = maxReduceTasks;
-    this.availableSpace = Long.MAX_VALUE; //not measured by default.
+    this.resStatus = new ResourceStatus();
   }
 
   /**
@@ -171,18 +256,15 @@ class TaskTrackerStatus implements Writable {
   }  
   
   /**
-   * Will return LONG_MAX if space hasn't been measured yet.
-   * @return bytes of available local disk space on this tasktracker.
+   * Return the {@link ResourceStatus} object configured with this
+   * status.
+   * 
+   * @return the resource status
    */
-  public long getAvailableSpace() {
-    return availableSpace;
-  }
-  
-  public void setAvailableSpace(long a) {
-    availableSpace = a;
+  ResourceStatus getResourceStatus() {
+    return resStatus;
   }
   
-  
   ///////////////////////////////////////////
   // Writable
   ///////////////////////////////////////////
@@ -193,8 +275,9 @@ class TaskTrackerStatus implements Writable {
     out.writeInt(failures);
     out.writeInt(maxMapTasks);
     out.writeInt(maxReduceTasks);
+    resStatus.write(out);
     out.writeInt(taskReports.size());
-    out.writeLong(availableSpace);
+
     for (TaskStatus taskStatus : taskReports) {
       TaskStatus.writeTaskStatus(out, taskStatus);
     }
@@ -207,9 +290,10 @@ class TaskTrackerStatus implements Writable {
     this.failures = in.readInt();
     this.maxMapTasks = in.readInt();
     this.maxReduceTasks = in.readInt();
+    resStatus.readFields(in);
     taskReports.clear();
     int numTasks = in.readInt();
-    this.availableSpace = in.readLong();
+
     for (int i = 0; i < numTasks; i++) {
       taskReports.add(TaskStatus.readTaskStatus(in));
     }

src/test/org/apache/hadoop/mapred/MiniMRCluster.java

@@ -73,7 +73,11 @@ public class MiniMRCluster {
     public int getJobTrackerInfoPort() {
       return tracker.getInfoPort();
     }
-        
+  
+    public JobTracker getJobTracker() {
+      return tracker;
+    }
+    
     /**
      * Create the job tracker and run it.
      */
@@ -116,12 +120,18 @@ public class MiniMRCluster {
     volatile boolean isDead = false;
     int numDir;
 
-    TaskTrackerRunner(int trackerId, int numDir, String hostname) 
+    TaskTrackerRunner(int trackerId, int numDir, String hostname, 
+                                    JobConf cfg) 
     throws IOException {
       this.trackerId = trackerId;
       this.numDir = numDir;
       localDirs = new String[numDir];
-      JobConf conf = createJobConf();
+      JobConf conf = null;
+      if (cfg == null) {
+        conf = createJobConf();
+      } else {
+        conf = createJobConf(cfg);
+      }
       if (hostname != null) {
         conf.set("slave.host.name", hostname);
       }
@@ -216,6 +226,10 @@ public class MiniMRCluster {
             taskTrackerList.get(taskTracker)).getLocalDir();
   }
 
+  public JobTrackerRunner getJobTrackerRunner() {
+    return jobTracker;
+  }
+  
   /**
    * Get the number of task trackers in the cluster
    */
@@ -413,7 +427,7 @@ public class MiniMRCluster {
       }
       TaskTrackerRunner taskTracker;
       taskTracker = new TaskTrackerRunner(idx, numDir, 
-          hosts == null ? null : hosts[idx]);
+          hosts == null ? null : hosts[idx], conf);
       
       Thread taskTrackerThread = new Thread(taskTracker);
       taskTrackerList.add(taskTracker);

src/test/org/apache/hadoop/mapred/TestHighRAMJobs.java

@@ -0,0 +1,261 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.mapred;
+
+import java.io.IOException;
+import java.util.List;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.examples.SleepJob;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hadoop.util.ToolRunner;
+
+import junit.framework.TestCase;
+
+/**
+ * This test class tests the functionality related to configuring, reporting
+ * and computing memory related parameters in a Map/Reduce cluster.
+ * 
+ * Each test sets up a {@link MiniMRCluster} with a locally defined 
+ * {@link org.apache.hadoop.mapred.TaskScheduler}. This scheduler validates 
+ * the memory related configuration is correctly computed and reported from 
+ * the tasktracker in 
+ * {@link org.apache.hadoop.mapred.TaskScheduler.assignTasks()}.
+ *  
+ */
+public class TestHighRAMJobs extends TestCase {
+
+  private static final Log LOG = LogFactory.getLog(TestHighRAMJobs.class);
+
+  private static final String DEFAULT_SLEEP_JOB_MAP_COUNT = "1";
+  private static final String DEFAULT_SLEEP_JOB_REDUCE_COUNT = "1";
+  private static final String DEFAULT_MAP_SLEEP_TIME = "1000";
+  private static final String DEFAULT_REDUCE_SLEEP_TIME = "1000";
+  private static final long DISABLED_VIRTUAL_MEMORY_LIMIT = -1L;
+  
+  private MiniDFSCluster miniDFSCluster;
+  private MiniMRCluster miniMRCluster;
+  
+  public static class FakeTaskScheduler extends JobQueueTaskScheduler {
+    
+    private boolean hasPassed = true;
+    private String message;
+    private boolean isFirstTime = true;
+    
+    public FakeTaskScheduler() {
+      super();
+    }
+    
+    public boolean hasTestPassed() {
+      return hasPassed;
+    }
+    
+    public String getFailureMessage() {
+      return message;
+    }
+    
+    @Override
+    public List<Task> assignTasks(TaskTrackerStatus status) 
+                                          throws IOException {
+      TestHighRAMJobs.LOG.info("status = " + status.getResourceStatus().getFreeVirtualMemory());
+
+      long initialFreeMemory = getConf().getLong("initialFreeMemory", 0L);
+      long memoryPerTaskOnTT = getConf().getLong("memoryPerTaskOnTT", 0L);
+
+      if (isFirstTime) {
+        isFirstTime = false;
+        if (initialFreeMemory != status.getResourceStatus().getFreeVirtualMemory()) {
+          hasPassed = false;
+          message = "Initial memory expected = " + initialFreeMemory
+                      + " reported = " + status.getResourceStatus().getFreeVirtualMemory();
+        }
+        if (memoryPerTaskOnTT != status.getResourceStatus().getDefaultVirtualMemoryPerTask()) {
+          hasPassed = false;
+          message = "Memory per task on TT expected = " + memoryPerTaskOnTT
+                      + " reported = " 
+                      + status.getResourceStatus().getDefaultVirtualMemoryPerTask();
+        }
+      } else if (initialFreeMemory != DISABLED_VIRTUAL_MEMORY_LIMIT) {
+        
+        long memoryPerTask = memoryPerTaskOnTT; // by default
+        if (getConf().getLong("memoryPerTask", 0L) != 
+                                            DISABLED_VIRTUAL_MEMORY_LIMIT) {
+          memoryPerTask = getConf().getLong("memoryPerTask", 0L);
+        }
+          
+        long expectedFreeMemory = 0;
+        int runningTaskCount = status.countMapTasks() +
+                              status.countReduceTasks();
+        expectedFreeMemory = initialFreeMemory - 
+                                (memoryPerTask * runningTaskCount);
+
+        TestHighRAMJobs.LOG.info("expected free memory = " + 
+                                  expectedFreeMemory + ", reported = " + 
+                                  status.getResourceStatus().getFreeVirtualMemory());
+        if (expectedFreeMemory != status.getResourceStatus().getFreeVirtualMemory()) {
+          hasPassed = false;
+          message = "Expected free memory after " + runningTaskCount
+                      + " tasks are scheduled = " + expectedFreeMemory
+                      + ", reported = " + status.getResourceStatus().getFreeVirtualMemory();
+        }
+      }
+      return super.assignTasks(status);
+    }
+  }
+  
+  /* Test that verifies default values are configured and reported
+   * correctly.
+   */
+  public void testDefaultValuesForHighRAMJobs() throws Exception {
+    long defaultMemoryLimit = DISABLED_VIRTUAL_MEMORY_LIMIT;
+    try {
+      setUpCluster(defaultMemoryLimit, defaultMemoryLimit, 
+                    defaultMemoryLimit, null);
+      runJob(defaultMemoryLimit, DEFAULT_MAP_SLEEP_TIME, 
+          DEFAULT_REDUCE_SLEEP_TIME, DEFAULT_SLEEP_JOB_MAP_COUNT, 
+          DEFAULT_SLEEP_JOB_REDUCE_COUNT);
+      verifyTestResults();
+    } finally {
+      tearDownCluster();
+    }
+  }
+  
+  /* Test that verifies default value for memory per task on TT
+   * when the number of slots is non-default.
+   */
+  public void testDefaultMemoryPerTask() throws Exception {
+    long maxVmem = 1024*1024*1024L;
+    JobConf conf = new JobConf();
+    conf.setInt("mapred.tasktracker.map.tasks.maximum", 1);
+    conf.setInt("mapred.tasktracker.reduce.tasks.maximum", 1);
+    // change number of slots to 2.
+    long defaultMemPerTaskOnTT = maxVmem / 2;
+    try {
+      setUpCluster(maxVmem, defaultMemPerTaskOnTT, 
+                    DISABLED_VIRTUAL_MEMORY_LIMIT, conf);
+      runJob(DISABLED_VIRTUAL_MEMORY_LIMIT, DEFAULT_MAP_SLEEP_TIME,
+              DEFAULT_REDUCE_SLEEP_TIME, DEFAULT_SLEEP_JOB_MAP_COUNT,
+              DEFAULT_SLEEP_JOB_REDUCE_COUNT);
+      verifyTestResults();
+    } finally {
+      tearDownCluster();
+    }
+  }
+  
+  /* Test that verifies configured value for free memory is
+   * reported correctly. The test does NOT configure a value for
+   * memory per task. Hence, it also verifies that the default value
+   * per task on the TT is calculated correctly.
+   */
+  public void testConfiguredValueForFreeMemory() throws Exception {
+    long maxVmem = 1024*1024*1024L;
+    long defaultMemPerTaskOnTT = maxVmem/4; // 4 = default number of slots.
+    try {
+      setUpCluster(maxVmem, defaultMemPerTaskOnTT,
+                    DISABLED_VIRTUAL_MEMORY_LIMIT, null);
+      runJob(DISABLED_VIRTUAL_MEMORY_LIMIT, "10000",
+              DEFAULT_REDUCE_SLEEP_TIME, DEFAULT_SLEEP_JOB_MAP_COUNT,
+              DEFAULT_SLEEP_JOB_REDUCE_COUNT);
+      verifyTestResults();
+    } finally {
+      tearDownCluster();
+    }
+  }
+  
+  public void testHighRAMJob() throws Exception {
+    long maxVmem = 1024*1024*1024L;
+    long defaultMemPerTaskOnTT = maxVmem/4; // 4 = default number of slots.
+    /* Set a HIGH RAM requirement for a job. As 4 is the
+     * default number of slots, we set up the memory limit
+     * per task to be more than 25%. 
+     */
+    long maxVmemPerTask = maxVmem/3;
+    try {
+      setUpCluster(maxVmem, defaultMemPerTaskOnTT,
+                    maxVmemPerTask, null);
+      /* set up sleep limits higher, so the scheduler will see varying
+       * number of running tasks at a time. Also modify the number of
+       * map tasks so we test the iteration over more than one task.
+       */
+      runJob(maxVmemPerTask, "10000", "10000", "2", 
+                      DEFAULT_SLEEP_JOB_REDUCE_COUNT);
+      verifyTestResults();
+    } finally {
+      tearDownCluster();
+    }
+  }
+  
+  private void setUpCluster(long initialFreeMemory, long memoryPerTaskOnTT,
+                            long memoryPerTask, JobConf conf) 
+                              throws Exception {
+    if (conf == null) {
+      conf = new JobConf();
+    }
+    conf.setClass("mapred.jobtracker.taskScheduler", 
+        TestHighRAMJobs.FakeTaskScheduler.class,
+        TaskScheduler.class);
+    if (initialFreeMemory != -1L) {
+      conf.setMaxVirtualMemoryForTasks(initialFreeMemory);  
+    }
+    conf.setLong("initialFreeMemory", initialFreeMemory);
+    conf.setLong("memoryPerTaskOnTT", memoryPerTaskOnTT);
+    conf.setLong("memoryPerTask", memoryPerTask);
+    miniDFSCluster = new MiniDFSCluster(conf, 1, true, null);
+    FileSystem fileSys = miniDFSCluster.getFileSystem();
+    String namenode = fileSys.getUri().toString();
+    miniMRCluster = new MiniMRCluster(1, namenode, 3, 
+                      null, null, conf);    
+  }
+  
+  private void runJob(long memoryPerTask, String mapSleepTime,
+                        String reduceSleepTime, String mapTaskCount,
+                        String reduceTaskCount) 
+                                        throws Exception {
+    Configuration sleepJobConf = new Configuration();
+    sleepJobConf.set("mapred.job.tracker", "localhost:"
+                              + miniMRCluster.getJobTrackerPort());
+    if (memoryPerTask != -1L) {
+      sleepJobConf.setLong("mapred.task.maxmemory", memoryPerTask);
+    }
+    launchSleepJob(mapSleepTime, reduceSleepTime, 
+                    mapTaskCount, reduceTaskCount, sleepJobConf);    
+  }
+
+  private void launchSleepJob(String mapSleepTime, String reduceSleepTime,
+                              String mapTaskCount, String reduceTaskCount,
+                              Configuration conf) throws Exception {
+    String[] args = { "-m", mapTaskCount, "-r", reduceTaskCount,
+                      "-mt", mapSleepTime, "-rt", reduceSleepTime };
+    ToolRunner.run(conf, new SleepJob(), args);
+  }
+
+  private void verifyTestResults() {
+    FakeTaskScheduler scheduler = 
+      (FakeTaskScheduler)miniMRCluster.getJobTrackerRunner().
+                              getJobTracker().getTaskScheduler();
+    assertTrue(scheduler.getFailureMessage(), scheduler.hasTestPassed());
+  }
+  
+  private void tearDownCluster() {
+    if (miniMRCluster != null) { miniMRCluster.shutdown(); }
+    if (miniDFSCluster != null) { miniDFSCluster.shutdown(); }
+  }
+}