HADOOP-2901. Fixes the creation of info servers in the JobClient and JobTracker. Removes the creation from JobClient and removes additional info server from the JobTracker. Also adds the command line utility to view the history files (HADOOP-2896), and fixes bugs in JSPs to do with analysis - HADOOP-2742, HADOOP-2792. Contributed by Amareshwari Sri Ramadasu.

git-svn-id: https://svn.apache.org/repos/asf/hadoop/core/trunk@637723 13f79535-47bb-0310-9956-ffa450edef68

CHANGES.txt

@@ -230,6 +230,14 @@ Trunk (unreleased changes)
     HADOOP-2976. When a lease expires, the Namenode ensures that 
     blocks of the file are adequately replicated. (dhruba)
 
+    HADOOP-2901. Fixes the creation of info servers in the JobClient
+    and JobTracker. Removes the creation from JobClient and removes
+    additional info server from the JobTracker. Also adds the command
+    line utility to view the history files (HADOOP-2896), and fixes
+    bugs in JSPs to do with analysis - HADOOP-2742, HADOOP-2792.
+    (Amareshwari Sri Ramadasu via ddas)
+
+
 Release 0.16.1 - 2008-03-13
 
   INCOMPATIBLE CHANGES

build.xml

@@ -179,7 +179,6 @@
     <mkdir dir="${build.webapps}/job/WEB-INF"/>
     <mkdir dir="${build.webapps}/dfs/WEB-INF"/>
     <mkdir dir="${build.webapps}/datanode/WEB-INF"/>
-    <mkdir dir="${build.webapps}/history/WEB-INF"/>
     <mkdir dir="${build.examples}"/>
     <mkdir dir="${build.anttasks}"/>
     <mkdir dir="${build.dir}/c++"/>
@@ -274,13 +273,6 @@
      webxml="${build.webapps}/dfs/WEB-INF/web.xml">
     </jsp-compile>
 
-    <jsp-compile
-     uriroot="${src.webapps}/history"
-     outputdir="${build.src}"
-     package="org.apache.hadoop.mapred"
-     webxml="${build.webapps}/history/WEB-INF/web.xml">
-    </jsp-compile>
-
     <jsp-compile
      uriroot="${src.webapps}/datanode"
      outputdir="${build.src}"

conf/hadoop-default.xml

@@ -523,15 +523,6 @@ creations/deletions), or "all".</description>
   </description>
 </property>
 
-<property>
-  <name>mapred.job.history.http.bindAddress</name>
-  <value>0.0.0.0:0</value>
-  <description>
-    The job history http server bind address and port.
-    If the port is 0 then the server will start on a free port.
-  </description>
-</property>
-
 <property>
   <name>mapred.job.tracker.handler.count</name>
   <value>10</value>

docs/changes.html

@@ -97,7 +97,7 @@ DFSClient and DataNode sockets have 10min write timeout.<br />(rangadi)</li>
     </ol>
   </li>
   <li><a href="javascript:toggleList('trunk_(unreleased_changes)_._improvements_')">  IMPROVEMENTS
-</a>&nbsp;&nbsp;&nbsp;(13)
+</a>&nbsp;&nbsp;&nbsp;(14)
     <ol id="trunk_(unreleased_changes)_._improvements_">
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-2655">HADOOP-2655</a>. Copy on write for data and metadata files in the
 presence of snapshots. Needed for supporting appends to HDFS
@@ -125,6 +125,8 @@ cryptography.<br />(omalley)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-2775">HADOOP-2775</a>.  Adds unit test framework for HOD.
 (Vinod Kumar Vavilapalli via ddas).
 </li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-2804">HADOOP-2804</a>.  Add support to publish CHANGES.txt as HTML when running
+the Ant 'docs' target.<br />(nigel)</li>
     </ol>
   </li>
   <li><a href="javascript:toggleList('trunk_(unreleased_changes)_._optimizations_')">  OPTIMIZATIONS
@@ -144,7 +146,7 @@ HDFS, without negatively affecting read throughput.<br />(rangadi)</li>
     </ol>
   </li>
   <li><a href="javascript:toggleList('trunk_(unreleased_changes)_._bug_fixes_')">  BUG FIXES
-</a>&nbsp;&nbsp;&nbsp;(34)
+</a>&nbsp;&nbsp;&nbsp;(36)
     <ol id="trunk_(unreleased_changes)_._bug_fixes_">
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-2195">HADOOP-2195</a>. '-mkdir' behaviour is now closer to Linux shell in case of
 errors.<br />(Mahadev Konar via rangadi)</li>
@@ -208,6 +210,11 @@ floating point number.<br />(Dave Brosius via ddas)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-2972">HADOOP-2972</a>. Fix for a NPE in FSDataset.invalidate.<br />(Mahadev Konar via dhruba)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-2994">HADOOP-2994</a>. Code cleanup for DFSClient: remove redundant
 conversions from string to string.<br />(Dave Brosius via dhruba)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3009">HADOOP-3009</a>. TestFileCreation sometimes fails because restarting
+minidfscluster sometimes creates datanodes with ports that are
+different from their original instance.<br />(dhruba)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-2992">HADOOP-2992</a>. Distributed Upgrade framework works correctly with
+more than one upgrade object.<br />(Konstantin Shvachko via dhruba)</li>
     </ol>
   </li>
 </ul>

docs/cluster_setup.html

@@ -637,8 +637,7 @@ document.write("Last Published: " + document.lastModified);
 <p> The job history files are stored in central location 
             <span class="codefrag"> hadoop.job.history.location </span> which can be on DFS also,
             whose default value is <span class="codefrag">${HADOOP_LOG_DIR}/history</span>. 
-            Job history server is started on job tracker. The history 
-            web UI is accessible from job tracker web UI.</p>
+            The history web UI is accessible from job tracker web UI.</p>
 <p> The history files are also logged to user specified directory
             <span class="codefrag">hadoop.job.history.user.location</span> 
             which defaults to job output directory. The files are stored in
@@ -647,18 +646,19 @@ document.write("Last Published: " + document.lastModified);
             logging by giving the value <span class="codefrag">none</span> for 
             <span class="codefrag">hadoop.job.history.user.location</span> 
 </p>
-<p> User can view logs in specified directory using 
-            the following command <br>
+<p> User can view the history logs summary in specified directory 
+            using the following command <br>
             
 <span class="codefrag">$ bin/hadoop job -history output-dir</span>
+<br> 
+            This command will print job details, failed and killed tip
+            details. <br>
+            More details about the job such as successful tasks and 
+            task attempts made for each task can be viewed using the  
+            following command <br>
+            
+<span class="codefrag">$ bin/hadoop job -history all output-dir</span>
 <br>
-            This will start a stand alone jetty on the client and 
-            load history jsp's. 
-            It will display the port where the server is up at. The server will
-            be up for 30 minutes. User has to use 
-            <span class="codefrag"> http://hostname:port </span> to view the history. User can 
-            also provide http bind address using 
-            <span class="codefrag">mapred.job.history.http.bindAddress</span>
 </p>
 <p>Once all the necessary configuration is complete, distribute the files
       to the <span class="codefrag">HADOOP_CONF_DIR</span> directory on all the machines, 
@@ -666,7 +666,7 @@ document.write("Last Published: " + document.lastModified);
 </div>
     
     
-<a name="N10343"></a><a name="Hadoop+Rack+Awareness"></a>
+<a name="N10345"></a><a name="Hadoop+Rack+Awareness"></a>
 <h2 class="h3">Hadoop Rack Awareness</h2>
 <div class="section">
 <p>The HDFS and the Map-Reduce components are rack-aware.</p>
@@ -689,7 +689,7 @@ document.write("Last Published: " + document.lastModified);
 </div>
     
     
-<a name="N10369"></a><a name="Hadoop+Startup"></a>
+<a name="N1036B"></a><a name="Hadoop+Startup"></a>
 <h2 class="h3">Hadoop Startup</h2>
 <div class="section">
 <p>To start a Hadoop cluster you will need to start both the HDFS and 
@@ -724,7 +724,7 @@ document.write("Last Published: " + document.lastModified);
 </div>
     
     
-<a name="N103AF"></a><a name="Hadoop+Shutdown"></a>
+<a name="N103B1"></a><a name="Hadoop+Shutdown"></a>
 <h2 class="h3">Hadoop Shutdown</h2>
 <div class="section">
 <p>

docs/hadoop-default.html

@@ -315,12 +315,6 @@ creations/deletions), or "all".</td>
   </td>
 </tr>
 <tr>
-<td><a name="mapred.job.history.http.bindAddress">mapred.job.history.http.bindAddress</a></td><td>0.0.0.0:0</td><td>
-    The job history http server bind address and port.
-    If the port is 0 then the server will start on a free port.
-  </td>
-</tr>
-<tr>
 <td><a name="mapred.job.tracker.handler.count">mapred.job.tracker.handler.count</a></td><td>10</td><td>
     The number of server threads for the JobTracker. This should be roughly
     4% of the number of tasktracker nodes.

docs/mapred_tutorial.html

@@ -289,7 +289,7 @@ document.write("Last Published: " + document.lastModified);
 <a href="#Example%3A+WordCount+v2.0">Example: WordCount v2.0</a>
 <ul class="minitoc">
 <li>
-<a href="#Source+Code-N10BDE">Source Code</a>
+<a href="#Source+Code-N10BE0">Source Code</a>
 </li>
 <li>
 <a href="#Sample+Runs">Sample Runs</a>
@@ -1579,23 +1579,24 @@ document.write("Last Published: " + document.lastModified);
 <p> Job history files are also logged to user specified directory
         <span class="codefrag">hadoop.job.history.user.location</span> 
         which defaults to job output directory. The files are stored in
-        "_logs/history/" in the specified directory. Hence, by default they will
-        be in mapred.output.dir/_logs/history. User can stop
+        "_logs/history/" in the specified directory. Hence, by default they
+        will be in mapred.output.dir/_logs/history. User can stop
         logging by giving the value <span class="codefrag">none</span> for 
         <span class="codefrag">hadoop.job.history.user.location</span>
 </p>
-<p> User can view logs in specified directory using 
-        the following command <br>
+<p> User can view the history logs summary in specified directory 
+        using the following command <br>
         
 <span class="codefrag">$ bin/hadoop job -history output-dir</span>
+<br> 
+        This command will print job details, failed and killed tip
+        details. <br>
+        More details about the job such as successful tasks and 
+        task attempts made for each task can be viewed using the  
+        following command <br>
+       
+<span class="codefrag">$ bin/hadoop job -history all output-dir</span>
 <br>
-        This will start a stand alone jetty on the client and 
-        load history jsp's. 
-        It will display the port where the server is up at. The server will
-        be up for 30 minutes. User has to use 
-        <span class="codefrag"> http://hostname:port </span> to view the history. User can 
-        also provide http bind address using 
-        <span class="codefrag">mapred.job.history.http.bindAddress</span>
 </p>
 <p> User can use 
         <a href="api/org/apache/hadoop/mapred/OutputLogFilter.html">OutputLogFilter</a>
@@ -1603,7 +1604,7 @@ document.write("Last Published: " + document.lastModified);
 <p>Normally the user creates the application, describes various facets 
         of the job via <span class="codefrag">JobConf</span>, and then uses the 
         <span class="codefrag">JobClient</span> to submit the job and monitor its progress.</p>
-<a name="N10917"></a><a name="Job+Control"></a>
+<a name="N10919"></a><a name="Job+Control"></a>
 <h4>Job Control</h4>
 <p>Users may need to chain map-reduce jobs to accomplish complex
           tasks which cannot be done via a single map-reduce job. This is fairly
@@ -1639,7 +1640,7 @@ document.write("Last Published: " + document.lastModified);
             </li>
           
 </ul>
-<a name="N10941"></a><a name="Job+Input"></a>
+<a name="N10943"></a><a name="Job+Input"></a>
 <h3 class="h4">Job Input</h3>
 <p>
 <a href="api/org/apache/hadoop/mapred/InputFormat.html">
@@ -1687,7 +1688,7 @@ document.write("Last Published: " + document.lastModified);
         appropriate <span class="codefrag">CompressionCodec</span>. However, it must be noted that
         compressed files with the above extensions cannot be <em>split</em> and 
         each compressed file is processed in its entirety by a single mapper.</p>
-<a name="N109AB"></a><a name="InputSplit"></a>
+<a name="N109AD"></a><a name="InputSplit"></a>
 <h4>InputSplit</h4>
 <p>
 <a href="api/org/apache/hadoop/mapred/InputSplit.html">
@@ -1701,7 +1702,7 @@ document.write("Last Published: " + document.lastModified);
           FileSplit</a> is the default <span class="codefrag">InputSplit</span>. It sets 
           <span class="codefrag">map.input.file</span> to the path of the input file for the
           logical split.</p>
-<a name="N109D0"></a><a name="RecordReader"></a>
+<a name="N109D2"></a><a name="RecordReader"></a>
 <h4>RecordReader</h4>
 <p>
 <a href="api/org/apache/hadoop/mapred/RecordReader.html">
@@ -1713,7 +1714,7 @@ document.write("Last Published: " + document.lastModified);
           for processing. <span class="codefrag">RecordReader</span> thus assumes the 
           responsibility of processing record boundaries and presents the tasks 
           with keys and values.</p>
-<a name="N109F3"></a><a name="Job+Output"></a>
+<a name="N109F5"></a><a name="Job+Output"></a>
 <h3 class="h4">Job Output</h3>
 <p>
 <a href="api/org/apache/hadoop/mapred/OutputFormat.html">
@@ -1738,7 +1739,7 @@ document.write("Last Published: " + document.lastModified);
 <p>
 <span class="codefrag">TextOutputFormat</span> is the default 
         <span class="codefrag">OutputFormat</span>.</p>
-<a name="N10A1C"></a><a name="Task+Side-Effect+Files"></a>
+<a name="N10A1E"></a><a name="Task+Side-Effect+Files"></a>
 <h4>Task Side-Effect Files</h4>
 <p>In some applications, component tasks need to create and/or write to
           side-files, which differ from the actual job-output files.</p>
@@ -1764,7 +1765,7 @@ document.write("Last Published: " + document.lastModified);
           JobConf.getOutputPath()</a>, and the framework will promote them 
           similarly for succesful task-attempts, thus eliminating the need to 
           pick unique paths per task-attempt.</p>
-<a name="N10A51"></a><a name="RecordWriter"></a>
+<a name="N10A53"></a><a name="RecordWriter"></a>
 <h4>RecordWriter</h4>
 <p>
 <a href="api/org/apache/hadoop/mapred/RecordWriter.html">
@@ -1772,9 +1773,9 @@ document.write("Last Published: " + document.lastModified);
           pairs to an output file.</p>
 <p>RecordWriter implementations write the job outputs to the 
           <span class="codefrag">FileSystem</span>.</p>
-<a name="N10A68"></a><a name="Other+Useful+Features"></a>
+<a name="N10A6A"></a><a name="Other+Useful+Features"></a>
 <h3 class="h4">Other Useful Features</h3>
-<a name="N10A6E"></a><a name="Counters"></a>
+<a name="N10A70"></a><a name="Counters"></a>
 <h4>Counters</h4>
 <p>
 <span class="codefrag">Counters</span> represent global counters, defined either by 
@@ -1788,7 +1789,7 @@ document.write("Last Published: " + document.lastModified);
           Reporter.incrCounter(Enum, long)</a> in the <span class="codefrag">map</span> and/or 
           <span class="codefrag">reduce</span> methods. These counters are then globally 
           aggregated by the framework.</p>
-<a name="N10A99"></a><a name="DistributedCache"></a>
+<a name="N10A9B"></a><a name="DistributedCache"></a>
 <h4>DistributedCache</h4>
 <p>
 <a href="api/org/apache/hadoop/filecache/DistributedCache.html">
@@ -1821,7 +1822,7 @@ document.write("Last Published: " + document.lastModified);
           <a href="api/org/apache/hadoop/filecache/DistributedCache.html#createSymlink(org.apache.hadoop.conf.Configuration)">
           DistributedCache.createSymlink(Path, Configuration)</a> api. Files 
           have <em>execution permissions</em> set.</p>
-<a name="N10AD7"></a><a name="Tool"></a>
+<a name="N10AD9"></a><a name="Tool"></a>
 <h4>Tool</h4>
 <p>The <a href="api/org/apache/hadoop/util/Tool.html">Tool</a> 
           interface supports the handling of generic Hadoop command-line options.
@@ -1861,7 +1862,7 @@ document.write("Last Published: " + document.lastModified);
             </span>
           
 </p>
-<a name="N10B09"></a><a name="IsolationRunner"></a>
+<a name="N10B0B"></a><a name="IsolationRunner"></a>
 <h4>IsolationRunner</h4>
 <p>
 <a href="api/org/apache/hadoop/mapred/IsolationRunner.html">
@@ -1885,13 +1886,13 @@ document.write("Last Published: " + document.lastModified);
 <p>
 <span class="codefrag">IsolationRunner</span> will run the failed task in a single 
           jvm, which can be in the debugger, over precisely the same input.</p>
-<a name="N10B3C"></a><a name="JobControl"></a>
+<a name="N10B3E"></a><a name="JobControl"></a>
 <h4>JobControl</h4>
 <p>
 <a href="api/org/apache/hadoop/mapred/jobcontrol/package-summary.html">
           JobControl</a> is a utility which encapsulates a set of Map-Reduce jobs
           and their dependencies.</p>
-<a name="N10B49"></a><a name="Data+Compression"></a>
+<a name="N10B4B"></a><a name="Data+Compression"></a>
 <h4>Data Compression</h4>
 <p>Hadoop Map-Reduce provides facilities for the application-writer to
           specify compression for both intermediate map-outputs and the
@@ -1905,7 +1906,7 @@ document.write("Last Published: " + document.lastModified);
           codecs for reasons of both performance (zlib) and non-availability of
           Java libraries (lzo). More details on their usage and availability are
           available <a href="native_libraries.html">here</a>.</p>
-<a name="N10B69"></a><a name="Intermediate+Outputs"></a>
+<a name="N10B6B"></a><a name="Intermediate+Outputs"></a>
 <h5>Intermediate Outputs</h5>
 <p>Applications can control compression of intermediate map-outputs
             via the 
@@ -1926,7 +1927,7 @@ document.write("Last Published: " + document.lastModified);
             <a href="api/org/apache/hadoop/mapred/JobConf.html#setMapOutputCompressionType(org.apache.hadoop.io.SequenceFile.CompressionType)">
             JobConf.setMapOutputCompressionType(SequenceFile.CompressionType)</a> 
             api.</p>
-<a name="N10B95"></a><a name="Job+Outputs"></a>
+<a name="N10B97"></a><a name="Job+Outputs"></a>
 <h5>Job Outputs</h5>
 <p>Applications can control compression of job-outputs via the
             <a href="api/org/apache/hadoop/mapred/OutputFormatBase.html#setCompressOutput(org.apache.hadoop.mapred.JobConf,%20boolean)">
@@ -1946,7 +1947,7 @@ document.write("Last Published: " + document.lastModified);
 </div>
 
     
-<a name="N10BC4"></a><a name="Example%3A+WordCount+v2.0"></a>
+<a name="N10BC6"></a><a name="Example%3A+WordCount+v2.0"></a>
 <h2 class="h3">Example: WordCount v2.0</h2>
 <div class="section">
 <p>Here is a more complete <span class="codefrag">WordCount</span> which uses many of the
@@ -1956,7 +1957,7 @@ document.write("Last Published: " + document.lastModified);
       <a href="quickstart.html#SingleNodeSetup">pseudo-distributed</a> or
       <a href="quickstart.html#Fully-Distributed+Operation">fully-distributed</a> 
       Hadoop installation.</p>
-<a name="N10BDE"></a><a name="Source+Code-N10BDE"></a>
+<a name="N10BE0"></a><a name="Source+Code-N10BE0"></a>
 <h3 class="h4">Source Code</h3>
 <table class="ForrestTable" cellspacing="1" cellpadding="4">
           
@@ -3166,7 +3167,7 @@ document.write("Last Published: " + document.lastModified);
 </tr>
         
 </table>
-<a name="N11340"></a><a name="Sample+Runs"></a>
+<a name="N11342"></a><a name="Sample+Runs"></a>
 <h3 class="h4">Sample Runs</h3>
 <p>Sample text-files as input:</p>
 <p>
@@ -3334,7 +3335,7 @@ document.write("Last Published: " + document.lastModified);
 <br>
         
 </p>
-<a name="N11414"></a><a name="Highlights"></a>
+<a name="N11416"></a><a name="Highlights"></a>
 <h3 class="h4">Highlights</h3>
 <p>The second version of <span class="codefrag">WordCount</span> improves upon the 
         previous one by using some features offered by the Map-Reduce framework:

src/docs/src/documentation/content/xdocs/cluster_setup.xml

@@ -373,8 +373,7 @@
             <p> The job history files are stored in central location 
             <code> hadoop.job.history.location </code> which can be on DFS also,
             whose default value is <code>${HADOOP_LOG_DIR}/history</code>. 
-            Job history server is started on job tracker. The history 
-            web UI is accessible from job tracker web UI.</p>
+            The history web UI is accessible from job tracker web UI.</p>
             
             <p> The history files are also logged to user specified directory
             <code>hadoop.job.history.user.location</code> 
@@ -384,16 +383,15 @@
             logging by giving the value <code>none</code> for 
             <code>hadoop.job.history.user.location</code> </p>
             
-            <p> User can view logs in specified directory using 
-            the following command <br/>
-            <code>$ bin/hadoop job -history output-dir</code><br/>
-            This will start a stand alone jetty on the client and 
-            load history jsp's. 
-            It will display the port where the server is up at. The server will
-            be up for 30 minutes. User has to use 
-            <code> http://hostname:port </code> to view the history. User can 
-            also provide http bind address using 
-            <code>mapred.job.history.http.bindAddress</code></p>
+            <p> User can view the history logs summary in specified directory 
+            using the following command <br/>
+            <code>$ bin/hadoop job -history output-dir</code><br/> 
+            This command will print job details, failed and killed tip
+            details. <br/>
+            More details about the job such as successful tasks and 
+            task attempts made for each task can be viewed using the  
+            following command <br/>
+            <code>$ bin/hadoop job -history all output-dir</code><br/></p> 
           </section>
         </section>
       </section>

src/docs/src/documentation/content/xdocs/mapred_tutorial.xml

@@ -1113,21 +1113,20 @@
         <p> Job history files are also logged to user specified directory
         <code>hadoop.job.history.user.location</code> 
         which defaults to job output directory. The files are stored in
-        "_logs/history/" in the specified directory. Hence, by default they will
-        be in mapred.output.dir/_logs/history. User can stop
+        "_logs/history/" in the specified directory. Hence, by default they
+        will be in mapred.output.dir/_logs/history. User can stop
         logging by giving the value <code>none</code> for 
         <code>hadoop.job.history.user.location</code></p>
 
-        <p> User can view logs in specified directory using 
-        the following command <br/>
-        <code>$ bin/hadoop job -history output-dir</code><br/>
-        This will start a stand alone jetty on the client and 
-        load history jsp's. 
-        It will display the port where the server is up at. The server will
-        be up for 30 minutes. User has to use 
-        <code> http://hostname:port </code> to view the history. User can 
-        also provide http bind address using 
-        <code>mapred.job.history.http.bindAddress</code></p>
+        <p> User can view the history logs summary in specified directory 
+        using the following command <br/>
+        <code>$ bin/hadoop job -history output-dir</code><br/> 
+        This command will print job details, failed and killed tip
+        details. <br/>
+        More details about the job such as successful tasks and 
+        task attempts made for each task can be viewed using the  
+        following command <br/>
+       <code>$ bin/hadoop job -history all output-dir</code><br/></p> 
             
         <p> User can use 
         <a href="ext:api/org/apache/hadoop/mapred/outputlogfilter">OutputLogFilter</a>

src/java/org/apache/hadoop/mapred/DefaultJobHistoryParser.java

@@ -127,20 +127,20 @@ public class DefaultJobHistoryParser {
   }
 
   // call this only for jobs that succeeded for better results. 
-  static class FailedOnNodesFilter implements JobHistory.Listener {
+  abstract static class NodesFilter implements JobHistory.Listener {
     private Map<String, Set<String>> badNodesToNumFailedTasks =
       new HashMap<String, Set<String>>();
     
     Map<String, Set<String>> getValues(){
       return badNodesToNumFailedTasks; 
     }
+    String failureType;
+    
     public void handle(JobHistory.RecordTypes recType, Map<Keys, String> values)
       throws IOException {
-      
       if (recType.equals(JobHistory.RecordTypes.MapAttempt) || 
           recType.equals(JobHistory.RecordTypes.ReduceAttempt)) {
-        
-        if (Values.FAILED.name().equals(values.get(Keys.TASK_STATUS)) ){
+        if (failureType.equals(values.get(Keys.TASK_STATUS)) ) {
           String hostName = values.get(Keys.HOSTNAME);
           String taskid = values.get(Keys.TASKID); 
           Set<String> tasks = badNodesToNumFailedTasks.get(hostName); 
@@ -154,33 +154,23 @@ public class DefaultJobHistoryParser {
         }
       }      
     }
+    abstract void setFailureType();
+    String getFailureType() {
+      return failureType;
+    }
+    NodesFilter() {
+      setFailureType();
+    }
   }
-  static class KilledOnNodesFilter implements JobHistory.Listener {
-    private Map<String, Set<String>> badNodesToNumFailedTasks =
-      new HashMap<String, Set<String>>();
-    
-    Map<String, Set<String>> getValues(){
-      return badNodesToNumFailedTasks; 
+ 
+  static class FailedOnNodesFilter extends NodesFilter {
+    void setFailureType() {
+      failureType = Values.FAILED.name();
     }
-    public void handle(JobHistory.RecordTypes recType, Map<Keys, String> values)
-      throws IOException {
-      
-      if (recType.equals(JobHistory.RecordTypes.MapAttempt) || 
-          recType.equals(JobHistory.RecordTypes.ReduceAttempt)) {
-        
-        if (Values.KILLED.name().equals(values.get(Keys.TASK_STATUS)) ){
-          String hostName = values.get(Keys.HOSTNAME);
-          String taskid = values.get(Keys.TASKID); 
-          Set<String> tasks = badNodesToNumFailedTasks.get(hostName); 
-          if (null == tasks ){
-            tasks = new TreeSet<String>(); 
-            tasks.add(taskid);
-            badNodesToNumFailedTasks.put(hostName, tasks);
-          }else{
-            tasks.add(taskid);
-          }
-        }
-      }      
+  }
+  static class KilledOnNodesFilter extends NodesFilter {
+    void setFailureType() {
+      failureType = Values.KILLED.name();
     }
   }
 }

src/java/org/apache/hadoop/mapred/HistoryViewer.java

@@ -0,0 +1,486 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.mapred;
+
+import java.io.IOException;
+import java.text.SimpleDateFormat;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.Date;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FileUtil;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.apache.hadoop.mapred.DefaultJobHistoryParser.*;
+import org.apache.hadoop.mapred.JobHistory.*;
+import org.apache.hadoop.util.StringUtils;
+
+/**
+ * This class is to view job history files.
+ */
+class HistoryViewer {
+  private static final Log LOG = LogFactory.getLog(
+                                   "org.apache.hadoop.mapred.HistoryViewer");
+  private static SimpleDateFormat dateFormat = new SimpleDateFormat(
+                                             "d-MMM-yyyy HH:mm:ss");
+  private FileSystem fs;
+  private Configuration conf;
+  private Path historyLogDir;
+  private String jobLogFile;
+  private JobHistory.JobInfo job;
+  private String trackerHostName;
+  private String trackerStartTime;
+  private String jobId;
+  private boolean printAll;
+  
+  private PathFilter jobLogFileFilter = new PathFilter() {
+    public boolean accept(Path path) {
+      return !(path.getName().endsWith(".xml"));
+    }
+  };
+
+  public HistoryViewer(String outputDir, Configuration conf, boolean printAll)
+  throws IOException {
+    this.conf = conf;
+    this.printAll = printAll;
+    Path output = new Path(outputDir);
+    historyLogDir = new Path(output, "_logs/history");
+    try {
+      fs = output.getFileSystem(this.conf);
+      if (!fs.exists(output)) {
+        throw new IOException("History directory " + historyLogDir.toString()
+                              + "does not exist");
+      }
+      Path[] jobFiles = FileUtil.stat2Paths(fs.listStatus(historyLogDir,
+                                                          jobLogFileFilter));
+      if (jobFiles.length == 0) {
+        throw new IOException("Not a valid history directory " 
+                              + historyLogDir.toString());
+      }
+      jobLogFile = jobFiles[0].toString();
+      String[] jobDetails = jobFiles[0].getName().split("_");
+      trackerHostName = jobDetails[0];
+      trackerStartTime = jobDetails[1];
+      jobId = jobDetails[2] + "_" + jobDetails[3] + "_" + jobDetails[4];
+      job = new JobHistory.JobInfo(jobId); 
+      DefaultJobHistoryParser.parseJobTasks(jobFiles[0].toString(), job, fs);
+    } catch(Exception e) {
+      throw new IOException("Not able to initialize History viewer");
+    }
+  }
+  
+  public void print() throws IOException{
+    printJobDetails();
+    printTaskSummary();
+    printJobAnalysis();
+    printTasks("MAP", "FAILED");
+    printTasks("MAP", "KILLED");
+    printTasks("REDUCE", "FAILED");
+    printTasks("REDUCE", "KILLED");
+    if (printAll) {
+      printTasks("MAP", "SUCCESS");
+      printTasks("REDUCE", "SUCCESS");
+      printAllTaskAttempts("MAP");
+      printAllTaskAttempts("REDUCE");
+    }
+    NodesFilter filter = new FailedOnNodesFilter();
+    printFailedAttempts(filter);
+    filter = new KilledOnNodesFilter();
+    printFailedAttempts(filter);
+  }
+
+  private void printJobDetails() {
+    StringBuffer jobDetails = new StringBuffer();
+    jobDetails.append("\nHadoop job: " ).append(jobId);
+    jobDetails.append("\n=====================================");
+    jobDetails.append("\nJob tracker host name: ").append(trackerHostName);
+    jobDetails.append("\njob tracker start time: ").append( 
+                      new Date(Long.parseLong(trackerStartTime))); 
+    jobDetails.append("\nUser: ").append(job.get(Keys.USER)); 
+    jobDetails.append("\nJobName: ").append(job.get(Keys.JOBNAME)); 
+    jobDetails.append("\nJobConf: ").append(job.get(Keys.JOBCONF)); 
+    jobDetails.append("\nSubmitted At: ").append(StringUtils.
+                        getFormattedTimeWithDiff(dateFormat,
+                        job.getLong(Keys.SUBMIT_TIME), 0)); 
+    jobDetails.append("\nLaunched At: ").append(StringUtils.
+                        getFormattedTimeWithDiff(dateFormat,
+                        job.getLong(Keys.LAUNCH_TIME),
+                        job.getLong(Keys.SUBMIT_TIME)));
+    jobDetails.append("\nFinished At: ").append(StringUtils.
+                        getFormattedTimeWithDiff(dateFormat,
+                        job.getLong(Keys.FINISH_TIME),
+                        job.getLong(Keys.LAUNCH_TIME)));
+    jobDetails.append("\nStatus: ").append(((job.get(Keys.JOB_STATUS) == "") ? 
+                      "Incomplete" :job.get(Keys.JOB_STATUS)));
+    jobDetails.append("\n=====================================");
+    System.out.println(jobDetails.toString());
+  }
+  
+  private void printTasks(String taskType, String taskStatus) {
+    Map<String, JobHistory.Task> tasks = job.getAllTasks();
+    StringBuffer taskList = new StringBuffer();
+    taskList.append("\n").append(taskStatus).append(" ");
+    taskList.append(taskType).append(" task list for ").append(jobId);
+    taskList.append("\nTaskId\t\tStartTime\tFinishTime\tError");
+    taskList.append("\n====================================================");
+    for (JobHistory.Task task : tasks.values()) {
+      if (taskType.equals(task.get(Keys.TASK_TYPE))){
+        Map <String, TaskAttempt> taskAttempts = task.getTaskAttempts();
+        for (JobHistory.TaskAttempt attempt : taskAttempts.values()) {
+          if (taskStatus.equals(attempt.get(Keys.TASK_STATUS))
+              || taskStatus.equals("all")){
+            taskList.append("\n").append(attempt.get(Keys.TASKID));
+            taskList.append("\t").append(StringUtils.getFormattedTimeWithDiff(
+                       dateFormat, attempt.getLong(Keys.START_TIME), 0));
+            taskList.append("\t").append(StringUtils.getFormattedTimeWithDiff(
+                       dateFormat, attempt.getLong(Keys.FINISH_TIME),
+                       task.getLong(Keys.START_TIME))); 
+            taskList.append("\t").append(attempt.get(Keys.ERROR));
+          }
+        }
+      }
+    }
+    System.out.println(taskList.toString());
+  }
+  
+  private void printAllTaskAttempts(String taskType) {
+    Map<String, JobHistory.Task> tasks = job.getAllTasks();
+    StringBuffer taskList = new StringBuffer();
+    taskList.append("\n").append(taskType);
+    taskList.append(" task list for ").append(jobId);
+    taskList.append("\nTaskId\t\tStartTime");
+    if (Values.REDUCE.name().equals(taskType)) {
+      taskList.append("\tShuffleFinished\tSortFinished");
+    }
+    taskList.append("\tFinishTime\tHostName\tError");
+    taskList.append("\n====================================================");
+    for (JobHistory.Task task : tasks.values()) {
+      for (JobHistory.TaskAttempt attempt : task.getTaskAttempts().values()) {
+        if (taskType.equals(task.get(Keys.TASK_TYPE))){
+          taskList.append("\n"); 
+          taskList.append(attempt.get(Keys.TASK_ATTEMPT_ID)).append("\t");
+          taskList.append(StringUtils.getFormattedTimeWithDiff(dateFormat,
+                          attempt.getLong(Keys.START_TIME), 0)).append("\t");
+          if (Values.REDUCE.name().equals(taskType)) {
+            ReduceAttempt reduceAttempt = (ReduceAttempt)attempt; 
+            taskList.append(StringUtils.getFormattedTimeWithDiff(dateFormat,
+                            reduceAttempt.getLong(Keys.SHUFFLE_FINISHED),
+                            reduceAttempt.getLong(Keys.START_TIME)));
+            taskList.append("\t"); 
+            taskList.append(StringUtils.getFormattedTimeWithDiff(dateFormat, 
+                            reduceAttempt.getLong(Keys.SORT_FINISHED),
+                            reduceAttempt.getLong(Keys.SHUFFLE_FINISHED))); 
+          } 
+          taskList.append(StringUtils.getFormattedTimeWithDiff(dateFormat,
+                          attempt.getLong(Keys.FINISH_TIME),
+                          attempt.getLong(Keys.START_TIME))); 
+          taskList.append("\t"); 
+          taskList.append(attempt.get(Keys.HOSTNAME)).append("\t");
+          taskList.append(attempt.get(Keys.ERROR));
+        }
+      }
+    }
+    taskList.append("\n");
+    System.out.println(taskList.toString());
+  }
+  
+  private void printTaskSummary() {
+    Map<String, JobHistory.Task> tasks = job.getAllTasks();
+    int totalMaps = 0; 
+    int totalReduces = 0; 
+    int numFailedMaps = 0; 
+    int numKilledMaps = 0;
+    int numFailedReduces = 0; 
+    int numKilledReduces = 0;
+    long mapStarted = 0; 
+    long mapFinished = 0; 
+    long reduceStarted = 0; 
+    long reduceFinished = 0; 
+
+    Map <String, String> allHosts = new TreeMap<String, String>();
+
+    for (JobHistory.Task task : tasks.values()) {
+      Map<String, TaskAttempt> attempts = task.getTaskAttempts();
+      allHosts.put(task.get(Keys.HOSTNAME), "");
+      for (TaskAttempt attempt : attempts.values()) {
+        long startTime = attempt.getLong(Keys.START_TIME); 
+        long finishTime = attempt.getLong(Keys.FINISH_TIME); 
+        if (Values.MAP.name().equals(task.get(Keys.TASK_TYPE))) {
+          if (mapStarted==0 || mapStarted > startTime) {
+            mapStarted = startTime; 
+          }
+          if (mapFinished < finishTime) {
+            mapFinished = finishTime; 
+          }
+          totalMaps++; 
+          if (Values.FAILED.name().equals(attempt.get(Keys.TASK_STATUS))) {
+            numFailedMaps++; 
+          } else if (Values.KILLED.name().equals(
+                                            attempt.get(Keys.TASK_STATUS))) {
+            numKilledMaps++;
+          }
+        } else {
+          if (reduceStarted==0||reduceStarted > startTime) {
+            reduceStarted = startTime; 
+          }
+          if (reduceFinished < finishTime) {
+            reduceFinished = finishTime; 
+          }
+          totalReduces++; 
+          if (Values.FAILED.name().equals(attempt.get(Keys.TASK_STATUS))) {
+            numFailedReduces++;
+          } else if (Values.KILLED.name().equals(
+                                            attempt.get(Keys.TASK_STATUS))) {
+            numKilledReduces++;
+          }
+        }
+      }
+    }
+    
+    StringBuffer taskSummary = new StringBuffer();
+    taskSummary.append("\nTask Summary");
+    taskSummary.append("\n============================");
+    taskSummary.append("\nKind\tTotal\t");
+    taskSummary.append("Successful\tFailed\tKilled\tStartTime\tFinishTime");
+    taskSummary.append("\n");
+    taskSummary.append("\nMap\t").append(totalMaps);
+    taskSummary.append("\t").append(job.getInt(Keys.FINISHED_MAPS));
+    taskSummary.append("\t\t").append(numFailedMaps);
+    taskSummary.append("\t").append(numKilledMaps);
+    taskSummary.append("\t").append(StringUtils.getFormattedTimeWithDiff(
+                               dateFormat, mapStarted, 0));
+    taskSummary.append("\t").append(StringUtils.getFormattedTimeWithDiff(
+                               dateFormat, mapFinished, mapStarted));
+    taskSummary.append("\nReduce\t").append(totalReduces);
+    taskSummary.append("\t").append(job.getInt(Keys.FINISHED_REDUCES));
+    taskSummary.append("\t\t").append(numFailedReduces);
+    taskSummary.append("\t").append(numKilledReduces);
+    taskSummary.append("\t").append(StringUtils.getFormattedTimeWithDiff(
+                               dateFormat, reduceStarted, 0));
+    taskSummary.append("\t").append(StringUtils.getFormattedTimeWithDiff(
+                               dateFormat, reduceFinished, reduceStarted)); 
+    taskSummary.append("\n============================\n");
+    System.out.println(taskSummary.toString());
+  }
+  
+  private void printFailedAttempts(NodesFilter filter) throws IOException {
+    JobHistory.parseHistoryFromFS(jobLogFile, filter, fs); 
+    Map<String, Set<String>> badNodes = filter.getValues();
+    StringBuffer attempts = new StringBuffer(); 
+    if (badNodes.size() > 0) {
+      attempts.append("\n").append(filter.getFailureType());
+      attempts.append(" task attempts by nodes");
+      attempts.append("\nHostname\tFailedTasks");
+      attempts.append("\n===============================");
+      for (Map.Entry<String, Set<String>> entry : badNodes.entrySet()) {
+        String node = entry.getKey();
+        Set<String> failedTasks = entry.getValue();
+        attempts.append("\n").append(node).append("\t");
+        for (String t : failedTasks) {
+          attempts.append(t).append(", ");
+        }
+      }
+    }
+    System.out.println(attempts.toString());
+  }
+  
+  private void printJobAnalysis() {
+    if (!Values.SUCCESS.name().equals(job.get(Keys.JOB_STATUS))) {
+      System.out.println("No Analysis available as job did not finish");
+      return;
+    }
+    
+    Map<String, JobHistory.Task> tasks = job.getAllTasks();
+    int finishedMaps = job.getInt(Keys.FINISHED_MAPS);
+    int finishedReduces = job.getInt(Keys.FINISHED_REDUCES);
+    JobHistory.Task [] mapTasks = new JobHistory.Task[finishedMaps]; 
+    JobHistory.Task [] reduceTasks = new JobHistory.Task[finishedReduces]; 
+    int mapIndex = 0 , reduceIndex=0; 
+    long avgMapTime = 0;
+    long avgReduceTime = 0;
+    long avgShuffleTime = 0;
+
+    for (JobHistory.Task task : tasks.values()) {
+      Map<String, TaskAttempt> attempts = task.getTaskAttempts();
+      for (JobHistory.TaskAttempt attempt : attempts.values()) {
+        if (attempt.get(Keys.TASK_STATUS).equals(Values.SUCCESS.name())) {
+          long avgFinishTime = (attempt.getLong(Keys.FINISH_TIME) -
+                                attempt.getLong(Keys.START_TIME));
+          if (Values.MAP.name().equals(task.get(Keys.TASK_TYPE))) {
+            mapTasks[mapIndex++] = attempt; 
+            avgMapTime += avgFinishTime;
+          } else { 
+            reduceTasks[reduceIndex++] = attempt;
+            avgShuffleTime += (attempt.getLong(Keys.SHUFFLE_FINISHED) - 
+                               attempt.getLong(Keys.START_TIME));
+            avgReduceTime += (attempt.getLong(Keys.FINISH_TIME) -
+                              attempt.getLong(Keys.SHUFFLE_FINISHED));
+          }
+          break;
+        }
+      }
+    }
+    if (finishedMaps > 0) {
+      avgMapTime /= finishedMaps;
+    }
+    if (finishedReduces > 0) {
+      avgReduceTime /= finishedReduces;
+      avgShuffleTime /= finishedReduces;
+    }
+    System.out.println("\nAnalysis");
+    System.out.println("=========");
+    printAnalysis(mapTasks, cMap, "map", avgMapTime, 10);
+    printLast(mapTasks, "map", cFinishMapRed);
+
+    if (reduceTasks.length > 0) {
+      printAnalysis(reduceTasks, cShuffle, "shuffle", avgShuffleTime, 10);
+      printLast(reduceTasks, "shuffle", cFinishShuffle);
+
+      printAnalysis(reduceTasks, cReduce, "reduce", avgReduceTime, 10);
+      printLast(reduceTasks, "reduce", cFinishMapRed);
+    }
+    System.out.println("=========");
+  }
+  
+  private void printLast(JobHistory.Task [] tasks,
+                         String taskType,
+                         Comparator<JobHistory.Task> cmp
+                         ) {
+    Arrays.sort(tasks, cFinishMapRed);
+    JobHistory.Task last = tasks[0];
+    StringBuffer lastBuf = new StringBuffer();
+    lastBuf.append("The last ").append(taskType);
+    lastBuf.append(" task ").append(last.get(Keys.TASKID));
+    Long finishTime;
+    if ("shuffle".equals(taskType)) {
+      finishTime = last.getLong(Keys.SHUFFLE_FINISHED);
+    } else {
+      finishTime = last.getLong(Keys.FINISH_TIME);
+    }
+    lastBuf.append(" finished at (relative to the Job launch time): ");
+    lastBuf.append(StringUtils.getFormattedTimeWithDiff(dateFormat,
+                                 finishTime, job.getLong(Keys.LAUNCH_TIME)));
+    System.out.println(lastBuf.toString());
+  }
+
+  private void printAnalysis(JobHistory.Task [] tasks,
+                             Comparator<JobHistory.Task> cmp,
+                             String taskType,
+                             long avg,
+                             int showTasks) {
+    Arrays.sort(tasks, cmp);
+    JobHistory.Task min = tasks[tasks.length-1];
+    StringBuffer details = new StringBuffer();
+    details.append("\nTime taken by best performing ");
+    details.append(taskType).append(" task ");
+    details.append(min.get(Keys.TASKID)).append(": ");
+    if ("map".equals(taskType)) {
+      details.append(StringUtils.formatTimeDiff(
+                     min.getLong(Keys.FINISH_TIME),
+                     min.getLong(Keys.START_TIME)));
+    } else if ("shuffle".equals(taskType)) {
+      details.append(StringUtils.formatTimeDiff(
+                     min.getLong(Keys.SHUFFLE_FINISHED),
+                     min.getLong(Keys.START_TIME)));
+    } else {
+      details.append(StringUtils.formatTimeDiff(
+                min.getLong(Keys.FINISH_TIME),
+                min.getLong(Keys.SHUFFLE_FINISHED)));
+    }
+    details.append("\nAverage time taken by ");
+    details.append(taskType).append(" tasks: "); 
+    details.append(StringUtils.formatTimeDiff(avg, 0));
+    details.append("\nWorse performing ");
+    details.append(taskType).append(" tasks: ");
+    details.append("\nTaskId\t\tTimetaken");
+    for (int i = 0; i < showTasks && i < tasks.length; i++) {
+      details.append("\n").append(tasks[i].get(Keys.TASKID)).append(" ");
+      if ("map".equals(taskType)) {
+        details.append(StringUtils.formatTimeDiff(
+                       tasks[i].getLong(Keys.FINISH_TIME),
+                       tasks[i].getLong(Keys.START_TIME)));
+      } else if ("shuffle".equals(taskType)) {
+        details.append(StringUtils.formatTimeDiff(
+                       tasks[i].getLong(Keys.SHUFFLE_FINISHED),
+                       tasks[i].getLong(Keys.START_TIME)));
+      } else {
+        details.append(StringUtils.formatTimeDiff(
+                       tasks[i].getLong(Keys.FINISH_TIME),
+                       tasks[i].getLong(Keys.SHUFFLE_FINISHED)));
+      }
+    }
+    System.out.println(details.toString());
+  }
+  
+  private Comparator<JobHistory.Task> cMap = 
+                                        new Comparator<JobHistory.Task>() {
+    public int compare(JobHistory.Task t1, JobHistory.Task t2) {
+      long l1 = t1.getLong(Keys.FINISH_TIME) - t1.getLong(Keys.START_TIME);
+      long l2 = t2.getLong(Keys.FINISH_TIME) - t2.getLong(Keys.START_TIME);
+      return (l2 < l1 ? -1 : (l2 == l1 ? 0 : 1));
+    }
+  };
+  
+  private Comparator<JobHistory.Task> cShuffle = 
+    new Comparator<JobHistory.Task>() {
+    public int compare(JobHistory.Task t1, JobHistory.Task t2) {
+      long l1 = t1.getLong(Keys.SHUFFLE_FINISHED) - 
+                t1.getLong(Keys.START_TIME);
+      long l2 = t2.getLong(Keys.SHUFFLE_FINISHED) -
+                t2.getLong(Keys.START_TIME);
+      return (l2 < l1 ? -1 : (l2 == l1 ? 0 : 1));
+    }
+  };
+
+  private Comparator<JobHistory.Task> cFinishShuffle = 
+    new Comparator<JobHistory.Task>() {
+    public int compare(JobHistory.Task t1, JobHistory.Task t2) {
+      long l1 = t1.getLong(Keys.SHUFFLE_FINISHED); 
+      long l2 = t2.getLong(Keys.SHUFFLE_FINISHED);
+      return (l2 < l1 ? -1 : (l2 == l1 ? 0 : 1));
+    }
+  };
+
+  private Comparator<JobHistory.Task> cFinishMapRed = 
+    new Comparator<JobHistory.Task>() {
+    public int compare(JobHistory.Task t1, JobHistory.Task t2) {
+      long l1 = t1.getLong(Keys.FINISH_TIME); 
+      long l2 = t2.getLong(Keys.FINISH_TIME);
+      return (l2 < l1 ? -1 : (l2 == l1 ? 0 : 1));
+    }
+  };
+  
+  private Comparator<JobHistory.Task> cReduce = 
+    new Comparator<JobHistory.Task>() {
+    public int compare(JobHistory.Task t1, JobHistory.Task t2) {
+      long l1 = t1.getLong(Keys.FINISH_TIME) -
+                t1.getLong(Keys.SHUFFLE_FINISHED);
+      long l2 = t2.getLong(Keys.FINISH_TIME) -
+                t2.getLong(Keys.SHUFFLE_FINISHED);
+      return (l2 < l1 ? -1 : (l2 == l1 ? 0 : 1));
+    }
+  }; 
+}

src/java/org/apache/hadoop/mapred/JobClient.java

@@ -61,6 +61,7 @@ import org.apache.hadoop.io.retry.RetryPolicy;
 import org.apache.hadoop.io.retry.RetryProxy;
 import org.apache.hadoop.ipc.RPC;
 import org.apache.hadoop.mapred.TaskInProgress;
+import org.apache.hadoop.mapred.DefaultJobHistoryParser.*;
 import org.apache.hadoop.net.NetUtils;
 import org.apache.hadoop.security.UnixUserGroupInformation;
 import org.apache.hadoop.util.StringUtils;
@@ -1034,6 +1035,7 @@ public class JobClient extends Configured implements MRConstants, Tool  {
     boolean killJob = false;
     boolean listEvents = false;
     boolean viewHistory = false;
+    boolean viewAllHistory = false;
     boolean listJobs = false;
     boolean listAllJobs = false;
     boolean killTask = false;
@@ -1064,10 +1066,15 @@ public class JobClient extends Configured implements MRConstants, Tool  {
       nEvents = Integer.parseInt(argv[3]);
       listEvents = true;
     } else if ("-history".equals(argv[0])) {
-      if (argv.length != 2)
-        displayUsage();
-        outputDir = argv[1];
-        viewHistory = true;
+      if (argv.length != 2 && !(argv.length == 3 && "all".equals(argv[1])))
+         displayUsage();
+      viewHistory = true;
+      if (argv.length == 3 && "all".equals(argv[1])) {
+         viewAllHistory = true;
+         outputDir = argv[2];
+      } else {
+         outputDir = argv[1];
+      }
     } else if ("-list".equals(argv[0])) {
       if (argv.length != 1 && !(argv.length == 2 && "all".equals(argv[1])))
         displayUsage();
@@ -1125,8 +1132,7 @@ public class JobClient extends Configured implements MRConstants, Tool  {
           exitCode = 0;
         }
       } else if (viewHistory) {
-    	// start http server
-        viewHistory(outputDir);
+        viewHistory(outputDir, viewAllHistory);
         exitCode = 0;
       } else if (listEvents) {
         listEvents(jobid, fromEvent, nEvents);
@@ -1160,43 +1166,11 @@ public class JobClient extends Configured implements MRConstants, Tool  {
     return exitCode;
   }
 
-  private void viewHistory(String outputDir) 
+  private void viewHistory(String outputDir, boolean all) 
     throws IOException {
-
-    Path output = new Path(outputDir);
-    FileSystem fs = output.getFileSystem(getConf());
-
-    // start http server used to provide an HTML view on Job history
-    StatusHttpServer infoServer;
-    String infoAddr = new JobConf(getConf()).get(
-             "mapred.job.history.http.bindAddress", "0.0.0.0:0");
-    InetSocketAddress infoSocAddr = NetUtils.createSocketAddr(infoAddr);
-    String infoBindAddress = infoSocAddr.getHostName();
-    int tmpInfoPort = infoSocAddr.getPort();
-    infoServer = new StatusHttpServer("history", infoBindAddress, tmpInfoPort,
-                                       tmpInfoPort == 0);
-    infoServer.setAttribute("fileSys", fs);
-    infoServer.setAttribute("historyLogDir", outputDir + "/_logs/history");
-    infoServer.start();
-    int infoPort = infoServer.getPort();
-    getConf().set("mapred.job.history.http.bindAddress", 
-        infoBindAddress + ":" + infoPort);
-    LOG.info("JobHistory webserver up at: " + infoPort);
-
-    // let the server be up for 30 minutes.
-    try {
-      Thread.sleep(30 * 60 * 1000);
-    } catch (InterruptedException ie) {}
-      
-    // stop infoServer
-    if (infoServer != null) {
-      LOG.info("Stopping infoServer");
-      try {
-        infoServer.stop();
-      } catch (InterruptedException ex) {
-        ex.printStackTrace();
-      }
-    } 
+    HistoryViewer historyViewer = new HistoryViewer(outputDir,
+                                        getConf(), all);
+    historyViewer.print();
   }
   
   /**

src/java/org/apache/hadoop/mapred/JobHistory.java

@@ -111,29 +111,33 @@ public class JobHistory {
 
   /**
    * Initialize JobHistory files. 
-   *
+   * @param conf Jobconf of the job tracker.
+   * @param hostname jobtracker's hostname
+   * @return true if intialized properly
+   *         false otherwise
    */
-  public static void init(JobConf conf, String hostname){
-    if (!disableHistory){
-      try{
-        LOG_DIR = conf.get("hadoop.job.history.location" ,
-          "file:///" + new File(System.getProperty(
-          "hadoop.log.dir")).getAbsolutePath() + File.separator + "history");
-        JOBTRACKER_UNIQUE_STRING = hostname + "_" + 
+  public static boolean init(JobConf conf, String hostname){
+    try {
+      LOG_DIR = conf.get("hadoop.job.history.location" ,
+        "file:///" + new File(
+        System.getProperty("hadoop.log.dir")).getAbsolutePath()
+        + File.separator + "history");
+      JOBTRACKER_UNIQUE_STRING = hostname + "_" + 
                                    JOBTRACKER_START_TIME + "_";
-        Path logDir = new Path(LOG_DIR);
-        FileSystem fs = logDir.getFileSystem(conf);
-        if (!fs.exists(logDir)){
-          if (!fs.mkdirs(logDir)){
-            throw new IOException("Mkdirs failed to create " + logDir.toString());
-          }
+      Path logDir = new Path(LOG_DIR);
+      FileSystem fs = logDir.getFileSystem(conf);
+      if (!fs.exists(logDir)){
+        if (!fs.mkdirs(logDir)){
+          throw new IOException("Mkdirs failed to create " + logDir.toString());
         }
-        conf.set("hadoop.job.history.location", LOG_DIR);
-      }catch(IOException e){
-        LOG.error("Failed to initialize JobHistory log file", e); 
-        disableHistory = true; 
       }
+      conf.set("hadoop.job.history.location", LOG_DIR);
+      disableHistory = false;
+    } catch(IOException e) {
+        LOG.error("Failed to initialize JobHistory log file", e); 
+        disableHistory = true;
     }
+    return !(disableHistory);
   }
 
   /**

src/java/org/apache/hadoop/mapred/JobTracker.java

@@ -611,7 +611,6 @@ public class JobTracker implements MRConstants, InterTrackerProtocol, JobSubmiss
 
   // Used to provide an HTML view on Job, Task, and TaskTracker structures
   StatusHttpServer infoServer;
-  StatusHttpServer historyServer;
   int infoPort;
 
   Server interTrackerServer;
@@ -676,6 +675,16 @@ public class JobTracker implements MRConstants, InterTrackerProtocol, JobSubmiss
     infoServer = new StatusHttpServer("job", infoBindAddress, tmpInfoPort, 
                                       tmpInfoPort == 0);
     infoServer.setAttribute("job.tracker", this);
+    // initialize history parameters.
+    boolean historyInitialized = JobHistory.init(conf, this.localMachine);
+    String historyLogDir = null;
+    FileSystem historyFS = null;
+    if (historyInitialized) {
+      historyLogDir = conf.get("hadoop.job.history.location");
+      infoServer.setAttribute("historyLogDir", historyLogDir);
+      historyFS = new Path(historyLogDir).getFileSystem(conf);
+      infoServer.setAttribute("fileSys", historyFS);
+    }
     infoServer.start();
 
     this.startTime = System.currentTimeMillis();
@@ -719,29 +728,19 @@ public class JobTracker implements MRConstants, InterTrackerProtocol, JobSubmiss
       }
       Thread.sleep(SYSTEM_DIR_CLEANUP_RETRY_PERIOD);
     }
-
-    // start history viewing server.
-    JobHistory.init(conf, this.localMachine); 
-    String histAddr = conf.get("mapred.job.history.http.bindAddress",
-                                  "0.0.0.0:0");
-    InetSocketAddress historySocAddr = NetUtils.createSocketAddr(histAddr);
-    String historyBindAddress = historySocAddr.getHostName();
-    int tmpHistoryPort = historySocAddr.getPort();
-    historyServer = new StatusHttpServer("history", historyBindAddress, 
-                       tmpHistoryPort, tmpHistoryPort == 0);
-    String historyLogDir = conf.get("hadoop.job.history.location");
-    historyServer.setAttribute("historyLogDir", historyLogDir);
-    FileSystem fileSys = new Path(historyLogDir).getFileSystem(conf);
-    historyServer.setAttribute("fileSys", fileSys);
-    historyServer.start();
-    this.conf.set("mapred.job.history.http.bindAddress", 
-                (this.localMachine + ":" + historyServer.getPort()));
-    LOG.info("JobHistory webserver on JobTracker up at: " +
-              historyServer.getPort());
-
-
     // Same with 'localDir' except it's always on the local disk.
     jobConf.deleteLocalFiles(SUBDIR);
+
+    // Initialize history again if it is not initialized
+    // because history was on dfs and namenode was in safemode.
+    if (!historyInitialized) {
+      JobHistory.init(conf, this.localMachine); 
+      historyLogDir = conf.get("hadoop.job.history.location");
+      infoServer.setAttribute("historyLogDir", historyLogDir);
+      historyFS = new Path(historyLogDir).getFileSystem(conf);
+      infoServer.setAttribute("fileSys", historyFS);
+    }
+
     this.dnsToSwitchMapping = (DNSToSwitchMapping)ReflectionUtils.newInstance(
         conf.getClass("topology.node.switch.mapping.impl", ScriptBasedMapping.class,
             DNSToSwitchMapping.class), conf);
@@ -763,10 +762,6 @@ public class JobTracker implements MRConstants, InterTrackerProtocol, JobSubmiss
     return NetUtils.createSocketAddr(jobTrackerStr);
   }
 
-  public String getHistoryAddress() {
-    return conf.get("mapred.job.history.http.bindAddress");
-  }
-
   /**
    * Run forever
    */
@@ -802,14 +797,6 @@ public class JobTracker implements MRConstants, InterTrackerProtocol, JobSubmiss
         ex.printStackTrace();
       }
     }
-    if (this.historyServer != null) {
-      LOG.info("Stopping historyServer");
-      try {
-        this.historyServer.stop();
-      } catch (InterruptedException ex) {
-        ex.printStackTrace();
-      }
-    }
     if (this.interTrackerServer != null) {
       LOG.info("Stopping interTrackerServer");
       this.interTrackerServer.stop();

src/webapps/history/analysejobhistory.jsp

@@ -1,247 +0,0 @@
-<%@ page
-  contentType="text/html; charset=UTF-8"
-  import="javax.servlet.http.*"
-  import="java.io.*"
-  import="java.util.*"
-  import="org.apache.hadoop.mapred.*"
-  import="org.apache.hadoop.util.*"
-  import="java.text.SimpleDateFormat"
-  import="org.apache.hadoop.mapred.JobHistory.*"
-%>
-<jsp:include page="loadhistory.jsp">
-	<jsp:param name="jobid" value="<%=request.getParameter("jobid") %>"/>
-	<jsp:param name="logFile" value="<%=request.getParameter("logFile") %>"/>
-</jsp:include>
-<%!	private static SimpleDateFormat dateFormat = new SimpleDateFormat("d/MM HH:mm:ss") ; %>
-<html><body>
-<%
-	String jobid = request.getParameter("jobid");
-	String logFile = request.getParameter("logFile");
-	String numTasks = request.getParameter("numTasks");
-	int showTasks = 10 ; 
-	if( numTasks != null ) {
-	  showTasks = Integer.parseInt(numTasks);  
-	}
-
-	JobInfo job = (JobInfo)request.getSession().getAttribute("job");
-
-%>
-<h2>Hadoop Job <a href="jobdetailshistory.jsp?jobid=<%=jobid%>&&logFile=<%=logFile %>"><%=jobid %> </a></h2>
-
-<b>User : </b> <%=job.get(Keys.USER) %><br/> 
-<b>JobName : </b> <%=job.get(Keys.JOBNAME) %><br/> 
-<b>JobConf : </b> <%=job.get(Keys.JOBCONF) %><br/> 
-<b>Submitted At : </b> <%=StringUtils.getFormattedTimeWithDiff(dateFormat, job.getLong(Keys.SUBMIT_TIME), 0 ) %><br/> 
-<b>Launched At : </b> <%=StringUtils.getFormattedTimeWithDiff(dateFormat, job.getLong(Keys.LAUNCH_TIME), job.getLong(Keys.SUBMIT_TIME)) %><br/>
-<b>Finished At : </b>  <%=StringUtils.getFormattedTimeWithDiff(dateFormat, job.getLong(Keys.FINISH_TIME), job.getLong(Keys.LAUNCH_TIME)) %><br/>
-<b>Status : </b> <%= ((job.get(Keys.JOB_STATUS) == null)?"Incomplete" :job.get(Keys.JOB_STATUS)) %><br/> 
-<hr/>
-<center>
-<%
-	if( ! Values.SUCCESS.name().equals(job.get(Keys.JOB_STATUS)) ){
-	  out.print("<h3>No Analysis available as job did not finish</h3>");
-	  return ;
-	}
-	Map<String, JobHistory.Task> tasks = job.getAllTasks();
-	int finishedMaps = job.getInt(Keys.FINISHED_MAPS)  ;
-	int finishedReduces = job.getInt(Keys.FINISHED_REDUCES) ;
-	JobHistory.Task [] mapTasks = new JobHistory.Task[finishedMaps]; 
-	JobHistory.Task [] reduceTasks = new JobHistory.Task[finishedReduces]; 
-	int mapIndex = 0 , reduceIndex=0; 
-	long avgMapTime = 0;
-	long avgReduceTime = 0;
-	long avgShuffleTime = 0;
-	
-	for( JobHistory.Task task : tasks.values() ) {
-	  long avgFinishTime = (task.getLong(Keys.FINISH_TIME) - 
-                      task.getLong(Keys.START_TIME));
-	  if( Values.MAP.name().equals(task.get(Keys.TASK_TYPE)) ){
-		  mapTasks[mapIndex++] = task ; 
-		  avgMapTime += avgFinishTime;
-	  }else{ 
-	    Map<String, TaskAttempt> attempts = task.getTaskAttempts();
-	    for (JobHistory.TaskAttempt attempt : attempts.values()) {
-	      if (attempt.get(Keys.TASK_STATUS).equals(Values.SUCCESS.name())) {
-	        reduceTasks[reduceIndex++] = attempt;
-   	        avgShuffleTime += (attempt.getLong(Keys.SHUFFLE_FINISHED) - 
-                               attempt.getLong(Keys.START_TIME));
-   	        avgReduceTime += (attempt.getLong(Keys.FINISH_TIME) - 
-                              attempt.getLong(Keys.SHUFFLE_FINISHED));
-   	        break;
-	      }
-	    }
-	  }
-	}
-	 
-	if (finishedMaps > 0) {
-	  avgMapTime /= finishedMaps;
-	}
-	if (finishedReduces > 0) {
-	  avgReduceTime /= finishedReduces;
-	  avgShuffleTime /= finishedReduces;
-	}
-	Comparator<JobHistory.Task> cMap = new Comparator<JobHistory.Task>(){
-	  public int compare(JobHistory.Task t1, JobHistory.Task t2){
-	    long l1 = t1.getLong(Keys.FINISH_TIME) - t1.getLong(Keys.START_TIME); 
-	    long l2 = t2.getLong(Keys.FINISH_TIME) - t2.getLong(Keys.START_TIME);
-      return (l2<l1 ? -1 : (l2==l1 ? 0 : 1));
-	  }
-	}; 
-	Comparator<JobHistory.Task> cShuffle = new Comparator<JobHistory.Task>(){
-	  public int compare(JobHistory.Task t1, JobHistory.Task t2){
-	    long l1 = t1.getLong(Keys.SHUFFLE_FINISHED) - 
-	                       t1.getLong(Keys.START_TIME); 
-	    long l2 = t2.getLong(Keys.SHUFFLE_FINISHED) - 
-	                       t2.getLong(Keys.START_TIME);
-      return (l2<l1 ? -1 : (l2==l1 ? 0 : 1));
-	  }
-	}; 
-	Arrays.sort(mapTasks, cMap);
-	Arrays.sort(reduceTasks, cShuffle); 
-	
-	JobHistory.Task minMap = mapTasks[mapTasks.length-1] ;
-	JobHistory.Task minShuffle = reduceTasks[reduceTasks.length-1] ;
-	
-%>
-
-<h3>Time taken by best performing Map task 
-<a href="taskdetailshistory.jsp?jobid=<%=jobid%>&logFile=<%=logFile%>&taskid=<%=minMap.get(Keys.TASKID)%>">
-<%=minMap.get(Keys.TASKID) %></a> : <%=StringUtils.formatTimeDiff(minMap.getLong(Keys.FINISH_TIME), minMap.getLong(Keys.START_TIME) ) %></h3>
-<h3>Average time taken by Map tasks: 
-<%=StringUtils.formatTimeDiff(avgMapTime, 0) %></h3>
-<h3>Worse performing map tasks</h3>
-<table border="2" cellpadding="5" cellspacing="2">
-<tr><td>Task Id</td><td>Time taken</td></tr>
-<%
-	for( int i=0;i<showTasks && i<mapTasks.length; i++){
-%>
-		<tr>
-			<td><a href="taskdetailshistory.jsp?jobid=<%=jobid%>&logFile=<%=logFile%>&taskid=<%=mapTasks[i].get(Keys.TASKID)%>">
-  		    <%=mapTasks[i].get(Keys.TASKID) %></a></td>
-			<td><%=StringUtils.formatTimeDiff(mapTasks[i].getLong(Keys.FINISH_TIME), mapTasks[i].getLong(Keys.START_TIME)) %></td>
-		</tr>
-<%
-	}
-%>
-</table>
-<%  
-    Comparator<JobHistory.Task> cFinishMapRed = 
-      new Comparator<JobHistory.Task>() {
-      public int compare(JobHistory.Task t1, JobHistory.Task t2){
-        long l1 = t1.getLong(Keys.FINISH_TIME); 
-        long l2 = t2.getLong(Keys.FINISH_TIME);
-        return (l2<l1 ? -1 : (l2==l1 ? 0 : 1));
-      }
-    };
-    Arrays.sort(mapTasks, cFinishMapRed);
-    JobHistory.Task lastMap = mapTasks[0] ;
-%>
-<h3>The last Map task 
-<a href="taskdetailshistory.jsp?jobid=<%=jobid%>&logFile=<%=logFile%>
-&taskid=<%=lastMap.get(Keys.TASKID)%>"><%=lastMap.get(Keys.TASKID) %></a> 
-finished at (relative to the Job launch time): 
-<%=StringUtils.getFormattedTimeWithDiff(dateFormat, 
-                              lastMap.getLong(Keys.FINISH_TIME), 
-                              job.getLong(Keys.LAUNCH_TIME) ) %></h3>
-<hr/>
-<h3>Time taken by best performing shuffle
-<a href="taskdetailshistory.jsp?jobid=<%=jobid%>&logFile=<%=logFile%>
-&taskid=<%=minShuffle.get(Keys.TASKID)%>"><%=minShuffle.get(Keys.TASKID)%></a> : 
-<%=StringUtils.formatTimeDiff(minShuffle.getLong(Keys.SHUFFLE_FINISHED), 
-                              minShuffle.getLong(Keys.START_TIME) ) %></h3>
-<h3>Average time taken by Shuffle: 
-<%=StringUtils.formatTimeDiff(avgShuffleTime, 0) %></h3>
-<h3>Worse performing Shuffle(s)</h3>
-<table border="2" cellpadding="5" cellspacing="2">
-<tr><td>Task Id</td><td>Time taken</td></tr>
-<%
-	for( int i=0;i<showTasks && i<reduceTasks.length; i++){
-%>
-      <tr>
-	    <td><a href="taskdetailshistory.jsp?jobid=<%=jobid%>&logFile=
-	    <%=logFile%>&taskid=<%=reduceTasks[i].get(Keys.TASKID)%>">
-			<%=reduceTasks[i].get(Keys.TASKID) %></a></td>
-	    <td><%=
-	      StringUtils.formatTimeDiff(
-	                       reduceTasks[i].getLong(Keys.SHUFFLE_FINISHED),
-	                       reduceTasks[i].getLong(Keys.START_TIME)) %>
-	    </td>
-	  </tr>
-<%
-	}
-%>
-</table>
-<%  
-    Comparator<JobHistory.Task> cFinishShuffle = 
-      new Comparator<JobHistory.Task>() {
-      public int compare(JobHistory.Task t1, JobHistory.Task t2){
-        long l1 = t1.getLong(Keys.SHUFFLE_FINISHED); 
-        long l2 = t2.getLong(Keys.SHUFFLE_FINISHED);
-        return (l2<l1 ? -1 : (l2==l1 ? 0 : 1));
-      }
-    };
-    Arrays.sort(reduceTasks, cFinishShuffle);
-    JobHistory.Task lastShuffle = reduceTasks[0] ;
-%>
-
-<h3>The last Shuffle  
-<a href="taskdetailshistory.jsp?jobid=<%=jobid%>&logFile=<%=logFile%>
-&taskid=<%=lastShuffle.get(Keys.TASKID)%>"><%=lastShuffle.get(Keys.TASKID)%>
-</a> finished at (relative to the Job launch time): 
-<%=StringUtils.getFormattedTimeWithDiff(dateFormat,
-                              lastShuffle.getLong(Keys.SHUFFLE_FINISHED), 
-                              job.getLong(Keys.LAUNCH_TIME) ) %></h3>
-
-<%
-	Comparator<JobHistory.Task> cReduce = new Comparator<JobHistory.Task>(){
-	  public int compare(JobHistory.Task t1, JobHistory.Task t2){
-	    long l1 = t1.getLong(Keys.FINISH_TIME) - 
-	                       t1.getLong(Keys.SHUFFLE_FINISHED); 
-	    long l2 = t2.getLong(Keys.FINISH_TIME) - 
-	                       t2.getLong(Keys.SHUFFLE_FINISHED);
-      return (l2<l1 ? -1 : (l2==l1 ? 0 : 1));
-	  }
-	}; 
-	Arrays.sort(reduceTasks, cReduce); 
-	JobHistory.Task minReduce = reduceTasks[reduceTasks.length-1] ;
-%>
-<hr/>
-<h3>Time taken by best performing Reduce task : 
-<a href="taskdetailshistory.jsp?jobid=<%=jobid%>&logFile=<%=logFile%>&taskid=<%=minReduce.get(Keys.TASKID)%>">
-<%=minReduce.get(Keys.TASKID) %></a> : 
-<%=StringUtils.formatTimeDiff(minReduce.getLong(Keys.FINISH_TIME),
-    minReduce.getLong(Keys.SHUFFLE_FINISHED) ) %></h3>
-
-<h3>Average time taken by Reduce tasks: 
-<%=StringUtils.formatTimeDiff(avgReduceTime, 0) %></h3>
-<h3>Worse performing reduce tasks</h3>
-<table border="2" cellpadding="5" cellspacing="2">
-<tr><td>Task Id</td><td>Time taken</td></tr>
-<%
-	for( int i=0;i<showTasks && i<reduceTasks.length; i++){
-%>
-		<tr>
-			<td><a href="taskdetailshistory.jsp?jobid=<%=jobid%>&logFile=<%=logFile%>&taskid=<%=reduceTasks[i].get(Keys.TASKID)%>">
-			<%=reduceTasks[i].get(Keys.TASKID) %></a></td>
-			<td><%=StringUtils.formatTimeDiff(
-			    reduceTasks[i].getLong(Keys.FINISH_TIME), 
-			    reduceTasks[i].getLong(Keys.SHUFFLE_FINISHED)) %></td>
-		</tr>
-<%
-	}
-%>
-</table>
-<%  
-    Arrays.sort(reduceTasks, cFinishMapRed);
-    JobHistory.Task lastReduce = reduceTasks[0] ;
-%>
-
-<h3>The last Reduce task 
-<a href="taskdetailshistory.jsp?jobid=<%=jobid%>&logFile=<%=logFile%>
-&taskid=<%=lastReduce.get(Keys.TASKID)%>"><%=lastReduce.get(Keys.TASKID)%>
-</a> finished at (relative to the Job launch time): 
-<%=StringUtils.getFormattedTimeWithDiff(dateFormat,
-                              lastReduce.getLong(Keys.FINISH_TIME), 
-                              job.getLong(Keys.LAUNCH_TIME) ) %></h3>
- </center>
- </body></html>

src/webapps/history/index.html

@@ -1,20 +0,0 @@
-<meta HTTP-EQUIV="REFRESH" content="0;url=jobhistory.jsp"/>
-<html>
-
-<head>
-<title>Hadoop Administration - History</title>
-</head>
-
-<body>
-
-<h1>Hadoop Administration - History</h1>
-
-<ul>
-
-<li><a href="jobhistory.jsp">Job History</a></li>
-
-</ul>
-
-</body>
-
-</html>

src/webapps/history/jobdetailshistory.jsp

@@ -1,201 +0,0 @@
-<%@ page
-  contentType="text/html; charset=UTF-8"
-  import="javax.servlet.http.*"
-  import="java.io.*"
-  import="java.util.*"
-  import="org.apache.hadoop.fs.*"
-  import="org.apache.hadoop.mapred.*"
-  import="org.apache.hadoop.util.*"
-  import="java.text.SimpleDateFormat"
-  import="org.apache.hadoop.mapred.JobHistory.*"
-%>
-<jsp:include page="loadhistory.jsp">
-	<jsp:param name="jobid" value="<%=request.getParameter("jobid") %>"/>
-	<jsp:param name="logFile" value="<%=request.getParameter("logFile") %>"/>
-</jsp:include>
-<%! static SimpleDateFormat dateFormat = new SimpleDateFormat("d-MMM-yyyy HH:mm:ss") ; %>
-<%
-	String jobid = request.getParameter("jobid");
-	String logFile = request.getParameter("logFile");
-	
-	Path jobFile = new Path(logFile);
-	String[] jobDetails = jobFile.getName().split("_");
-    String jobUniqueString = jobDetails[0] + "_" +jobDetails[1] + "_" + jobid ;
-	
-	JobInfo job = (JobInfo)request.getSession().getAttribute("job");
-	FileSystem fs = (FileSystem)request.getSession().getAttribute("fs");
-%>
-<html><body>
-<h2>Hadoop Job <%=jobid %> on <a href="jobhistory.jsp">History Viewer</a></h2>
-
-<b>User: </b> <%=job.get(Keys.USER) %><br/> 
-<b>JobName: </b> <%=job.get(Keys.JOBNAME) %><br/> 
-<b>JobConf: </b> <a href="jobconf_history.jsp?jobid=<%=jobid%>&jobLogDir=<%=new Path(logFile).getParent().toString()%>&jobUniqueString=<%=jobUniqueString%>"> 
-                 <%=job.get(Keys.JOBCONF) %></a><br/> 
-<b>Submitted At: </b> <%=StringUtils.getFormattedTimeWithDiff(dateFormat, job.getLong(Keys.SUBMIT_TIME), 0 )  %><br/> 
-<b>Launched At: </b> <%=StringUtils.getFormattedTimeWithDiff(dateFormat, job.getLong(Keys.LAUNCH_TIME), job.getLong(Keys.SUBMIT_TIME)) %><br/>
-<b>Finished At: </b>  <%=StringUtils.getFormattedTimeWithDiff(dateFormat, job.getLong(Keys.FINISH_TIME), job.getLong(Keys.LAUNCH_TIME)) %><br/>
-<b>Status: </b> <%= ((job.get(Keys.JOB_STATUS) == "")?"Incomplete" :job.get(Keys.JOB_STATUS)) %><br/> 
-<%
-	Map<String, JobHistory.Task> tasks = job.getAllTasks();
-	int totalMaps = 0 ; 
-	int totalReduces = 0; 
-	int failedMaps = 0; 
-	int killedMaps = 0;
-	int failedReduces = 0 ; 
-	int killedReduces = 0;
-	
-	long mapStarted = 0 ; 
-	long mapFinished = 0 ; 
-	long reduceStarted = 0 ; 
-	long reduceFinished = 0; 
-        
-        Map <String,String> allHosts = new TreeMap<String,String>();
-	
-	for( JobHistory.Task task : tasks.values() ) {
-	  
-	  long startTime = task.getLong(Keys.START_TIME) ; 
-	  long finishTime = task.getLong(Keys.FINISH_TIME) ; 
-	  
-          allHosts.put(task.get(Keys.HOSTNAME), "");
-
-	  if( Values.MAP.name().equals(task.get(Keys.TASK_TYPE)) ){
-	    if( mapStarted==0 || mapStarted > startTime ){
-	      mapStarted = startTime; 
-	    }
-	    if(  mapFinished < finishTime ){
-	      mapFinished = finishTime ; 
-	    }
-	    
-	    Map<String, TaskAttempt> attempts = task.getTaskAttempts();
-	    for( TaskAttempt attempt : attempts.values() ) {
-	        totalMaps++; 
-	        if( Values.FAILED.name().equals(attempt.get(Keys.TASK_STATUS)) ) {
-	            failedMaps++; 
-	        }
-	        if( Values.KILLED.name().equals(attempt.get(Keys.TASK_STATUS)) ) {
-	            killedMaps++; 
-	        }
-	    }
-	  }else{
-	    if( reduceStarted==0||reduceStarted > startTime ){
-	      reduceStarted = startTime ; 
-	    }
-	    if(  reduceFinished < finishTime ){
-	      reduceFinished = finishTime; 
-	    }
-	    Map<String, TaskAttempt> attempts = task.getTaskAttempts();
-	    for( TaskAttempt attempt : attempts.values() ) {
-	        totalReduces++; 
-	        if( Values.FAILED.name().equals(attempt.get(Keys.TASK_STATUS)) ) {
-	            failedReduces++; 
-	        }
-	        if( Values.KILLED.name().equals(attempt.get(Keys.TASK_STATUS)) ) {
-	            killedReduces++; 
-	        }
-	    }
-	  }
-	}
-%>
-<b><a href="analysejobhistory.jsp?jobid=<%=jobid %>&logFile=<%=logFile %>">Analyse This Job</a></b> 
-<hr/>
-<center>
-<table border="2" cellpadding="5" cellspacing="2">
-<tr>
-<td>Kind</td><td>Total Tasks(successful+failed+killed)</td><td>Successful tasks</td><td>Failed tasks</td><td>Killed tasks</td><td>Start Time</td><td>Finish Time</td>
-</tr>
-<tr>
-<td>Map</td>
-	<td><a href="jobtaskshistory.jsp?jobid=<%=jobid %>&logFile=<%=logFile %>&taskType=<%=Values.MAP.name() %>&status=all">
-	  <%=totalMaps %></a></td>
-	<td><a href="jobtaskshistory.jsp?jobid=<%=jobid %>&logFile=<%=logFile %>&taskType=<%=Values.MAP.name() %>&status=<%=Values.SUCCESS %>">
-	  <%=job.getInt(Keys.FINISHED_MAPS) %></a></td>
-	<td><a href="jobtaskshistory.jsp?jobid=<%=jobid %>&logFile=<%=logFile %>&taskType=<%=Values.MAP.name() %>&status=<%=Values.FAILED %>">
-	  <%=failedMaps %></a></td>
-	<td><a href="jobtaskshistory.jsp?jobid=<%=jobid %>&logFile=<%=logFile %>&taskType=<%=Values.MAP.name() %>&status=<%=Values.KILLED %>">
-	  <%=killedMaps %></a></td>
-	<td><%=StringUtils.getFormattedTimeWithDiff(dateFormat, mapStarted, 0) %></td>
-	<td><%=StringUtils.getFormattedTimeWithDiff(dateFormat, mapFinished, mapStarted) %></td>
-</tr>
-<tr>
-<td>Reduce</td>
-	<td><a href="jobtaskshistory.jsp?jobid=<%=jobid %>&logFile=<%=logFile %>&taskType=<%=Values.REDUCE.name() %>&status=all">
-	  <%=totalReduces%></a></td>
-	<td><a href="jobtaskshistory.jsp?jobid=<%=jobid %>&logFile=<%=logFile %>&taskType=<%=Values.REDUCE.name() %>&status=<%=Values.SUCCESS %>">
-	  <%=job.getInt(Keys.FINISHED_REDUCES)%></a></td>
-	<td><a href="jobtaskshistory.jsp?jobid=<%=jobid %>&logFile=<%=logFile %>&taskType=<%=Values.REDUCE.name() %>&status=<%=Values.FAILED %>">
-	  <%=failedReduces%></a></td>
-	<td><a href="jobtaskshistory.jsp?jobid=<%=jobid %>&logFile=<%=logFile %>&taskType=<%=Values.REDUCE.name() %>&status=<%=Values.KILLED %>">
-	  <%=killedReduces%></a></td>  
-	<td><%=StringUtils.getFormattedTimeWithDiff(dateFormat, reduceStarted, 0) %></td>
-	<td><%=StringUtils.getFormattedTimeWithDiff(dateFormat, reduceFinished, reduceStarted) %></td>
-</tr>
- </table>
-
-<br/>
- <%
-	DefaultJobHistoryParser.FailedOnNodesFilter filter = new DefaultJobHistoryParser.FailedOnNodesFilter();
-	JobHistory.parseHistoryFromFS(logFile, filter, fs); 
-	Map<String, Set<String>> badNodes = filter.getValues(); 
-	if( badNodes.size() > 0 ) {
- %>
-<h3>Failed tasks attempts by nodes </h3>
-<table border="1">
-<tr><td>Hostname</td><td>Failed Tasks</td></tr>
- <%	  
-  for (Map.Entry<String, Set<String>> entry : badNodes.entrySet()) {
-    String node = entry.getKey();
-    Set<String> failedTasks = entry.getValue();
-%>
-	<tr>
-		<td><%=node %></td>
-		<td>
-<%
-		for( String t : failedTasks ) {
-%>
-		 <a href="taskdetailshistory.jsp?jobid=<%=jobid%>&logFile=<%=logFile %>&taskid=<%=t %>"><%=t %></a>,&nbsp;
-<%		  
-		}
-%>	
-		</td>
-	</tr>
-<%	  
-     }
-	}
- %>
-</table>
-<br/>
- <%
-	DefaultJobHistoryParser.KilledOnNodesFilter killedFilter = new DefaultJobHistoryParser.KilledOnNodesFilter();
-	JobHistory.parseHistoryFromFS(logFile, filter, fs); 
-	badNodes = killedFilter.getValues(); 
-	if( badNodes.size() > 0 ) {
- %>
-<h3>Killed tasks attempts by nodes </h3>
-<table border="1">
-<tr><td>Hostname</td><td>Killed Tasks</td></tr>
- <%	  
-  for (Map.Entry<String, Set<String>> entry : badNodes.entrySet()) {
-    String node = entry.getKey();
-    Set<String> killedTasks = entry.getValue();
-%>
-	<tr>
-		<td><%=node %></td>
-		<td>
-<%
-		for( String t : killedTasks ) {
-%>
-		 <a href="taskdetailshistory.jsp?jobid=<%=jobid%>&logFile=<%=logFile %>&taskid=<%=t %>"><%=t %></a>,&nbsp;
-<%		  
-		}
-%>	
-		</td>
-	</tr>
-<%	  
-     }
-	}
- %>
-</table>
- </center>
-
-</body></html>

src/webapps/history/jobtaskshistory.jsp

@@ -1,62 +0,0 @@
-<%@ page
-  contentType="text/html; charset=UTF-8"
-  import="javax.servlet.http.*"
-  import="java.io.*"
-  import="java.util.*"
-  import="org.apache.hadoop.mapred.*"
-  import="org.apache.hadoop.util.*"
-  import="java.text.SimpleDateFormat"
-  import="org.apache.hadoop.mapred.JobHistory.*"
-%>
-<jsp:include page="loadhistory.jsp">
-	<jsp:param name="jobid" value="<%=request.getParameter("jobid") %>"/>
-	<jsp:param name="logFile" value="<%=request.getParameter("logFile") %>"/>
-</jsp:include>
-<%!	private static SimpleDateFormat dateFormat = new SimpleDateFormat("d/MM HH:mm:ss") ; %>
-
-<%	
-	String jobid = request.getParameter("jobid");
-	String logFile = request.getParameter("logFile");
-	String taskStatus = request.getParameter("status"); 
-	String taskType = request.getParameter("taskType"); 
-	
-	JobHistory.JobInfo job = (JobHistory.JobInfo)request.getSession().getAttribute("job");
-	
-	Map<String, JobHistory.Task> tasks = job.getAllTasks(); 
-%>
-<html>
-<body>
-<h2><%=taskStatus%> <%=taskType %> task list for <a href="jobdetailshistory.jsp?jobid=<%=jobid%>&&logFile=<%=logFile %>"><%=jobid %> </a></h2>
-<center>
-<table border="2" cellpadding="5" cellspacing="2">
-<tr><td>Task Id</td><td>Start Time</td><td>Finish Time<br/></td><td>Error</td></tr>
-<%
-	for( JobHistory.Task task : tasks.values() ) {
-	  if( taskType.equals(task.get(Keys.TASK_TYPE) ) ){
-            Map <String, TaskAttempt> taskAttempts = task.getTaskAttempts();
-            for (JobHistory.TaskAttempt taskAttempt : taskAttempts.values()) {
-	      if( taskStatus.equals(taskAttempt.get(Keys.TASK_STATUS)) || taskStatus.equals("all")){
-	         printTask(jobid, logFile, task, out); 
-	      }
-            }
-	  }
-	}
-%>
-</table>
-<%!
-	private void printTask(String jobid, String trackerId, JobHistory.Task task, JspWriter out) throws IOException{
-  		out.print("<tr>"); 
-  		out.print("<td>" + "<a href=\"taskdetailshistory.jsp?jobid=" + jobid + 
-  		    "&logFile="+ trackerId +"&taskid="+task.get(Keys.TASKID)+"\">" + 
-  		    task.get(Keys.TASKID) + "</a></td>");
-  		out.print("<td>" + StringUtils.getFormattedTimeWithDiff(dateFormat, 
-  		    task.getLong(Keys.START_TIME), 0 ) + "</td>");
-		out.print("<td>" + StringUtils.getFormattedTimeWithDiff(dateFormat, 
-		    task.getLong(Keys.FINISH_TIME), task.getLong(Keys.START_TIME) ) + "</td>");
-  		out.print("<td>" + task.get(Keys.ERROR) + "</td>");
-  		out.print("</tr>"); 
-	}
-%>
-</center>
-</body>
-</html>

src/webapps/history/taskdetailshistory.jsp

@@ -1,69 +0,0 @@
-<%@ page
-  contentType="text/html; charset=UTF-8"
-  import="javax.servlet.http.*"
-  import="java.io.*"
-  import="java.util.*"
-  import="org.apache.hadoop.mapred.*"
-  import="org.apache.hadoop.util.*"
-  import="java.text.SimpleDateFormat"
-  import="org.apache.hadoop.mapred.JobHistory.*"
-%>
-<jsp:include page="loadhistory.jsp">
-	<jsp:param name="jobid" value="<%=request.getParameter("jobid") %>"/>
-	<jsp:param name="jobTrackerId" value="<%=request.getParameter("jobTrackerId") %>"/>
-</jsp:include>
-<%!	private static SimpleDateFormat dateFormat = new SimpleDateFormat("d/MM HH:mm:ss") ; %>
-
-<%	
-	String jobid = request.getParameter("jobid");
-	String logFile = request.getParameter("logFile");
-	String taskid = request.getParameter("taskid"); 
-
-	JobHistory.JobInfo job = (JobHistory.JobInfo)request.getSession().getAttribute("job");
-	JobHistory.Task task = job.getAllTasks().get(taskid); 
-%>
-<html>
-<body>
-<h2><%=taskid %> attempts for <a href="jobdetailshistory.jsp?jobid=<%=jobid%>&&logFile=<%=logFile %>"> <%=jobid %> </a></h2>
-<center>
-<table border="2" cellpadding="5" cellspacing="2">
-<tr><td>Task Id</td><td>Start Time</td>
-<%	
-	if( Values.REDUCE.name().equals(task.get(Keys.TASK_TYPE) ) ){
-%>
-		<td>Shuffle Finished</td><td>Sort Finished</td>
-<%
-	}
-%>
-<td>Finish Time</td><td>Host</td><td>Error</td></tr>
-<%
-	for( JobHistory.TaskAttempt attempt : task.getTaskAttempts().values() ) {
-	  printTaskAttempt(attempt, task.get(Keys.TASK_TYPE), out); 
-	}
-%>
-</table>
-<%!
-	private void printTaskAttempt(JobHistory.TaskAttempt taskAttempt, String type, JspWriter out) throws IOException{
-  		out.print("<tr>"); 
-  		out.print("<td>" + taskAttempt.get(Keys.TASK_ATTEMPT_ID) + "</td>");
-         out.print("<td>" + StringUtils.getFormattedTimeWithDiff(dateFormat, taskAttempt.getLong(Keys.START_TIME), 0 ) + "</td>") ; 
-  		if(Values.REDUCE.name().equals(type) ){
-  		  JobHistory.ReduceAttempt reduceAttempt = (JobHistory.ReduceAttempt)taskAttempt ; 
-	      out.print("<td>" + 
-	          StringUtils.getFormattedTimeWithDiff(dateFormat, 
-	              reduceAttempt.getLong(Keys.SHUFFLE_FINISHED), 
-	              reduceAttempt.getLong(Keys.START_TIME)) + "</td>") ; 
-	      out.print("<td>" + StringUtils.getFormattedTimeWithDiff(dateFormat, 
-	          	 reduceAttempt.getLong(Keys.SORT_FINISHED), 
-	          	 reduceAttempt.getLong(Keys.SHUFFLE_FINISHED)) + "</td>") ; 
-  		}
-	      out.print("<td>"+ StringUtils.getFormattedTimeWithDiff(dateFormat, taskAttempt.getLong(Keys.FINISH_TIME), 
-					          taskAttempt.getLong(Keys.START_TIME) ) + "</td>") ; 
-  		out.print("<td>" + taskAttempt.get(Keys.HOSTNAME) + "</td>");
-  		out.print("<td>" + taskAttempt.get(Keys.ERROR) + "</td>");
-  		out.print("</tr>"); 
-	}
-%>
-</center>
-</body>
-</html>

src/webapps/job/analysejobhistory.jsp

@@ -0,0 +1,249 @@
+<%@ page
+  contentType="text/html; charset=UTF-8"
+  import="javax.servlet.http.*"
+  import="java.io.*"
+  import="java.util.*"
+  import="org.apache.hadoop.mapred.*"
+  import="org.apache.hadoop.util.*"
+  import="java.text.SimpleDateFormat"
+  import="org.apache.hadoop.mapred.JobHistory.*"
+%>
+<jsp:include page="loadhistory.jsp">
+  <jsp:param name="jobid" value="<%=request.getParameter("jobid") %>"/>
+  <jsp:param name="logFile" value="<%=request.getParameter("logFile") %>"/>
+</jsp:include>
+<%!	private static SimpleDateFormat dateFormat 
+                              = new SimpleDateFormat("d/MM HH:mm:ss") ; 
+%>
+<html><body>
+<%
+  String jobid = request.getParameter("jobid");
+  String logFile = request.getParameter("logFile");
+  String numTasks = request.getParameter("numTasks");
+  int showTasks = 10 ; 
+  if (numTasks != null) {
+    showTasks = Integer.parseInt(numTasks);  
+  }
+  JobInfo job = (JobInfo)request.getSession().getAttribute("job");
+%>
+<h2>Hadoop Job <a href="jobdetailshistory.jsp?jobid=<%=jobid%>&&logFile=<%=logFile %>"><%=jobid %> </a></h2>
+<b>User : </b> <%=job.get(Keys.USER) %><br/> 
+<b>JobName : </b> <%=job.get(Keys.JOBNAME) %><br/> 
+<b>JobConf : </b> <%=job.get(Keys.JOBCONF) %><br/> 
+<b>Submitted At : </b> <%=StringUtils.getFormattedTimeWithDiff(dateFormat, job.getLong(Keys.SUBMIT_TIME), 0 ) %><br/> 
+<b>Launched At : </b> <%=StringUtils.getFormattedTimeWithDiff(dateFormat, job.getLong(Keys.LAUNCH_TIME), job.getLong(Keys.SUBMIT_TIME)) %><br/>
+<b>Finished At : </b>  <%=StringUtils.getFormattedTimeWithDiff(dateFormat, job.getLong(Keys.FINISH_TIME), job.getLong(Keys.LAUNCH_TIME)) %><br/>
+<b>Status : </b> <%= ((job.get(Keys.JOB_STATUS) == null)?"Incomplete" :job.get(Keys.JOB_STATUS)) %><br/> 
+<hr/>
+<center>
+<%
+  if (!Values.SUCCESS.name().equals(job.get(Keys.JOB_STATUS))) {
+    out.print("<h3>No Analysis available as job did not finish</h3>");
+    return;
+  }
+  Map<String, JobHistory.Task> tasks = job.getAllTasks();
+  int finishedMaps = job.getInt(Keys.FINISHED_MAPS)  ;
+  int finishedReduces = job.getInt(Keys.FINISHED_REDUCES) ;
+  JobHistory.Task [] mapTasks = new JobHistory.Task[finishedMaps]; 
+  JobHistory.Task [] reduceTasks = new JobHistory.Task[finishedReduces]; 
+  int mapIndex = 0 , reduceIndex=0; 
+  long avgMapTime = 0;
+  long avgReduceTime = 0;
+  long avgShuffleTime = 0;
+
+  for (JobHistory.Task task : tasks.values()) {
+    Map<String, TaskAttempt> attempts = task.getTaskAttempts();
+    for (JobHistory.TaskAttempt attempt : attempts.values()) {
+      if (attempt.get(Keys.TASK_STATUS).equals(Values.SUCCESS.name())) {
+        long avgFinishTime = (attempt.getLong(Keys.FINISH_TIME) -
+      		                attempt.getLong(Keys.START_TIME));
+        if (Values.MAP.name().equals(task.get(Keys.TASK_TYPE))) {
+          mapTasks[mapIndex++] = attempt ; 
+          avgMapTime += avgFinishTime;
+        } else { 
+          reduceTasks[reduceIndex++] = attempt;
+          avgShuffleTime += (attempt.getLong(Keys.SHUFFLE_FINISHED) - 
+                             attempt.getLong(Keys.START_TIME));
+          avgReduceTime += (attempt.getLong(Keys.FINISH_TIME) -
+                            attempt.getLong(Keys.SHUFFLE_FINISHED));
+        }
+        break;
+      }
+    }
+  }
+	 
+  if (finishedMaps > 0) {
+    avgMapTime /= finishedMaps;
+  }
+  if (finishedReduces > 0) {
+    avgReduceTime /= finishedReduces;
+    avgShuffleTime /= finishedReduces;
+  }
+  Comparator<JobHistory.Task> cMap = new Comparator<JobHistory.Task>(){
+    public int compare(JobHistory.Task t1, JobHistory.Task t2){
+      long l1 = t1.getLong(Keys.FINISH_TIME) - t1.getLong(Keys.START_TIME); 
+      long l2 = t2.getLong(Keys.FINISH_TIME) - t2.getLong(Keys.START_TIME);
+      return (l2<l1 ? -1 : (l2==l1 ? 0 : 1));
+    }
+  }; 
+  Comparator<JobHistory.Task> cShuffle = new Comparator<JobHistory.Task>(){
+    public int compare(JobHistory.Task t1, JobHistory.Task t2){
+      long l1 = t1.getLong(Keys.SHUFFLE_FINISHED) - 
+                t1.getLong(Keys.START_TIME); 
+      long l2 = t2.getLong(Keys.SHUFFLE_FINISHED) - 
+                t2.getLong(Keys.START_TIME); 
+      return (l2<l1 ? -1 : (l2==l1 ? 0 : 1));
+    }
+  }; 
+  Arrays.sort(mapTasks, cMap);
+  JobHistory.Task minMap = mapTasks[mapTasks.length-1] ;
+%>
+
+<h3>Time taken by best performing Map task 
+<a href="taskdetailshistory.jsp?jobid=<%=jobid%>&logFile=<%=logFile%>&taskid=<%=minMap.get(Keys.TASKID)%>">
+<%=minMap.get(Keys.TASKID) %></a> : <%=StringUtils.formatTimeDiff(minMap.getLong(Keys.FINISH_TIME), minMap.getLong(Keys.START_TIME) ) %></h3>
+<h3>Average time taken by Map tasks: 
+<%=StringUtils.formatTimeDiff(avgMapTime, 0) %></h3>
+<h3>Worse performing map tasks</h3>
+<table border="2" cellpadding="5" cellspacing="2">
+<tr><td>Task Id</td><td>Time taken</td></tr>
+<%
+  for (int i=0;i<showTasks && i<mapTasks.length; i++) {
+%>
+    <tr>
+    <td><a href="taskdetailshistory.jsp?jobid=<%=jobid%>&logFile=<%=logFile%>&taskid=<%=mapTasks[i].get(Keys.TASKID)%>">
+        <%=mapTasks[i].get(Keys.TASKID) %></a></td>
+    <td><%=StringUtils.formatTimeDiff(mapTasks[i].getLong(Keys.FINISH_TIME), mapTasks[i].getLong(Keys.START_TIME)) %></td>
+    </tr>
+<%
+  }
+%>
+</table>
+<%  
+  Comparator<JobHistory.Task> cFinishMapRed = 
+    new Comparator<JobHistory.Task>() {
+    public int compare(JobHistory.Task t1, JobHistory.Task t2){
+      long l1 = t1.getLong(Keys.FINISH_TIME); 
+      long l2 = t2.getLong(Keys.FINISH_TIME);
+      return (l2<l1 ? -1 : (l2==l1 ? 0 : 1));
+    }
+  };
+  Arrays.sort(mapTasks, cFinishMapRed);
+  JobHistory.Task lastMap = mapTasks[0] ;
+%>
+
+<h3>The last Map task 
+<a href="taskdetailshistory.jsp?jobid=<%=jobid%>&logFile=<%=logFile%>
+&taskid=<%=lastMap.get(Keys.TASKID)%>"><%=lastMap.get(Keys.TASKID) %></a> 
+finished at (relative to the Job launch time): 
+<%=StringUtils.getFormattedTimeWithDiff(dateFormat, 
+                              lastMap.getLong(Keys.FINISH_TIME), 
+                              job.getLong(Keys.LAUNCH_TIME) ) %></h3>
+<hr/>
+
+<%
+  if (reduceTasks.length <= 0) return;
+  Arrays.sort(reduceTasks, cShuffle); 
+  JobHistory.Task minShuffle = reduceTasks[reduceTasks.length-1] ;
+%>
+<h3>Time taken by best performing shuffle
+<a href="taskdetailshistory.jsp?jobid=<%=jobid%>&logFile=<%=logFile%>
+&taskid=<%=minShuffle.get(Keys.TASKID)%>"><%=minShuffle.get(Keys.TASKID)%></a> : 
+<%=StringUtils.formatTimeDiff(minShuffle.getLong(Keys.SHUFFLE_FINISHED), 
+                              minShuffle.getLong(Keys.START_TIME) ) %></h3>
+<h3>Average time taken by Shuffle: 
+<%=StringUtils.formatTimeDiff(avgShuffleTime, 0) %></h3>
+<h3>Worse performing Shuffle(s)</h3>
+<table border="2" cellpadding="5" cellspacing="2">
+<tr><td>Task Id</td><td>Time taken</td></tr>
+<%
+  for (int i=0;i<showTasks && i<reduceTasks.length; i++) {
+%>
+    <tr>
+    <td><a href="taskdetailshistory.jsp?jobid=<%=jobid%>&logFile=
+        <%=logFile%>&taskid=<%=reduceTasks[i].get(Keys.TASKID)%>">
+        <%=reduceTasks[i].get(Keys.TASKID) %></a></td>
+    <td><%=
+           StringUtils.formatTimeDiff(
+                       reduceTasks[i].getLong(Keys.SHUFFLE_FINISHED),
+                       reduceTasks[i].getLong(Keys.START_TIME)) %>
+    </td>
+    </tr>
+<%
+  }
+%>
+</table>
+<%  
+  Comparator<JobHistory.Task> cFinishShuffle = 
+    new Comparator<JobHistory.Task>() {
+    public int compare(JobHistory.Task t1, JobHistory.Task t2){
+      long l1 = t1.getLong(Keys.SHUFFLE_FINISHED); 
+      long l2 = t2.getLong(Keys.SHUFFLE_FINISHED);
+      return (l2<l1 ? -1 : (l2==l1 ? 0 : 1));
+    }
+  };
+  Arrays.sort(reduceTasks, cFinishShuffle);
+  JobHistory.Task lastShuffle = reduceTasks[0] ;
+%>
+
+<h3>The last Shuffle  
+<a href="taskdetailshistory.jsp?jobid=<%=jobid%>&logFile=<%=logFile%>
+&taskid=<%=lastShuffle.get(Keys.TASKID)%>"><%=lastShuffle.get(Keys.TASKID)%>
+</a> finished at (relative to the Job launch time): 
+<%=StringUtils.getFormattedTimeWithDiff(dateFormat,
+                              lastShuffle.getLong(Keys.SHUFFLE_FINISHED), 
+                              job.getLong(Keys.LAUNCH_TIME) ) %></h3>
+
+<%
+  Comparator<JobHistory.Task> cReduce = new Comparator<JobHistory.Task>(){
+    public int compare(JobHistory.Task t1, JobHistory.Task t2){
+      long l1 = t1.getLong(Keys.FINISH_TIME) - 
+                t1.getLong(Keys.SHUFFLE_FINISHED); 
+      long l2 = t2.getLong(Keys.FINISH_TIME) - 
+                t2.getLong(Keys.SHUFFLE_FINISHED);
+      return (l2<l1 ? -1 : (l2==l1 ? 0 : 1));
+    }
+  }; 
+  Arrays.sort(reduceTasks, cReduce); 
+  JobHistory.Task minReduce = reduceTasks[reduceTasks.length-1] ;
+%>
+<hr/>
+<h3>Time taken by best performing Reduce task : 
+<a href="taskdetailshistory.jsp?jobid=<%=jobid%>&logFile=<%=logFile%>&taskid=<%=minReduce.get(Keys.TASKID)%>">
+<%=minReduce.get(Keys.TASKID) %></a> : 
+<%=StringUtils.formatTimeDiff(minReduce.getLong(Keys.FINISH_TIME),
+    minReduce.getLong(Keys.SHUFFLE_FINISHED) ) %></h3>
+
+<h3>Average time taken by Reduce tasks: 
+<%=StringUtils.formatTimeDiff(avgReduceTime, 0) %></h3>
+<h3>Worse performing reduce tasks</h3>
+<table border="2" cellpadding="5" cellspacing="2">
+<tr><td>Task Id</td><td>Time taken</td></tr>
+<%
+  for (int i=0;i<showTasks && i<reduceTasks.length; i++) {
+%>
+    <tr>
+    <td><a href="taskdetailshistory.jsp?jobid=<%=jobid%>&logFile=<%=logFile%>&taskid=<%=reduceTasks[i].get(Keys.TASKID)%>">
+        <%=reduceTasks[i].get(Keys.TASKID) %></a></td>
+    <td><%=StringUtils.formatTimeDiff(
+             reduceTasks[i].getLong(Keys.FINISH_TIME), 
+             reduceTasks[i].getLong(Keys.SHUFFLE_FINISHED)) %></td>
+    </tr>
+<%
+  }
+%>
+</table>
+<%  
+  Arrays.sort(reduceTasks, cFinishMapRed);
+  JobHistory.Task lastReduce = reduceTasks[0] ;
+%>
+
+<h3>The last Reduce task 
+<a href="taskdetailshistory.jsp?jobid=<%=jobid%>&logFile=<%=logFile%>
+&taskid=<%=lastReduce.get(Keys.TASKID)%>"><%=lastReduce.get(Keys.TASKID)%>
+</a> finished at (relative to the Job launch time): 
+<%=StringUtils.getFormattedTimeWithDiff(dateFormat,
+                              lastReduce.getLong(Keys.FINISH_TIME), 
+                              job.getLong(Keys.LAUNCH_TIME) ) %></h3>
+</center>
+</body></html>

src/webapps/history/jobconf_history.jsp → src/webapps/job/jobconf_history.jsp

@@ -51,8 +51,6 @@
 %>
 
 <br>
-<hr>
-<a href="http://lucene.apache.org/hadoop">Hadoop</a>, 2007.<br>
-
-</body>
-</html>
+<%
+out.println(ServletUtil.htmlFooter());
+%>

src/webapps/job/jobdetailshistory.jsp

@@ -0,0 +1,192 @@
+<%@ page
+  contentType="text/html; charset=UTF-8"
+  import="javax.servlet.http.*"
+  import="java.io.*"
+  import="java.util.*"
+  import="org.apache.hadoop.fs.*"
+  import="org.apache.hadoop.mapred.*"
+  import="org.apache.hadoop.util.*"
+  import="java.text.SimpleDateFormat"
+  import="org.apache.hadoop.mapred.JobHistory.*"
+%>
+<jsp:include page="loadhistory.jsp">
+  <jsp:param name="jobid" value="<%=request.getParameter("jobid") %>"/>
+  <jsp:param name="logFile" value="<%=request.getParameter("logFile") %>"/>
+</jsp:include>
+<%! static SimpleDateFormat dateFormat = new SimpleDateFormat("d-MMM-yyyy HH:mm:ss") ; %>
+<%
+    String jobid = request.getParameter("jobid");
+    String logFile = request.getParameter("logFile");
+	
+    Path jobFile = new Path(logFile);
+    String[] jobDetails = jobFile.getName().split("_");
+    String jobUniqueString = jobDetails[0] + "_" +jobDetails[1] + "_" + jobid ;
+	
+    JobInfo job = (JobInfo)request.getSession().getAttribute("job");
+    FileSystem fs = (FileSystem)request.getSession().getAttribute("fs");
+%>
+<html><body>
+<h2>Hadoop Job <%=jobid %> on <a href="jobhistory.jsp">History Viewer</a></h2>
+
+<b>User: </b> <%=job.get(Keys.USER) %><br/> 
+<b>JobName: </b> <%=job.get(Keys.JOBNAME) %><br/> 
+<b>JobConf: </b> <a href="jobconf_history.jsp?jobid=<%=jobid%>&jobLogDir=<%=new Path(logFile).getParent().toString()%>&jobUniqueString=<%=jobUniqueString%>"> 
+                 <%=job.get(Keys.JOBCONF) %></a><br/> 
+<b>Submitted At: </b> <%=StringUtils.getFormattedTimeWithDiff(dateFormat, job.getLong(Keys.SUBMIT_TIME), 0 )  %><br/> 
+<b>Launched At: </b> <%=StringUtils.getFormattedTimeWithDiff(dateFormat, job.getLong(Keys.LAUNCH_TIME), job.getLong(Keys.SUBMIT_TIME)) %><br/>
+<b>Finished At: </b>  <%=StringUtils.getFormattedTimeWithDiff(dateFormat, job.getLong(Keys.FINISH_TIME), job.getLong(Keys.LAUNCH_TIME)) %><br/>
+<b>Status: </b> <%= ((job.get(Keys.JOB_STATUS) == "")?"Incomplete" :job.get(Keys.JOB_STATUS)) %><br/> 
+<%
+    Map<String, JobHistory.Task> tasks = job.getAllTasks();
+    int totalMaps = 0 ; 
+    int totalReduces = 0; 
+    int numFailedMaps = 0; 
+    int numKilledMaps = 0;
+    int numFailedReduces = 0 ; 
+    int numKilledReduces = 0;
+	
+    long mapStarted = 0 ; 
+    long mapFinished = 0 ; 
+    long reduceStarted = 0 ; 
+    long reduceFinished = 0; 
+        
+    Map <String,String> allHosts = new TreeMap<String,String>();
+    for (JobHistory.Task task : tasks.values()) {
+      Map<String, TaskAttempt> attempts = task.getTaskAttempts();
+      allHosts.put(task.get(Keys.HOSTNAME), "");
+      for (TaskAttempt attempt : attempts.values()) {
+        long startTime = attempt.getLong(Keys.START_TIME) ; 
+        long finishTime = attempt.getLong(Keys.FINISH_TIME) ; 
+        if (Values.MAP.name().equals(task.get(Keys.TASK_TYPE))){
+          if (mapStarted==0 || mapStarted > startTime ) {
+            mapStarted = startTime; 
+          }
+          if (mapFinished < finishTime ) {
+            mapFinished = finishTime ; 
+          }
+          totalMaps++; 
+          if (Values.FAILED.name().equals(attempt.get(Keys.TASK_STATUS))) {
+            numFailedMaps++; 
+          } else if (Values.KILLED.name().equals(attempt.get(Keys.TASK_STATUS))) {
+            numKilledMaps++;
+          }
+        } else {
+          if (reduceStarted==0||reduceStarted > startTime) {
+            reduceStarted = startTime ; 
+          }
+          if (reduceFinished < finishTime) {
+            reduceFinished = finishTime; 
+          }
+          totalReduces++; 
+          if (Values.FAILED.name().equals(attempt.get(Keys.TASK_STATUS))) {
+            numFailedReduces++;
+          } else if (Values.KILLED.name().equals(attempt.get(Keys.TASK_STATUS))) {
+            numKilledReduces++;
+          }
+        }
+      }
+    }
+%>
+<b><a href="analysejobhistory.jsp?jobid=<%=jobid %>&logFile=<%=logFile %>">Analyse This Job</a></b> 
+<hr/>
+<center>
+<table border="2" cellpadding="5" cellspacing="2">
+<tr>
+<td>Kind</td><td>Total Tasks(successful+failed+killed)</td><td>Successful tasks</td><td>Failed tasks</td><td>Killed tasks</td><td>Start Time</td><td>Finish Time</td>
+</tr>
+<tr>
+<td>Map</td>
+    <td><a href="jobtaskshistory.jsp?jobid=<%=jobid %>&logFile=<%=logFile %>&taskType=<%=Values.MAP.name() %>&status=all">
+        <%=totalMaps %></a></td>
+    <td><a href="jobtaskshistory.jsp?jobid=<%=jobid %>&logFile=<%=logFile %>&taskType=<%=Values.MAP.name() %>&status=<%=Values.SUCCESS %>">
+        <%=job.getInt(Keys.FINISHED_MAPS) %></a></td>
+    <td><a href="jobtaskshistory.jsp?jobid=<%=jobid %>&logFile=<%=logFile %>&taskType=<%=Values.MAP.name() %>&status=<%=Values.FAILED %>">
+        <%=numFailedMaps %></a></td>
+    <td><a href="jobtaskshistory.jsp?jobid=<%=jobid %>&logFile=<%=logFile %>&taskType=<%=Values.MAP.name() %>&status=<%=Values.KILLED %>">
+        <%=numKilledMaps %></a></td>
+    <td><%=StringUtils.getFormattedTimeWithDiff(dateFormat, mapStarted, 0) %></td>
+    <td><%=StringUtils.getFormattedTimeWithDiff(dateFormat, mapFinished, mapStarted) %></td>
+</tr>
+<tr>
+<td>Reduce</td>
+    <td><a href="jobtaskshistory.jsp?jobid=<%=jobid %>&logFile=<%=logFile %>&taskType=<%=Values.REDUCE.name() %>&status=all">
+        <%=totalReduces%></a></td>
+    <td><a href="jobtaskshistory.jsp?jobid=<%=jobid %>&logFile=<%=logFile %>&taskType=<%=Values.REDUCE.name() %>&status=<%=Values.SUCCESS %>">
+        <%=job.getInt(Keys.FINISHED_REDUCES)%></a></td>
+    <td><a href="jobtaskshistory.jsp?jobid=<%=jobid %>&logFile=<%=logFile %>&taskType=<%=Values.REDUCE.name() %>&status=<%=Values.FAILED %>">
+        <%=numFailedReduces%></a></td>
+    <td><a href="jobtaskshistory.jsp?jobid=<%=jobid %>&logFile=<%=logFile %>&taskType=<%=Values.REDUCE.name() %>&status=<%=Values.KILLED %>">
+        <%=numKilledReduces%></a></td>  
+    <td><%=StringUtils.getFormattedTimeWithDiff(dateFormat, reduceStarted, 0) %></td>
+    <td><%=StringUtils.getFormattedTimeWithDiff(dateFormat, reduceFinished, reduceStarted) %></td>
+</tr>
+</table>
+
+<br/>
+ <%
+    DefaultJobHistoryParser.FailedOnNodesFilter filter = 
+                 new DefaultJobHistoryParser.FailedOnNodesFilter();
+    JobHistory.parseHistoryFromFS(logFile, filter, fs); 
+    Map<String, Set<String>> badNodes = filter.getValues(); 
+    if (badNodes.size() > 0) {
+ %>
+<h3>Failed tasks attempts by nodes </h3>
+<table border="1">
+<tr><td>Hostname</td><td>Failed Tasks</td></tr>
+ <%	  
+      for (Map.Entry<String, Set<String>> entry : badNodes.entrySet()) {
+        String node = entry.getKey();
+        Set<String> failedTasks = entry.getValue();
+%>
+        <tr>
+        <td><%=node %></td>
+        <td>
+<%
+        for (String t : failedTasks) {
+%>
+          <a href="taskdetailshistory.jsp?jobid=<%=jobid%>&logFile=<%=logFile %>&taskid=<%=t %>"><%=t %></a>,&nbsp;
+<%		  
+        }
+%>	
+        </td>
+        </tr>
+<%	  
+      }
+	}
+ %>
+</table>
+<br/>
+ <%
+    DefaultJobHistoryParser.KilledOnNodesFilter killedFilter =
+                 new DefaultJobHistoryParser.KilledOnNodesFilter();
+    JobHistory.parseHistoryFromFS(logFile, filter, fs); 
+    badNodes = killedFilter.getValues(); 
+    if (badNodes.size() > 0) {
+ %>
+<h3>Killed tasks attempts by nodes </h3>
+<table border="1">
+<tr><td>Hostname</td><td>Killed Tasks</td></tr>
+ <%	  
+      for (Map.Entry<String, Set<String>> entry : badNodes.entrySet()) {
+        String node = entry.getKey();
+        Set<String> killedTasks = entry.getValue();
+%>
+        <tr>
+        <td><%=node %></td>
+        <td>
+<%
+        for (String t : killedTasks) {
+%>
+          <a href="taskdetailshistory.jsp?jobid=<%=jobid%>&logFile=<%=logFile %>&taskid=<%=t %>"><%=t %></a>,&nbsp;
+<%		  
+        }
+%>	
+        </td>
+        </tr>
+<%	  
+      }
+    }
+%>
+</table>
+</center>
+</body></html>

src/webapps/history/jobhistory.jsp → src/webapps/job/jobhistory.jsp

@@ -11,7 +11,8 @@
   import="org.apache.hadoop.mapred.JobHistory.*"
 %>
 <%!	
-	private static SimpleDateFormat dateFormat = new SimpleDateFormat("d/MM HH:mm:ss") ;
+  private static SimpleDateFormat dateFormat = 
+                                    new SimpleDateFormat("d/MM HH:mm:ss");
 %>
 <html>
 <head>
@@ -29,9 +30,17 @@
       }
     };
     
-	FileSystem fs = (FileSystem) application.getAttribute("fileSys");
-	String historyLogDir = (String) application.getAttribute("historyLogDir");
-	Path[] jobFiles = fs.listPaths(new Path(historyLogDir), jobLogFileFilter);
+    FileSystem fs = (FileSystem) application.getAttribute("fileSys");
+    String historyLogDir = (String) application.getAttribute("historyLogDir");
+    if (fs == null) {
+      out.println("Null file system. May be namenode is in safemode!");
+      return;
+    }
+    Path[] jobFiles = fs.listPaths(new Path(historyLogDir), jobLogFileFilter);
+    if (null == jobFiles )  {
+      out.println("NULL files !!!"); 
+      return ; 
+    }
 
     // sort the files on creation time.
     Arrays.sort(jobFiles, new Comparator<Path>() {
@@ -50,16 +59,10 @@
           Long l1 = Long.parseLong(split1[4]);
           res = l1.compareTo(Long.parseLong(split2[4]));
         }
-        
         return res;
       }
     });
 
-    if (null == jobFiles ){
-      out.println("NULL !!!"); 
-      return ; 
-    }
-       
     out.print("<table align=center border=2 cellpadding=\"5\" cellspacing=\"2\">");
     out.print("<tr><td align=\"center\" colspan=\"9\"><b>Available Jobs </b></td></tr>\n");
     out.print("<tr>");
@@ -74,31 +77,29 @@
       String jobId = jobDetails[2] + "_" +jobDetails[3] + "_" + jobDetails[4] ;
       String user = jobDetails[5];
       String jobName = jobDetails[6];
-      
 %>
 <center>
 <%	
-
-	  printJob(trackerHostName, trackerStartTime, jobId,
+      printJob(trackerHostName, trackerStartTime, jobId,
                jobName, user, jobFile.toString(), out) ; 
 %>
 </center> 
 <%
-	} // end while trackers 
+    } // end while trackers 
 %>
 <%!
-	private void printJob(String trackerHostName, String trackerid,
+    private void printJob(String trackerHostName, String trackerid,
                           String jobId, String jobName,
                           String user, String logFile, JspWriter out)
-    throws IOException{
-	    out.print("<tr>"); 
-	    out.print("<td>" + trackerHostName + "</td>"); 
-	    out.print("<td>" + new Date(Long.parseLong(trackerid)) + "</td>"); 
-	    out.print("<td>" + "<a href=\"jobdetailshistory.jsp?jobid="+ jobId + 
-	        "&logFile=" + logFile +"\">" + jobId + "</a></td>"); 
-	    out.print("<td>" + jobName + "</td>"); 
-	    out.print("<td>" + user + "</td>"); 
-	    out.print("</tr>");
-	}
- %> 
+    throws IOException {
+      out.print("<tr>"); 
+      out.print("<td>" + trackerHostName + "</td>"); 
+      out.print("<td>" + new Date(Long.parseLong(trackerid)) + "</td>"); 
+      out.print("<td>" + "<a href=\"jobdetailshistory.jsp?jobid=" + jobId + 
+                "&logFile=" + logFile + "\">" + jobId + "</a></td>"); 
+      out.print("<td>" + jobName + "</td>"); 
+      out.print("<td>" + user + "</td>"); 
+      out.print("</tr>");
+    }
+%> 
 </body></html>

src/webapps/job/jobtaskshistory.jsp

@@ -0,0 +1,68 @@
+<%@ page
+  contentType="text/html; charset=UTF-8"
+  import="javax.servlet.http.*"
+  import="java.io.*"
+  import="java.util.*"
+  import="org.apache.hadoop.mapred.*"
+  import="org.apache.hadoop.util.*"
+  import="java.text.SimpleDateFormat"
+  import="org.apache.hadoop.mapred.JobHistory.*"
+%>
+<jsp:include page="loadhistory.jsp">
+	<jsp:param name="jobid" value="<%=request.getParameter("jobid") %>"/>
+	<jsp:param name="logFile" value="<%=request.getParameter("logFile") %>"/>
+</jsp:include>
+<%!	
+  private static SimpleDateFormat dateFormat =
+                                    new SimpleDateFormat("d/MM HH:mm:ss") ; 
+%>
+
+<%	
+  String jobid = request.getParameter("jobid");
+  String logFile = request.getParameter("logFile");
+  String taskStatus = request.getParameter("status"); 
+  String taskType = request.getParameter("taskType"); 
+  
+  JobHistory.JobInfo job = (JobHistory.JobInfo)request.
+                            getSession().getAttribute("job");
+  Map<String, JobHistory.Task> tasks = job.getAllTasks(); 
+%>
+<html>
+<body>
+<h2><%=taskStatus%> <%=taskType %> task list for <a href="jobdetailshistory.jsp?jobid=<%=jobid%>&&logFile=<%=logFile %>"><%=jobid %> </a></h2>
+<center>
+<table border="2" cellpadding="5" cellspacing="2">
+<tr><td>Task Id</td><td>Start Time</td><td>Finish Time<br/></td><td>Error</td></tr>
+<%
+  for (JobHistory.Task task : tasks.values()) {
+    if (taskType.equals(task.get(Keys.TASK_TYPE))){
+      Map <String, TaskAttempt> taskAttempts = task.getTaskAttempts();
+      for (JobHistory.TaskAttempt taskAttempt : taskAttempts.values()) {
+        if (taskStatus.equals(taskAttempt.get(Keys.TASK_STATUS)) || 
+          taskStatus.equals("all")){
+          printTask(jobid, logFile, taskAttempt, out); 
+        }
+      }
+    }
+  }
+%>
+</table>
+<%!
+  private void printTask(String jobid, String trackerId,
+    JobHistory.TaskAttempt attempt, JspWriter out) throws IOException{
+    out.print("<tr>"); 
+    out.print("<td>" + "<a href=\"taskdetailshistory.jsp?jobid=" + jobid + 
+          "&logFile="+ trackerId +"&taskid="+attempt.get(Keys.TASKID)+"\">" +
+          attempt.get(Keys.TASKID) + "</a></td>");
+    out.print("<td>" + StringUtils.getFormattedTimeWithDiff(dateFormat, 
+          attempt.getLong(Keys.START_TIME), 0 ) + "</td>");
+    out.print("<td>" + StringUtils.getFormattedTimeWithDiff(dateFormat, 
+          attempt.getLong(Keys.FINISH_TIME),
+          attempt.getLong(Keys.START_TIME) ) + "</td>");
+    out.print("<td>" + attempt.get(Keys.ERROR) + "</td>");
+    out.print("</tr>"); 
+  }
+%>
+</center>
+</body>
+</html>

src/webapps/job/jobtracker.jsp

@@ -134,7 +134,7 @@
 <hr>
 
 <h2>Local logs</h2>
-<a href="logs/">Log</a> directory, <a href="http://<%=tracker.getHistoryAddress()%>">
+<a href="logs/">Log</a> directory, <a href="jobhistory.jsp">
 Job Tracker History</a>
 
 <%

src/webapps/history/loadhistory.jsp → src/webapps/job/loadhistory.jsp

@@ -16,20 +16,21 @@
       }
     };
     
-	FileSystem fs = (FileSystem) application.getAttribute("fileSys");
-	String jobId =  (String)request.getParameter("jobid");
-	JobHistory.JobInfo job = (JobHistory.JobInfo)request.getSession().getAttribute("job");
-	if (null != job && (!jobId.equals(job.get(Keys.JOBID)))){
+    FileSystem fs = (FileSystem) application.getAttribute("fileSys");
+    String jobId =  (String)request.getParameter("jobid");
+    JobHistory.JobInfo job = (JobHistory.JobInfo)
+                               request.getSession().getAttribute("job");
+    if (null != job && (!jobId.equals(job.get(Keys.JOBID)))) {
       // remove jobInfo from session, keep only one job in session at a time
       request.getSession().removeAttribute("job"); 
       job = null ; 
     }
 	
-	if (null == job) {
+    if (null == job) {
       String jobLogFile = (String)request.getParameter("logFile");
       job = new JobHistory.JobInfo(jobId); 
       DefaultJobHistoryParser.parseJobTasks(jobLogFile, job, fs) ; 
       request.getSession().setAttribute("job", job);
       request.getSession().setAttribute("fs", fs);
-	}
+    }
 %>

src/webapps/job/taskdetailshistory.jsp

@@ -0,0 +1,74 @@
+<%@ page
+  contentType="text/html; charset=UTF-8"
+  import="javax.servlet.http.*"
+  import="java.io.*"
+  import="java.util.*"
+  import="org.apache.hadoop.mapred.*"
+  import="org.apache.hadoop.util.*"
+  import="java.text.SimpleDateFormat"
+  import="org.apache.hadoop.mapred.JobHistory.*"
+%>
+<jsp:include page="loadhistory.jsp">
+  <jsp:param name="jobid" value="<%=request.getParameter("jobid") %>"/>
+  <jsp:param name="jobTrackerId" value="<%=request.getParameter("jobTrackerId") %>"/>
+</jsp:include>
+<%!	private static SimpleDateFormat dateFormat = new SimpleDateFormat("d/MM HH:mm:ss") ; %>
+
+<%	
+  String jobid = request.getParameter("jobid");
+  String logFile = request.getParameter("logFile");
+  String taskid = request.getParameter("taskid"); 
+  JobHistory.JobInfo job = (JobHistory.JobInfo)
+                              request.getSession().getAttribute("job");
+  JobHistory.Task task = job.getAllTasks().get(taskid); 
+%>
+<html>
+<body>
+<h2><%=taskid %> attempts for <a href="jobdetailshistory.jsp?jobid=<%=jobid%>&&logFile=<%=logFile %>"> <%=jobid %> </a></h2>
+<center>
+<table border="2" cellpadding="5" cellspacing="2">
+<tr><td>Task Id</td><td>Start Time</td>
+<%	
+  if (Values.REDUCE.name().equals(task.get(Keys.TASK_TYPE))) {
+%>
+    <td>Shuffle Finished</td><td>Sort Finished</td>
+<%
+  }
+%>
+<td>Finish Time</td><td>Host</td><td>Error</td></tr>
+<%
+  for (JobHistory.TaskAttempt attempt : task.getTaskAttempts().values()) {
+    printTaskAttempt(attempt, task.get(Keys.TASK_TYPE), out); 
+  }
+%>
+</table>
+<%!
+  private void printTaskAttempt(JobHistory.TaskAttempt taskAttempt,
+                                String type, JspWriter out) 
+  throws IOException {
+    out.print("<tr>"); 
+    out.print("<td>" + taskAttempt.get(Keys.TASK_ATTEMPT_ID) + "</td>");
+    out.print("<td>" + StringUtils.getFormattedTimeWithDiff(dateFormat,
+              taskAttempt.getLong(Keys.START_TIME), 0 ) + "</td>"); 
+    if (Values.REDUCE.name().equals(type)) {
+      JobHistory.ReduceAttempt reduceAttempt = 
+            (JobHistory.ReduceAttempt)taskAttempt; 
+      out.print("<td>" + 
+                StringUtils.getFormattedTimeWithDiff(dateFormat, 
+                reduceAttempt.getLong(Keys.SHUFFLE_FINISHED), 
+                reduceAttempt.getLong(Keys.START_TIME)) + "</td>"); 
+      out.print("<td>" + StringUtils.getFormattedTimeWithDiff(dateFormat, 
+                reduceAttempt.getLong(Keys.SORT_FINISHED), 
+                reduceAttempt.getLong(Keys.SHUFFLE_FINISHED)) + "</td>"); 
+    }
+    out.print("<td>"+ StringUtils.getFormattedTimeWithDiff(dateFormat,
+              taskAttempt.getLong(Keys.FINISH_TIME), 
+              taskAttempt.getLong(Keys.START_TIME) ) + "</td>"); 
+    out.print("<td>" + taskAttempt.get(Keys.HOSTNAME) + "</td>");
+    out.print("<td>" + taskAttempt.get(Keys.ERROR) + "</td>");
+    out.print("</tr>"); 
+  }
+%>
+</center>
+</body>
+</html>