HADOOP-2796. Enables distinguishing exit codes from user code vis-a-vis HOD's exit code. Contributed by Hemanth Yamijala.

git-svn-id: https://svn.apache.org/repos/asf/hadoop/core/trunk@639655 13f79535-47bb-0310-9956-ffa450edef68

CHANGES.txt

@@ -131,6 +131,9 @@ Trunk (unreleased changes)
 
     HADOOP-2373. Improvement in safe-mode reporting. (shv)
 
+    HADOOP-2796. Enables distinguishing exit codes from user code vis-a-vis
+    HOD's exit code. (Hemanth Yamijala via ddas)
+
   OPTIMIZATIONS
 
     HADOOP-2790.  Fixed inefficient method hasSpeculativeTask by removing

docs/changes.html

@@ -56,7 +56,7 @@
 </a></h2>
 <ul id="trunk_(unreleased_changes)_">
   <li><a href="javascript:toggleList('trunk_(unreleased_changes)_._incompatible_changes_')">  INCOMPATIBLE CHANGES
-</a>&nbsp;&nbsp;&nbsp;(10)
+</a>&nbsp;&nbsp;&nbsp;(11)
     <ol id="trunk_(unreleased_changes)_._incompatible_changes_">
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-2786">HADOOP-2786</a>.  Move hbase out of hadoop core
 </li>
@@ -79,6 +79,7 @@ from 26 to 27. (Tsz Wo (Nicholas), SZE via cdouglas)
 </li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-2822">HADOOP-2822</a>. Remove depreceted code for classes InputFormatBase and
 PhasedFileSystem.<br />(Amareshwari Sriramadasu via enis)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-2116">HADOOP-2116</a>. Changes the layout of the task execution directory.<br />(Amareshwari Sriramadasu via ddas)</li>
     </ol>
   </li>
   <li><a href="javascript:toggleList('trunk_(unreleased_changes)_._new_features_')">  NEW FEATURES
@@ -101,7 +102,7 @@ build or update Lucene indexes using Map/Reduce.<br />(Ning Li via cutting)</li>
     </ol>
   </li>
   <li><a href="javascript:toggleList('trunk_(unreleased_changes)_._improvements_')">  IMPROVEMENTS
-</a>&nbsp;&nbsp;&nbsp;(20)
+</a>&nbsp;&nbsp;&nbsp;(22)
     <ol id="trunk_(unreleased_changes)_._improvements_">
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-2655">HADOOP-2655</a>. Copy on write for data and metadata files in the
 presence of snapshots. Needed for supporting appends to HDFS
@@ -143,6 +144,8 @@ requests or server slowdown.<br />(Hairong Kuang via dhruba)</li>
 the cluster directory.<br />(Hemanth Yamijala via ddas)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-2899">HADOOP-2899</a>. [HOD] Cleans up hdfs:///mapredsystem directory after
 deallocation.<br />(Hemanth Yamijala via ddas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-2886">HADOOP-2886</a>.  Track individual RPC metrics.<br />(girish vaitheeswaran via dhruba)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-2373">HADOOP-2373</a>. Improvement in safe-mode reporting.<br />(shv)</li>
     </ol>
   </li>
   <li><a href="javascript:toggleList('trunk_(unreleased_changes)_._optimizations_')">  OPTIMIZATIONS
@@ -168,7 +171,7 @@ via a single rpc to the namenode to speed up job planning.<br />(Lohit Vijaya Re
     </ol>
   </li>
   <li><a href="javascript:toggleList('trunk_(unreleased_changes)_._bug_fixes_')">  BUG FIXES
-</a>&nbsp;&nbsp;&nbsp;(51)
+</a>&nbsp;&nbsp;&nbsp;(53)
     <ol id="trunk_(unreleased_changes)_._bug_fixes_">
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-2195">HADOOP-2195</a>. '-mkdir' behaviour is now closer to Linux shell in case of
 errors.<br />(Mahadev Konar via rangadi)</li>
@@ -268,6 +271,10 @@ Shvachko via cdouglas)</li>
 hod/hodlib/Common/xmlrpc.py.<br />(Vinod Kumar Vavilapalli via ddas)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-2936">HADOOP-2936</a>. Fixes HOD in a way that it generates hdfs://host:port on the
 client side configs.<br />(Vinod Kumar Vavilapalli via ddas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3025">HADOOP-3025</a>. ChecksumFileSystem supports the delete method with
+the recursive flag.<br />(Mahadev Konar via dhruba)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3012">HADOOP-3012</a>. dfs -mv file to user home directory throws exception if
+the user home directory does not exist.<br />(Mahadev Konar via dhruba)</li>
     </ol>
   </li>
 </ul>

docs/hod_user_guide.html

@@ -1080,10 +1080,18 @@ document.write("Last Published: " + document.lastModified);
         <td colspan="1" rowspan="1"> An operation which requires an allocated cluster is given a cluster directory with no state information. </td>
       
 </tr>
-    
+   
+      
+<tr>
+        
+<td colspan="1" rowspan="1"> Any non-zero exit code </td>
+        <td colspan="1" rowspan="1"> HOD script error </td>
+        <td colspan="1" rowspan="1"> If the hod script option was used, it is likely that the exit code is from the script. Unfortunately, this could clash with the exit codes of the hod command itself. In order to help users differentiate these two, hod writes the script's exit code to a file called script.exitcode in the cluster directory, if the script returned an exit code. You can cat this file to determine the script's exit code. If it does not exist, then it is a hod command exit code.</td> 
+      
+</tr>
   
 </table>
-<a name="N1072E"></a><a name="Hadoop+Jobs+Not+Running+on+a+Successfully+Allocated+Cluster"></a>
+<a name="N10740"></a><a name="Hadoop+Jobs+Not+Running+on+a+Successfully+Allocated+Cluster"></a>
 <h3 class="h4"> Hadoop Jobs Not Running on a Successfully Allocated Cluster </h3>
 <a name="Hadoop_Jobs_Not_Running_on_a_Suc" id="Hadoop_Jobs_Not_Running_on_a_Suc"></a>
 <p>This scenario generally occurs when a cluster is allocated, and is left inactive for sometime, and then hadoop jobs are attempted to be run on them. Then Hadoop jobs fail with the following exception:</p>
@@ -1102,31 +1110,31 @@ document.write("Last Published: " + document.lastModified);
 <em>Possible Cause:</em> There is a version mismatch between the version of the hadoop client being used to submit jobs and the hadoop used in provisioning (typically via the tarball option). Ensure compatible versions are being used.</p>
 <p>
 <em>Possible Cause:</em> You used one of the options for specifying Hadoop configuration <span class="codefrag">-M or -H</span>, which had special characters like space or comma that were not escaped correctly. Refer to the section <em>Options Configuring HOD</em> for checking how to specify such options correctly.</p>
-<a name="N10769"></a><a name="My+Hadoop+Job+Got+Killed"></a>
+<a name="N1077B"></a><a name="My+Hadoop+Job+Got+Killed"></a>
 <h3 class="h4"> My Hadoop Job Got Killed </h3>
 <a name="My_Hadoop_Job_Got_Killed" id="My_Hadoop_Job_Got_Killed"></a>
 <p>
 <em>Possible Cause:</em> The wallclock limit specified by the Torque administrator or the <span class="codefrag">-l</span> option defined in the section <em>Specifying Additional Job Attributes</em> was exceeded since allocation time. Thus the cluster would have got released. Deallocate the cluster and allocate it again, this time with a larger wallclock time.</p>
 <p>
 <em>Possible Cause:</em> Problems with the JobTracker node. Refer to the section in <em>Collecting and Viewing Hadoop Logs</em> to get more information.</p>
-<a name="N10784"></a><a name="Hadoop+Job+Fails+with+Message%3A+%27Job+tracker+still+initializing%27"></a>
+<a name="N10796"></a><a name="Hadoop+Job+Fails+with+Message%3A+%27Job+tracker+still+initializing%27"></a>
 <h3 class="h4"> Hadoop Job Fails with Message: 'Job tracker still initializing' </h3>
 <a name="Hadoop_Job_Fails_with_Message_Jo" id="Hadoop_Job_Fails_with_Message_Jo"></a>
 <p>
 <em>Possible Cause:</em> The hadoop job was being run as part of the HOD script command, and it started before the JobTracker could come up fully. Allocate the cluster using a large value for the configuration option <span class="codefrag">--hod.script-wait-time</span>. Typically a value of 120 should work, though it is typically unnecessary to be that large.</p>
-<a name="N10794"></a><a name="The+Exit+Codes+For+HOD+Are+Not+Getting+Into+Torque"></a>
+<a name="N107A6"></a><a name="The+Exit+Codes+For+HOD+Are+Not+Getting+Into+Torque"></a>
 <h3 class="h4"> The Exit Codes For HOD Are Not Getting Into Torque </h3>
 <a name="The_Exit_Codes_For_HOD_Are_Not_G" id="The_Exit_Codes_For_HOD_Are_Not_G"></a>
 <p>
 <em>Possible Cause:</em> Version 0.16 of hadoop is required for this functionality to work. The version of Hadoop used does not match. Use the required version of Hadoop.</p>
 <p>
 <em>Possible Cause:</em> The deallocation was done without using the <span class="codefrag">hod</span> command; for e.g. directly using <span class="codefrag">qdel</span>. When the cluster is deallocated in this manner, the HOD processes are terminated using signals. This results in the exit code to be based on the signal number, rather than the exit code of the program.</p>
-<a name="N107AC"></a><a name="The+Hadoop+Logs+are+Not+Uploaded+to+DFS"></a>
+<a name="N107BE"></a><a name="The+Hadoop+Logs+are+Not+Uploaded+to+DFS"></a>
 <h3 class="h4"> The Hadoop Logs are Not Uploaded to DFS </h3>
 <a name="The_Hadoop_Logs_are_Not_Uploaded" id="The_Hadoop_Logs_are_Not_Uploaded"></a>
 <p>
 <em>Possible Cause:</em> There is a version mismatch between the version of the hadoop being used for uploading the logs and the external HDFS. Ensure that the correct version is specified in the <span class="codefrag">hodring.pkgs</span> option.</p>
-<a name="N107BC"></a><a name="Locating+Ringmaster+Logs"></a>
+<a name="N107CE"></a><a name="Locating+Ringmaster+Logs"></a>
 <h3 class="h4"> Locating Ringmaster Logs </h3>
 <a name="Locating_Ringmaster_Logs" id="Locating_Ringmaster_Logs"></a>
 <p>To locate the ringmaster logs, follow these steps: </p>
@@ -1143,7 +1151,7 @@ document.write("Last Published: " + document.lastModified);
 <li> If you don't get enough information, you may want to set the ringmaster debug level to 4. This can be done by passing <span class="codefrag">--ringmaster.debug 4</span> to the hod command line.</li>
   
 </ul>
-<a name="N107E8"></a><a name="Locating+Hodring+Logs"></a>
+<a name="N107FA"></a><a name="Locating+Hodring+Logs"></a>
 <h3 class="h4"> Locating Hodring Logs </h3>
 <a name="Locating_Hodring_Logs" id="Locating_Hodring_Logs"></a>
 <p>To locate hodring logs, follow the steps below: </p>

src/contrib/hod/hodlib/Common/setup.py

@@ -600,6 +600,7 @@ class formatter(IndentedHelpFormatter):
         return ", ".join(opts)    
 
 class options(OptionParser, baseConfig):
+
     def __init__(self, optionDef, usage, version, originalDir=None, 
                  withConfig=False, defaultConfig=None, defaultLocation=None,
                  name=None):
@@ -730,6 +731,9 @@ class options(OptionParser, baseConfig):
               clusterDir = getattr(self.__parsedOptions, 'hod.clusterdir')
               numNodes = getattr(self.__parsedOptions, 'hod.nodecount')
 
+              if clusterDir:
+                self.remove_exit_code_file(originalDir, clusterDir)
+
               if not _script or not clusterDir or not numNodes:
                 print getattr(hodhelp, "help_%s" % cmdstr)()
                 sys.exit(3)
@@ -771,6 +775,24 @@ class options(OptionParser, baseConfig):
                         self.__defaultLoc, self.config)
         self.__build_dict()   
 
+    def norm_cluster_dir(self, orig_dir, directory):
+        directory = os.path.expanduser(directory)
+        if not os.path.isabs(directory):
+            directory = os.path.join(orig_dir, directory)
+        directory = os.path.abspath(directory)
+
+        return directory
+
+
+    def remove_exit_code_file(self, orig_dir, dir):
+        try:
+            dir = self.norm_cluster_dir(orig_dir, dir)
+            if os.path.exists(dir):
+                exit_code_file = os.path.join(dir, "script.exitcode")
+                if os.path.exists(exit_code_file):
+                    os.remove(exit_code_file)
+        except:
+            print >>sys.stderr, "Could not remove the script.exitcode file."
     
     def __init_display_options(self):
         self.__orig_option_list = self.option_list[:]

src/contrib/hod/hodlib/Hod/hod.py

@@ -453,6 +453,7 @@ class hodRunner:
   def script(self):
     errorFlag = False
     errorMsgs = []
+    scriptRet = 0 # return from the script, if run
     
     script = self.__cfg['hod']['script']
     nodes = self.__cfg['hod']['nodecount']
@@ -480,9 +481,9 @@ class hodRunner:
     if errorFlag:
       for msg in errorMsgs:
         self.__log.critical(msg)
+      self.handle_script_exit_code(scriptRet, clusterDir)
       sys.exit(3)
 
-    ret = 0
     try:
       self._op_allocate(('allocate', clusterDir, str(nodes)))
       if self.__opCode == 0:
@@ -495,8 +496,8 @@ class hodRunner:
           scriptRunner = hadoopScript(clusterDir, 
                                   self.__cfg['hod']['original-dir'])
           self.__opCode = scriptRunner.run(script)
-          ret = self.__opCode
-          self.__log.debug("Exit code from running the script: %d" % self.__opCode)
+          scriptRet = self.__opCode
+          self.__log.info("Exit code from running the script: %d" % self.__opCode)
       else:
         self.__log.critical("Error %d in allocating the cluster. Cannot run the script." % self.__opCode)
 
@@ -513,14 +514,28 @@ class hodRunner:
                           get_exception_error_string()))
       self.__log.debug(get_exception_string())
     
-    self.__cleanup()      
-    
-    # We want to give importance to a failed script's exit code.
-    if ret != 0:
-      self.__opCode = ret
+    self.__cleanup()
 
+    self.handle_script_exit_code(scriptRet, clusterDir)
+    
     return self.__opCode
 
+  def handle_script_exit_code(self, scriptRet, clusterDir):
+    # We want to give importance to a failed script's exit code, and write out exit code to a file separately
+    # so users can easily get it if required. This way they can differentiate between the script's exit code
+    # and hod's exit code.
+    if os.path.exists(clusterDir):
+      exit_code_file_name = (os.path.join(clusterDir, 'script.exitcode'))
+      if scriptRet != 0:
+        exit_code_file = open(exit_code_file_name, 'w')
+        print >>exit_code_file, scriptRet
+        exit_code_file.close()
+        self.__opCode = scriptRet
+      else:
+        #ensure script exit code file is not there:
+        if (os.path.exists(exit_code_file_name)):
+          os.remove(exit_code_file_name)
+
 class hodHelp:
   def __init__(self):
     self.ops = ['allocate', 'deallocate', 'info', 'list','script',  'help']

src/docs/src/documentation/content/xdocs/hod_user_guide.xml

@@ -464,7 +464,12 @@
         <td> Cluster not allocated </td>
         <td> An operation which requires an allocated cluster is given a cluster directory with no state information. </td>
       </tr>
-    
+   
+      <tr>
+        <td> Any non-zero exit code </td>
+        <td> HOD script error </td>
+        <td> If the hod script option was used, it is likely that the exit code is from the script. Unfortunately, this could clash with the exit codes of the hod command itself. In order to help users differentiate these two, hod writes the script's exit code to a file called script.exitcode in the cluster directory, if the script returned an exit code. You can cat this file to determine the script's exit code. If it does not exist, then it is a hod command exit code.</td> 
+      </tr>
   </table>
     </section>
   <section><title> Hadoop Jobs Not Running on a Successfully Allocated Cluster </title><anchor id="Hadoop_Jobs_Not_Running_on_a_Suc"></anchor>