HADOOP-3376: Provide a mechanism to detect and handle violations to resource manager limits. Contributed by Vinod Kumar Vavilapalli.

git-svn-id: https://svn.apache.org/repos/asf/hadoop/core/trunk@662795 13f79535-47bb-0310-9956-ffa450edef68

docs/changes.html

@@ -56,7 +56,7 @@
 </a></h2>
 <ul id="trunk_(unreleased_changes)_">
   <li><a href="javascript:toggleList('trunk_(unreleased_changes)_._incompatible_changes_')">  INCOMPATIBLE CHANGES
-</a>&nbsp;&nbsp;&nbsp;(11)
+</a>&nbsp;&nbsp;&nbsp;(14)
     <ol id="trunk_(unreleased_changes)_._incompatible_changes_">
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-2703">HADOOP-2703</a>.  The default options to fsck skips checking files
 that are being written to. The output of fsck is incompatible
@@ -97,11 +97,19 @@ Existing blocks get a generation stamp of 0. This is needed to support
 appends.<br />(dhruba)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3390">HADOOP-3390</a>. Removed deprecated ClientProtocol.abandonFileInProgress().
 (Tsz Wo (Nicholas), SZE via rangadi)
+</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3405">HADOOP-3405</a>. Made some map/reduce internal classes non-public:
+MapTaskStatus, ReduceTaskStatus, JobSubmissionProtocol,
+CompletedJobStatusStore.<br />(enis via omaley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3265">HADOOP-3265</a>. Removed depcrecated API getFileCacheHints().<br />(Lohit Vijayarenu via rangadi)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3310">HADOOP-3310</a>. The namenode instructs the primary datanode to do lease
+recovery. The block gets a new  generation stamp.
+(Tsz Wo (Nicholas), SZE via dhruba)
 </li>
     </ol>
   </li>
   <li><a href="javascript:toggleList('trunk_(unreleased_changes)_._new_features_')">  NEW FEATURES
-</a>&nbsp;&nbsp;&nbsp;(11)
+</a>&nbsp;&nbsp;&nbsp;(15)
     <ol id="trunk_(unreleased_changes)_._new_features_">
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3074">HADOOP-3074</a>. Provides a UrlStreamHandler for DFS and other FS,
 relying on FileSystem<br />(taton)</li>
@@ -129,10 +137,18 @@ configuration property "mapred.line.input.format.linespermap", which
 defaults to 1.<br />(Amareshwari Sriramadasu via ddas)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3336">HADOOP-3336</a>. Direct a subset of annotated FSNamesystem calls for audit
 logging.<br />(cdouglas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3400">HADOOP-3400</a>. A new API FileSystem.deleteOnExit() that facilitates
+handling of temporary files in HDFS.<br />(dhruba)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-4">HADOOP-4</a>.  Add fuse-dfs to contrib, permitting one to mount an
+HDFS filesystem on systems that support FUSE, e.g., Linux.<br />(Pete Wyckoff via cutting)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3246">HADOOP-3246</a>. Add FTPFileSystem.<br />(Ankur Goel via cutting)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3250">HADOOP-3250</a>. Extend FileSystem API to allow appending to files.
+(Tsz Wo (Nicholas), SZE via cdouglas)
+</li>
     </ol>
   </li>
   <li><a href="javascript:toggleList('trunk_(unreleased_changes)_._improvements_')">  IMPROVEMENTS
-</a>&nbsp;&nbsp;&nbsp;(19)
+</a>&nbsp;&nbsp;&nbsp;(22)
     <ol id="trunk_(unreleased_changes)_._improvements_">
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-2928">HADOOP-2928</a>. Remove deprecated FileSystem.getContentLength().<br />(Lohit Vjayarenu via rangadi)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3130">HADOOP-3130</a>. Make the connect timeout smaller for getFile.<br />(Amar Ramesh Kamat via ddas)</li>
@@ -173,6 +189,13 @@ in backoff calculation.<br />(cdouglas)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3381">HADOOP-3381</a>. Clear referenced when directories are deleted so that
 effect of memory leaks are not multiplied.<br />(rangadi)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-2867">HADOOP-2867</a>. Adds the task's CWD to its LD_LIBRARY_PATH.<br />(Amareshwari Sriramadasu via ddas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3232">HADOOP-3232</a>. DU class runs the 'du' command in a seperate thread so
+that it does not block user. DataNode misses heartbeats in large
+nodes otherwise.<br />(Johan Oskarsson via rangadi)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3035">HADOOP-3035</a>. During block transfers between datanodes, the receiving
+datanode, now can report corrupt replicas received from src node to
+the namenode.<br />(Lohit Vijayarenu via rangadi)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3434">HADOOP-3434</a>. Retain the cause of the bind failure in Server::bind.<br />(Steve Loughran via cdouglas)</li>
     </ol>
   </li>
   <li><a href="javascript:toggleList('trunk_(unreleased_changes)_._optimizations_')">  OPTIMIZATIONS
@@ -197,7 +220,7 @@ DataNodes take 30% less CPU while writing data.<br />(rangadi)</li>
     </ol>
   </li>
   <li><a href="javascript:toggleList('trunk_(unreleased_changes)_._bug_fixes_')">  BUG FIXES
-</a>&nbsp;&nbsp;&nbsp;(42)
+</a>&nbsp;&nbsp;&nbsp;(58)
     <ol id="trunk_(unreleased_changes)_._bug_fixes_">
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-2905">HADOOP-2905</a>. 'fsck -move' triggers NPE in NameNode.<br />(Lohit Vjayarenu via rangadi)</li>
       <li>Increment ClientProtocol.versionID missed by <a href="http://issues.apache.org/jira/browse/HADOOP-2585">HADOOP-2585</a>.<br />(shv)</li>
@@ -287,6 +310,39 @@ accommodate collectors that don't support long values.<br />(lohit vijayarenu
 via cdouglas)</li>
       <li><a href="http://issues.apache.org/jira/browse/HADOOP-3403">HADOOP-3403</a>. Fixes a problem in the JobTracker to do with handling of lost
 tasktrackers.<br />(Arun Murthy via ddas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-1318">HADOOP-1318</a>. Completed maps are not failed if the number of reducers are
+zero. (Amareshwari Sriramadasu via ddas).
+</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3351">HADOOP-3351</a>. Fixes the history viewer tool to not do huge StringBuffer
+allocations.<br />(Amareshwari Sriramadasu via ddas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3419">HADOOP-3419</a>. Fixes TestFsck to wait for updates to happen before
+checking results to make the test more reliable.<br />(Lohit Vijaya
+Renu via omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3259">HADOOP-3259</a>. Makes failure to read system properties due to a
+security manager non-fatal.<br />(Edward Yoon via omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3451">HADOOP-3451</a>. Update libhdfs to use FileSystem::getFileBlockLocations
+instead of removed getFileCacheHints.<br />(lohit vijayarenu via cdouglas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3401">HADOOP-3401</a>. Update FileBench to set the new
+"mapred.work.output.dir" property to work post-3041.<br />(cdouglas via omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-2159">HADOOP-2159</a> Namenode stuck in safemode. The counter blockSafe should
+not be decremented for invalid blocks.<br />(hairong)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-2669">HADOOP-2669</a>. DFSClient locks pendingCreates appropriately.<br />(dhruba)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3477">HADOOP-3477</a>. Fix build to not package contrib/*/bin twice in
+distributions.<br />(Adam Heath via cutting)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3410">HADOOP-3410</a>. Fix KFS implemenation to return correct file
+modification time.<br />(Sriram Rao via cutting)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3340">HADOOP-3340</a>. Fix DFS metrics for BlocksReplicated, HeartbeatsNum, and
+BlockReportsAverageTime.<br />(lohit vijayarenu via cdouglas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3435">HADOOP-3435</a>. Remove the assuption in the scripts that bash is at
+/bin/bash and fix the test patch to require bash instead of sh.<br />(Brice Arnould via omalley)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3471">HADOOP-3471</a>. Fix spurious errors from TestIndexedSort and add additional
+logging to let failures be reproducible.<br />(cdouglas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3475">HADOOP-3475</a>. Fix MapTask to correctly size the accounting allocation of
+io.sort.mb.<br />(cdouglas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3443">HADOOP-3443</a>. Avoid copying map output across partitions when renaming a
+single spill.<br />(omalley via cdouglas)</li>
+      <li><a href="http://issues.apache.org/jira/browse/HADOOP-3454">HADOOP-3454</a>. Fix Text::find to search only valid byte ranges.<br />(Chad Whipkey
+via cdouglas)</li>
     </ol>
   </li>
 </ul>

docs/hadoop-default.html

@@ -114,6 +114,9 @@ creations/deletions), or "all".</td>
 <td><a name="fs.hsftp.impl">fs.hsftp.impl</a></td><td>org.apache.hadoop.dfs.HsftpFileSystem</td><td></td>
 </tr>
 <tr>
+<td><a name="fs.ftp.impl">fs.ftp.impl</a></td><td>org.apache.hadoop.fs.ftp.FTPFileSystem</td><td>The FileSystem for ftp: uris.</td>
+</tr>
+<tr>
 <td><a name="fs.ramfs.impl">fs.ramfs.impl</a></td><td>org.apache.hadoop.fs.InMemoryFileSystem</td><td>The FileSystem for ramfs: uris.</td>
 </tr>
 <tr>

docs/hod_admin_guide.html

@@ -222,6 +222,14 @@ document.write("Last Published: " + document.lastModified);
 </li>
 </ul>
 </li>
+<li>
+<a href="#checklimits.sh+-+Tool+to+update+torque+comment+field+reflecting+resource+limits">checklimits.sh - Tool to update torque comment field reflecting resource limits</a>
+<ul class="minitoc">
+<li>
+<a href="#Running+checklimits.sh">Running checklimits.sh</a>
+</li>
+</ul>
+</li>
 </ul>
 </li>
 </ul>
@@ -586,6 +594,30 @@ it.</p>
 <p>
 <em>python logcondense.py -p ~/hadoop-0.17.0/bin/hadoop -d 7 -c ~/hadoop-conf -l /user</em>
 </p>
+<a name="N10205"></a><a name="checklimits.sh+-+Tool+to+update+torque+comment+field+reflecting+resource+limits"></a>
+<h3 class="h4">checklimits.sh - Tool to update torque comment field reflecting resource limits</h3>
+<p>checklimits is a HOD tool specific to torque/maui environment. It
+      updates torque comment field when newly submitted job(s) violate/cross
+      over user limits set up in maui scheduler. It uses qstat, does one pass
+      over torque job list to find out queued or unfinished jobs, runs maui
+      tool checkjob on each job to see if user limits are violated and then
+      runs torque's qalter utility to update job attribute 'comment'. Currently
+      it updates the comment as <em>User-limits exceeded. Requested:([0-9]*)
+      Used:([0-9]*) MaxLimit:([0-9]*)</em> for those jobs that violate limits.
+      This comment field is then used by HOD to behave accordingly depending on
+      the type of violation.</p>
+<a name="N10211"></a><a name="Running+checklimits.sh"></a>
+<h4>Running checklimits.sh</h4>
+<p>checklimits.sh is available under hod_install_location/support
+        folder. This is a shell script and can be run directly as <em>sh
+        checklimits.sh </em>or as <em>./checklimits.sh</em> after enabling
+        execute permissions. In order for this tool to be able to update
+        comment field of jobs from different users, it has to be run with
+        torque administrative privileges. This tool has to be run repeatedly
+        after specific intervals of time to frequently update jobs violating
+        constraints, for e.g. via cron. Please note that the resource manager
+        and scheduler commands used in this script can be expensive and so
+        it is better not to run this inside a tight loop without sleeping.</p>
 </div>
 
 </div>

docs/hod_config_guide.html

@@ -320,9 +320,30 @@ document.write("Last Published: " + document.lastModified);
                            specified as key-value pairs. These will be used to
                            generate a hadoop-site.xml on the submit node that 
                            should be used for running MapReduce jobs.</li>
+          
+<li>job-feasibility-attr: A regular expression string that specifies
+                           whether and how to check job feasibility - resource
+                           manager or scheduler limits. The current
+                           implementation corresponds to the torque job
+                           attribute 'comment' and by default is disabled.
+                           When set, HOD uses it to decide what type
+                           of limit violation is triggered and either
+                           deallocates the cluster or stays in queued state
+                           according as the request is beyond maximum limits or
+                           the cumulative usage has crossed maxumum limits. 
+                           The torque comment attribute may be updated
+                           periodically by an external mechanism. For e.g.,
+                           comment attribute can be updated by running <a href="hod_admin_guide.html#checklimits.sh+-+Tool+to+update+torque+comment+field+reflecting+resource+limits">
+                           checklimits.sh</a> script in hod/support directory,
+                           and then setting job-feasibility-attr equal to the
+                           value TORQUE_USER_LIMITS_COMMENT_FIELD i.e
+                           "User-limits exceeded. Requested:([0-9]*)
+                           Used:([0-9]*) MaxLimit:([0-9]*)" will make HOD
+                           behave accordingly.
+                           </li>
          
 </ul>
-<a name="N1007E"></a><a name="3.3+resource_manager+options"></a>
+<a name="N10085"></a><a name="3.3+resource_manager+options"></a>
 <h3 class="h4">3.3 resource_manager options</h3>
 <ul>
           
@@ -345,7 +366,7 @@ document.write("Last Published: " + document.lastModified);
                       can then use this variable.</li>
         
 </ul>
-<a name="N10091"></a><a name="3.4+ringmaster+options"></a>
+<a name="N10098"></a><a name="3.4+ringmaster+options"></a>
 <h3 class="h4">3.4 ringmaster options</h3>
 <ul>
           
@@ -358,7 +379,7 @@ document.write("Last Published: " + document.lastModified);
                        and notes for the temp-dir variable apply here too.</li>
         
 </ul>
-<a name="N1009E"></a><a name="3.5+gridservice-hdfs+options"></a>
+<a name="N100A5"></a><a name="3.5+gridservice-hdfs+options"></a>
 <h3 class="h4">3.5 gridservice-hdfs options</h3>
 <ul>
           
@@ -399,7 +420,7 @@ document.write("Last Published: " + document.lastModified);
 <li>final-server-params: Same as above, except they will be marked final.</li>
         
 </ul>
-<a name="N100BD"></a><a name="3.6+gridservice-mapred+options"></a>
+<a name="N100C4"></a><a name="3.6+gridservice-mapred+options"></a>
 <h3 class="h4">3.6 gridservice-mapred options</h3>
 <ul>
           
@@ -432,7 +453,7 @@ document.write("Last Published: " + document.lastModified);
 <li>final-server-params: Same as above, except they will be marked final.</li>
         
 </ul>
-<a name="N100DC"></a><a name="3.7+hodring+options"></a>
+<a name="N100E3"></a><a name="3.7+hodring+options"></a>
 <h3 class="h4">3.7 hodring options</h3>
 <ul>
           

docs/hod_user_guide.html

@@ -994,7 +994,8 @@ document.write("Last Published: " + document.lastModified);
 <td colspan="1" rowspan="1"> 4 </td>
         <td colspan="1" rowspan="1"> Scheduler failure </td>
         <td colspan="1" rowspan="1"> 1. Requested more resources than available. Run <span class="codefrag">checknodes cluster_name</span> to see if enough nodes are available. <br>
-          2. Torque is misconfigured, the path to Torque binaries is misconfigured, or other Torque problems. Contact system administrator. </td>
+          2. Requested resources exceed resource manager limits. <br>
+          3. Torque is misconfigured, the path to Torque binaries is misconfigured, or other Torque problems. Contact system administrator. </td>
       
 </tr>
       
@@ -1094,7 +1095,7 @@ document.write("Last Published: " + document.lastModified);
 </tr>
   
 </table>
-<a name="N10740"></a><a name="Hadoop+Jobs+Not+Running+on+a+Successfully+Allocated+Cluster"></a>
+<a name="N10742"></a><a name="Hadoop+Jobs+Not+Running+on+a+Successfully+Allocated+Cluster"></a>
 <h3 class="h4"> Hadoop Jobs Not Running on a Successfully Allocated Cluster </h3>
 <a name="Hadoop_Jobs_Not_Running_on_a_Suc" id="Hadoop_Jobs_Not_Running_on_a_Suc"></a>
 <p>This scenario generally occurs when a cluster is allocated, and is left inactive for sometime, and then hadoop jobs are attempted to be run on them. Then Hadoop jobs fail with the following exception:</p>
@@ -1113,31 +1114,31 @@ document.write("Last Published: " + document.lastModified);
 <em>Possible Cause:</em> There is a version mismatch between the version of the hadoop client being used to submit jobs and the hadoop used in provisioning (typically via the tarball option). Ensure compatible versions are being used.</p>
 <p>
 <em>Possible Cause:</em> You used one of the options for specifying Hadoop configuration <span class="codefrag">-M or -H</span>, which had special characters like space or comma that were not escaped correctly. Refer to the section <em>Options Configuring HOD</em> for checking how to specify such options correctly.</p>
-<a name="N1077B"></a><a name="My+Hadoop+Job+Got+Killed"></a>
+<a name="N1077D"></a><a name="My+Hadoop+Job+Got+Killed"></a>
 <h3 class="h4"> My Hadoop Job Got Killed </h3>
 <a name="My_Hadoop_Job_Got_Killed" id="My_Hadoop_Job_Got_Killed"></a>
 <p>
 <em>Possible Cause:</em> The wallclock limit specified by the Torque administrator or the <span class="codefrag">-l</span> option defined in the section <em>Specifying Additional Job Attributes</em> was exceeded since allocation time. Thus the cluster would have got released. Deallocate the cluster and allocate it again, this time with a larger wallclock time.</p>
 <p>
 <em>Possible Cause:</em> Problems with the JobTracker node. Refer to the section in <em>Collecting and Viewing Hadoop Logs</em> to get more information.</p>
-<a name="N10796"></a><a name="Hadoop+Job+Fails+with+Message%3A+%27Job+tracker+still+initializing%27"></a>
+<a name="N10798"></a><a name="Hadoop+Job+Fails+with+Message%3A+%27Job+tracker+still+initializing%27"></a>
 <h3 class="h4"> Hadoop Job Fails with Message: 'Job tracker still initializing' </h3>
 <a name="Hadoop_Job_Fails_with_Message_Jo" id="Hadoop_Job_Fails_with_Message_Jo"></a>
 <p>
 <em>Possible Cause:</em> The hadoop job was being run as part of the HOD script command, and it started before the JobTracker could come up fully. Allocate the cluster using a large value for the configuration option <span class="codefrag">--hod.script-wait-time</span>. Typically a value of 120 should work, though it is typically unnecessary to be that large.</p>
-<a name="N107A6"></a><a name="The+Exit+Codes+For+HOD+Are+Not+Getting+Into+Torque"></a>
+<a name="N107A8"></a><a name="The+Exit+Codes+For+HOD+Are+Not+Getting+Into+Torque"></a>
 <h3 class="h4"> The Exit Codes For HOD Are Not Getting Into Torque </h3>
 <a name="The_Exit_Codes_For_HOD_Are_Not_G" id="The_Exit_Codes_For_HOD_Are_Not_G"></a>
 <p>
 <em>Possible Cause:</em> Version 0.16 of hadoop is required for this functionality to work. The version of Hadoop used does not match. Use the required version of Hadoop.</p>
 <p>
 <em>Possible Cause:</em> The deallocation was done without using the <span class="codefrag">hod</span> command; for e.g. directly using <span class="codefrag">qdel</span>. When the cluster is deallocated in this manner, the HOD processes are terminated using signals. This results in the exit code to be based on the signal number, rather than the exit code of the program.</p>
-<a name="N107BE"></a><a name="The+Hadoop+Logs+are+Not+Uploaded+to+DFS"></a>
+<a name="N107C0"></a><a name="The+Hadoop+Logs+are+Not+Uploaded+to+DFS"></a>
 <h3 class="h4"> The Hadoop Logs are Not Uploaded to DFS </h3>
 <a name="The_Hadoop_Logs_are_Not_Uploaded" id="The_Hadoop_Logs_are_Not_Uploaded"></a>
 <p>
 <em>Possible Cause:</em> There is a version mismatch between the version of the hadoop being used for uploading the logs and the external HDFS. Ensure that the correct version is specified in the <span class="codefrag">hodring.pkgs</span> option.</p>
-<a name="N107CE"></a><a name="Locating+Ringmaster+Logs"></a>
+<a name="N107D0"></a><a name="Locating+Ringmaster+Logs"></a>
 <h3 class="h4"> Locating Ringmaster Logs </h3>
 <a name="Locating_Ringmaster_Logs" id="Locating_Ringmaster_Logs"></a>
 <p>To locate the ringmaster logs, follow these steps: </p>
@@ -1154,7 +1155,7 @@ document.write("Last Published: " + document.lastModified);
 <li> If you don't get enough information, you may want to set the ringmaster debug level to 4. This can be done by passing <span class="codefrag">--ringmaster.debug 4</span> to the hod command line.</li>
   
 </ul>
-<a name="N107FA"></a><a name="Locating+Hodring+Logs"></a>
+<a name="N107FC"></a><a name="Locating+Hodring+Logs"></a>
 <h3 class="h4"> Locating Hodring Logs </h3>
 <a name="Locating_Hodring_Logs" id="Locating_Hodring_Logs"></a>
 <p>To locate hodring logs, follow the steps below: </p>

src/contrib/hod/CHANGES.txt

@@ -8,6 +8,10 @@ Trunk (unreleased changes)
   NEW FEATURES
 
   IMPROVEMENTS
+
+    HADOOP-3376: Provide a mechanism to detect and handle violations to 
+    resource manager limits. (Vinod Kumar Vavilapalli via ddas)
+
  
   BUG FIXES
 

src/contrib/hod/bin/hod

@@ -49,7 +49,8 @@ from hodlib.Common.setup import *
 from hodlib.Common.descGenerator import *
 from hodlib.Common.util import local_fqdn, need_to_allocate, filter_warnings,\
     get_exception_error_string, hodInterrupt, \
-    HOD_INTERRUPTED_MESG, HOD_INTERRUPTED_CODE
+    HOD_INTERRUPTED_MESG, HOD_INTERRUPTED_CODE,\
+    TORQUE_USER_LIMITS_COMMENT_FIELD
 from hodlib.Common.tcp import tcpError, tcpSocket
 from hodlib.Hod.hod import hodHelp
 
@@ -145,6 +146,9 @@ defList = { 'hod' : (
              ('update-worker-info', 'bool', 'Specifies whether to update Worker Info after allocation',
               False, False, False, True),
 
+             ('job-feasibility-attr', 'string', 'Specifies whether to check job feasibility - resource manager and/or scheduler limits, also gives the attribute value',
+              False, None, False, True),
+
              ('title', 'string', 'Title for the current HOD allocation.',
                True, "HOD", False, True, 'N'),
 

src/contrib/hod/hodlib/Common/setup.py

@@ -34,7 +34,7 @@ reEmailDelimit = re.compile("@")
 reComma = re.compile("\s*,\s*")
 reDot = re.compile("\.")
 reCommentHack = re.compile("^.*?\s+#|;.*", flags=re.S)
-reCommentNewline = re.compile("\W$")
+reCommentNewline = re.compile("\n|\r$")
 reKeyVal = r"(?<!\\)="
 reKeyVal = re.compile(reKeyVal)
 reKeyValList = r"(?<!\\),"

src/contrib/hod/hodlib/Common/util.py

@@ -24,6 +24,10 @@ reEscapeSeq = re.compile(reEscapeSeq)
 
 HOD_INTERRUPTED_CODE = 127
 HOD_INTERRUPTED_MESG = "Hod interrupted. Cleaning up and exiting"
+TORQUE_USER_LIMITS_COMMENT_FIELD = "User-limits exceeded. " + \
+        "Requested:([0-9]*) Used:([0-9]*) MaxLimit:([0-9]*)"
+TORQUE_USER_LIMITS_EXCEEDED_MSG = "Requested number of nodes exceeded " + \
+                                  "maximum user limits. "
 
 class AlarmException(Exception):
     def __init__(self, msg=''):

src/contrib/hod/hodlib/Hod/hadoop.py

@@ -186,11 +186,14 @@ class hadoopCluster:
     count = 0
     status = False
     state = 'Q'
+    userLimitsFirstFlag = True
+
     while state == 'Q':
       if hodInterrupt.isSet():
         raise HodInterruptException()
 
-      state = self.__nodePool.getJobState()
+      jobInfo = self.__nodePool.getJobInfo()
+      state = jobInfo['job_state']
       if (state==False) or (state!='Q'):
         break
       count = count + 1
@@ -198,11 +201,28 @@ class hadoopCluster:
         time.sleep(0.5)
       else:
         time.sleep(10)
-    
+
+      if self.__cfg['hod'].has_key('job-feasibility-attr') and \
+                      self.__cfg['hod']['job-feasibility-attr']:
+        (status, msg) = self.__isJobFeasible()
+        if status == "Never":
+          self.__log.critical(TORQUE_USER_LIMITS_EXCEEDED_MSG + msg + \
+                "This cluster cannot be allocated now.")
+          return -1
+        elif status == False:
+          if userLimitsFirstFlag:
+            self.__log.critical(TORQUE_USER_LIMITS_EXCEEDED_MSG + msg + \
+                "This cluster allocation will succeed only after other " + \
+                "clusters are deallocated.")
+            userLimitsFirstFlag = False
+   
     if state and state != 'C':
       status = True
     
     return status
+
+  def __isJobFeasible(self):
+    return self.__nodePool.isJobFeasible()
   
   def __get_ringmaster_client(self):
     ringmasterXRS = None
@@ -434,13 +454,19 @@ class hadoopCluster:
         walltime = self.__cfg['hod']['walltime']
       self.jobId, exitCode = self.__nodePool.submitNodeSet(nodeSet, walltime)
       if self.jobId:
+        jobStatus = None
         try:
           jobStatus = self.__check_job_status()
         except HodInterruptException, h:
           self.__log.info(HOD_INTERRUPTED_MESG)
           self.delete_job(self.jobId)
-          self.__log.info("Job %s removed from queue." % self.jobId)
+          self.__log.info("Cluster %s removed from queue." % self.jobId)
           raise h
+        else:
+          if jobStatus == -1:
+            self.delete_job(self.jobId);
+            status = 4
+            return status
 
         if jobStatus:
           self.__log.info("Cluster Id %s" \
@@ -545,13 +571,13 @@ class hadoopCluster:
               self.__log.debug("Calling rm.stop()")
               ringClient.stopRM()
               self.__log.debug("Returning from rm.stop()")
-              self.__log.info("Job Shutdown by informing ringmaster.")
+              self.__log.info("Cluster Shutdown by informing ringmaster.")
             else:
               self.delete_job(self.jobId)
-              self.__log.info("Job %s removed from queue directly." % self.jobId)
+              self.__log.info("Cluster %s removed from queue directly." % self.jobId)
             raise h
         else:
-          self.__log.critical("No job found, ringmaster failed to run.")
+          self.__log.critical("No cluster found, ringmaster failed to run.")
           status = 5 
 
       elif self.jobId == False:

src/contrib/hod/hodlib/Hod/nodePool.py

@@ -101,13 +101,17 @@ class NodePool:
   def getServiceId(self):
     raise NotImplementedError
  
-  def getJobState(self):
+  def getJobInfo(self):
     raise NotImplementedError
 
   def deleteJob(self, jobId):
     """Delete a job, given it's id"""
     raise NotImplementedError
 
+  def isJobFeasible(self):
+    """Check if job can run by looking at any user/job limits"""
+    raise NotImplementedError
+
   def updateWorkerInfo(self, workerInfoMap):
     """Update information about the workers started by this NodePool."""
     raise NotImplementedError

src/contrib/hod/hodlib/NodePools/torque.py

@@ -21,7 +21,8 @@ import os, sys, csv, socket, time, re, pprint
 from hodlib.Hod.nodePool import *
 from hodlib.Schedulers.torque import torqueInterface
 from hodlib.Common.threads import simpleCommand
-from hodlib.Common.util import get_exception_string, args_to_string, local_fqdn
+from hodlib.Common.util import get_exception_string, args_to_string, local_fqdn, \
+                        TORQUE_USER_LIMITS_COMMENT_FIELD
 
 class TorqueNodeSet(NodeSet):
   def __init__(self, id, numNodes, preferredList, isPreemptee):
@@ -201,6 +202,7 @@ class TorquePool(NodePool):
     
     jobId, exitCode = self.__torque.qsub(argList, stdinList)
     
+    ## UNUSED CODE: LINE ##
     nodeSet.qsubId = jobId
 
     return jobId, exitCode
@@ -224,6 +226,7 @@ class TorquePool(NodePool):
       
     return status
     
+  ## UNUSED METHOD ?? ##
   def getWorkers(self):
     hosts = []
     
@@ -233,6 +236,7 @@ class TorquePool(NodePool):
     
     return hosts
  
+  ## UNUSED METHOD ?? ##
   def pollNodeSet(self, nodeSet):
     status = NodeSet.COMPLETE  
     nodeSet = self.nodeSetDict[0] 
@@ -264,17 +268,17 @@ class TorquePool(NodePool):
       
     return id
 
-  def getJobState(self):
+  def getJobInfo(self):
     #torque error code when credentials fail, a temporary condition sometimes.
     credFailureErrorCode = 171 
     credFailureRetries = 10
     i = 0
-    jobState = False
+    self.__jobInfo = None
     
     while i < credFailureRetries:
       qstatInfo, exitCode = self.__torque.qstat(self.getServiceId())
       if exitCode == 0:
-        jobState = qstatInfo['job_state'] 
+        self.__jobInfo = qstatInfo
         break
       else:
         if exitCode == credFailureErrorCode:
@@ -282,11 +286,35 @@ class TorquePool(NodePool):
           i = i+1
         else:
           break
-    return jobState
+    return self.__jobInfo
 
   def deleteJob(self, jobId):
     exitCode = self.__torque.qdel(jobId)
     return exitCode
+
+  def isJobFeasible(self):
+    comment = None
+    msg = None
+    if self.__jobInfo.has_key('comment'):
+      comment = self.__jobInfo['comment']
+    try:
+      if comment:
+        commentField = re.compile(self._cfg['hod']['job-feasibility-attr'])
+        match = commentField.search(comment)
+        if match:
+          reqUsage = int(match.group(1))
+          currentUsage = int(match.group(2))
+          maxUsage = int(match.group(3))
+          msg = "Current Usage:%s, Requested:%s, Maximum Limit:%s " % \
+                                  (currentUsage, reqUsage, maxUsage)
+          if reqUsage > maxUsage:
+            return "Never", msg
+          if reqUsage + currentUsage > maxUsage:
+            return False, msg
+    except Exception, e:
+      self._log.error("Error in isJobFeasible : %s" %e)
+      raise Exception(e)
+    return True, msg
     
   def runWorkers(self, args):
     return self.__torque.pbsdsh(args)

src/contrib/hod/support/checklimits.sh

@@ -0,0 +1,57 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+COMMANDS=( "qstat" "qalter" "checkjob" )
+ERROR=0
+for (( i=0; i<${#COMMANDS[@]}; i++ ))
+do
+  cmd=${COMMANDS[$i]}
+  CMD_PATH=`which $cmd 2>/dev/null`
+  if [ $? -ne 0 ]
+  then
+    echo Could not find $cmd in PATH
+    ERROR=1
+  fi
+done
+if [ $ERROR -ne 0 ]
+then
+  exit 1
+fi
+
+jobs=`qstat -i |grep -o -e '^[0-9]*'`
+for job in $jobs
+do
+  echo -en "$job\t"
+  PATTERN="job [^ ]* violates active HARD MAXPROC limit of \([0-9]*\) for user [^ ]*[ ]*(R: \([0-9]*\), U: \([0-9]*\))"
+  OUT=`checkjob $job 2>&1|grep -o -e "$PATTERN"`
+  if [ $? -eq 0 ]
+  then
+    echo -en "| Exceeds resource limits\t"
+    COMMENT_FIELD=`echo $OUT|sed -e "s/$PATTERN/User-limits exceeded. Requested:\2 Used:\3 MaxLimit:\1/"`
+    qstat -f $job|grep '^[ \t]*comment = .*$' >/dev/null
+    if [ $? -ne 0 ]
+    then
+      echo -en "| Comment field updated\t"
+      qalter $job -W comment="$COMMENT_FIELD" >/dev/null
+    else
+      echo -en "| Comment field already set\t"
+    fi
+  else
+    echo -en "| Doesn't exceed limits.\t"
+  fi
+  echo
+done

src/docs/src/documentation/content/xdocs/hod_admin_guide.xml

@@ -313,6 +313,32 @@ it.</p>
         <p><em>python logcondense.py -p ~/hadoop-0.17.0/bin/hadoop -d 7 -c ~/hadoop-conf -l /user</em></p>
       </section>
     </section>
+    <section>
+      <title>checklimits.sh - Tool to update torque comment field reflecting resource limits</title>
+      <p>checklimits is a HOD tool specific to torque/maui environment. It
+      updates torque comment field when newly submitted job(s) violate/cross
+      over user limits set up in maui scheduler. It uses qstat, does one pass
+      over torque job list to find out queued or unfinished jobs, runs maui
+      tool checkjob on each job to see if user limits are violated and then
+      runs torque's qalter utility to update job attribute 'comment'. Currently
+      it updates the comment as <em>User-limits exceeded. Requested:([0-9]*)
+      Used:([0-9]*) MaxLimit:([0-9]*)</em> for those jobs that violate limits.
+      This comment field is then used by HOD to behave accordingly depending on
+      the type of violation.</p>
+      <section>
+        <title>Running checklimits.sh</title>
+        <p>checklimits.sh is available under hod_install_location/support
+        folder. This is a shell script and can be run directly as <em>sh
+        checklimits.sh </em>or as <em>./checklimits.sh</em> after enabling
+        execute permissions. In order for this tool to be able to update
+        comment field of jobs from different users, it has to be run with
+        torque administrative privileges. This tool has to be run repeatedly
+        after specific intervals of time to frequently update jobs violating
+        constraints, for e.g. via cron. Please note that the resource manager
+        and scheduler commands used in this script can be expensive and so
+        it is better not to run this inside a tight loop without sleeping.</p>
+      </section>
+    </section>
   </section>
 </body>
 </document>

src/docs/src/documentation/content/xdocs/hod_config_guide.xml

@@ -104,6 +104,27 @@
                            specified as key-value pairs. These will be used to
                            generate a hadoop-site.xml on the submit node that 
                            should be used for running MapReduce jobs.</li>
+          <li>job-feasibility-attr: A regular expression string that specifies
+                           whether and how to check job feasibility - resource
+                           manager or scheduler limits. The current
+                           implementation corresponds to the torque job
+                           attribute 'comment' and by default is disabled.
+                           When set, HOD uses it to decide what type
+                           of limit violation is triggered and either
+                           deallocates the cluster or stays in queued state
+                           according as the request is beyond maximum limits or
+                           the cumulative usage has crossed maxumum limits. 
+                           The torque comment attribute may be updated
+                           periodically by an external mechanism. For e.g.,
+                           comment attribute can be updated by running <a href=
+"hod_admin_guide.html#checklimits.sh+-+Tool+to+update+torque+comment+field+reflecting+resource+limits">
+                           checklimits.sh</a> script in hod/support directory,
+                           and then setting job-feasibility-attr equal to the
+                           value TORQUE_USER_LIMITS_COMMENT_FIELD i.e
+                           "User-limits exceeded. Requested:([0-9]*)
+                           Used:([0-9]*) MaxLimit:([0-9]*)" will make HOD
+                           behave accordingly.
+                           </li>
          </ul>
       </section>
       

src/docs/src/documentation/content/xdocs/hod_user_guide.xml

@@ -404,7 +404,8 @@
         <td> 4 </td>
         <td> Scheduler failure </td>
         <td> 1. Requested more resources than available. Run <code>checknodes cluster_name</code> to see if enough nodes are available. <br />
-          2. Torque is misconfigured, the path to Torque binaries is misconfigured, or other Torque problems. Contact system administrator. </td>
+          2. Requested resources exceed resource manager limits. <br />
+          3. Torque is misconfigured, the path to Torque binaries is misconfigured, or other Torque problems. Contact system administrator. </td>
       </tr>
       <tr>
         <td> 5 </td>