apache
/
hadoop
mirror of https://github.com/apache/hadoop.git


			
				
					
						
						
							123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525
							<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<META http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta content="Apache Forrest" name="Generator">
<meta name="Forrest-version" content="0.8">
<meta name="Forrest-skin-name" content="pelt">
<title>Capacity Scheduler</title>
<link type="text/css" href="skin/basic.css" rel="stylesheet">
<link media="screen" type="text/css" href="skin/screen.css" rel="stylesheet">
<link media="print" type="text/css" href="skin/print.css" rel="stylesheet">
<link type="text/css" href="skin/profile.css" rel="stylesheet">
<script src="skin/getBlank.js" language="javascript" type="text/javascript"></script><script src="skin/getMenu.js" language="javascript" type="text/javascript"></script><script src="skin/fontsize.js" language="javascript" type="text/javascript"></script>
<link rel="shortcut icon" href="images/favicon.ico">
</head>
<body onload="init()">
<script type="text/javascript">ndeSetTextSize();</script>
<div id="top">
<!--+
    |breadtrail
    +-->
<div class="breadtrail">
<a href="http://www.apache.org/">Apache</a> &gt; <a href="http://hadoop.apache.org/">Hadoop</a> &gt; <a href="http://hadoop.apache.org/core/">Core</a><script src="skin/breadcrumbs.js" language="JavaScript" type="text/javascript"></script>
</div>
<!--+
    |header
    +-->
<div class="header">
<!--+
    |start group logo
    +-->
<div class="grouplogo">
<a href="http://hadoop.apache.org/"><img class="logoImage" alt="Hadoop" src="images/hadoop-logo.jpg" title="Apache Hadoop"></a>
</div>
<!--+
    |end group logo
    +-->
<!--+
    |start Project Logo
    +-->
<div class="projectlogo">
<a href="http://hadoop.apache.org/core/"><img class="logoImage" alt="Hadoop" src="images/core-logo.gif" title="Scalable Computing Platform"></a>
</div>
<!--+
    |end Project Logo
    +-->
<!--+
    |start Search
    +-->
<div class="searchbox">
<form action="http://www.google.com/search" method="get" class="roundtopsmall">
<input value="hadoop.apache.org" name="sitesearch" type="hidden"><input onFocus="getBlank (this, 'Search the site with google');" size="25" name="q" id="query" type="text" value="Search the site with google">&nbsp; 
                    <input name="Search" value="Search" type="submit">
</form>
</div>
<!--+
    |end search
    +-->
<!--+
    |start Tabs
    +-->
<ul id="tabs">
<li>
<a class="unselected" href="http://hadoop.apache.org/core/">Project</a>
</li>
<li>
<a class="unselected" href="http://wiki.apache.org/hadoop">Wiki</a>
</li>
<li class="current">
<a class="selected" href="index.html">Hadoop 0.19 Documentation</a>
</li>
</ul>
<!--+
    |end Tabs
    +-->
</div>
</div>
<div id="main">
<div id="publishedStrip">
<!--+
    |start Subtabs
    +-->
<div id="level2tabs"></div>
<!--+
    |end Endtabs
    +-->
<script type="text/javascript"><!--
document.write("Last Published: " + document.lastModified);
//  --></script>
</div>
<!--+
    |breadtrail
    +-->
<div class="breadtrail">

             &nbsp;
           </div>
<!--+
    |start Menu, mainarea
    +-->
<!--+
    |start Menu
    +-->
<div id="menu">
<div onclick="SwitchMenu('menu_selected_1.1', 'skin/')" id="menu_selected_1.1Title" class="menutitle" style="background-image: url('skin/images/chapter_open.gif');">Documentation</div>
<div id="menu_selected_1.1" class="selectedmenuitemgroup" style="display: block;">
<div class="menuitem">
<a href="index.html">Overview</a>
</div>
<div class="menuitem">
<a href="quickstart.html">Hadoop Quick Start</a>
</div>
<div class="menuitem">
<a href="cluster_setup.html">Hadoop Cluster Setup</a>
</div>
<div class="menuitem">
<a href="mapred_tutorial.html">Hadoop Map/Reduce Tutorial</a>
</div>
<div class="menuitem">
<a href="commands_manual.html">Hadoop Command Guide</a>
</div>
<div class="menuitem">
<a href="hdfs_shell.html">Hadoop FS Shell Guide</a>
</div>
<div class="menuitem">
<a href="distcp.html">Hadoop DistCp Guide</a>
</div>
<div class="menuitem">
<a href="native_libraries.html">Hadoop Native Libraries</a>
</div>
<div class="menuitem">
<a href="streaming.html">Hadoop Streaming</a>
</div>
<div class="menuitem">
<a href="hadoop_archives.html">Hadoop Archives</a>
</div>
<div class="menuitem">
<a href="hdfs_user_guide.html">HDFS User Guide</a>
</div>
<div class="menuitem">
<a href="hdfs_design.html">HDFS Architecture</a>
</div>
<div class="menuitem">
<a href="hdfs_permissions_guide.html">HDFS Admin Guide: Permissions</a>
</div>
<div class="menuitem">
<a href="hdfs_quota_admin_guide.html">HDFS Admin Guide: Quotas</a>
</div>
<div class="menuitem">
<a href="SLG_user_guide.html">HDFS Utilities</a>
</div>
<div class="menuitem">
<a href="libhdfs.html">HDFS C API</a>
</div>
<div class="menuitem">
<a href="hod_user_guide.html">HOD User Guide</a>
</div>
<div class="menuitem">
<a href="hod_admin_guide.html">HOD Admin Guide</a>
</div>
<div class="menuitem">
<a href="hod_config_guide.html">HOD Config Guide</a>
</div>
<div class="menupage">
<div class="menupagetitle">Capacity Scheduler</div>
</div>
<div class="menuitem">
<a href="api/index.html">API Docs</a>
</div>
<div class="menuitem">
<a href="jdiff/changes.html">API Changes</a>
</div>
<div class="menuitem">
<a href="http://wiki.apache.org/hadoop/">Wiki</a>
</div>
<div class="menuitem">
<a href="http://wiki.apache.org/hadoop/FAQ">FAQ</a>
</div>
<div class="menuitem">
<a href="releasenotes.html">Release Notes</a>
</div>
<div class="menuitem">
<a href="changes.html">Change Log</a>
</div>
</div>
<div id="credit"></div>
<div id="roundbottom">
<img style="display: none" class="corner" height="15" width="15" alt="" src="skin/images/rc-b-l-15-1body-2menu-3menu.png"></div>
<!--+
  |alternative credits
  +-->
<div id="credit2"></div>
</div>
<!--+
    |end Menu
    +-->
<!--+
    |start content
    +-->
<div id="content">
<div title="Portable Document Format" class="pdflink">
<a class="dida" href="capacity_scheduler.pdf"><img alt="PDF -icon" src="skin/images/pdfdoc.gif" class="skin"><br>
        PDF</a>
</div>
<h1>Capacity Scheduler</h1>
<div id="minitoc-area">
<ul class="minitoc">
<li>
<a href="#Purpose">Purpose</a>
</li>
<li>
<a href="#Features">Features</a>
</li>
<li>
<a href="#Picking+a+task+to+run">Picking a task to run</a>
</li>
<li>
<a href="#Reclaiming+capacity">Reclaiming capacity</a>
</li>
<li>
<a href="#Installation">Installation</a>
</li>
<li>
<a href="#Configuration">Configuration</a>
<ul class="minitoc">
<li>
<a href="#Using+the+capacity+scheduler">Using the capacity scheduler</a>
</li>
<li>
<a href="#Setting+up+queues">Setting up queues</a>
</li>
<li>
<a href="#Configuring+properties+for+queues">Configuring properties for queues</a>
</li>
<li>
<a href="#Reviewing+the+configuration+of+the+capacity+scheduler">Reviewing the configuration of the capacity scheduler</a>
</li>
</ul>
</li>
</ul>
</div>
  
    
<a name="N1000D"></a><a name="Purpose"></a>
<h2 class="h3">Purpose</h2>
<div class="section">
<p>This document describes the Capacity Scheduler, a pluggable Map/Reduce scheduler for Hadoop which provides a way to share large clusters.</p>
</div>
    
    
<a name="N10017"></a><a name="Features"></a>
<h2 class="h3">Features</h2>
<div class="section">
<p>The Capacity Scheduler supports the following features:</p>
<ul>
        
<li>
          Support for multiple queues, where a job is submitted to a queue.
        </li>
        
<li>
          Queues are guaranteed a fraction of the capacity of the grid (their 
 	      'guaranteed capacity') in the sense that a certain capacity of 
 	      resources will be at their disposal. All jobs submitted to a 
 	      queue will have access to the capacity guaranteed to the queue.
        </li>
        
<li>
          Free resources can be allocated to any queue beyond its guaranteed 
          capacity. These excess allocated resources can be reclaimed and made 
          available to another queue in order to meet its capacity guarantee.
        </li>
        
<li>
          The scheduler guarantees that excess resources taken from a queue 
          will be restored to it within N minutes of its need for them.
        </li>
        
<li>
          Queues optionally support job priorities (disabled by default).
        </li>
        
<li>
          Within a queue, jobs with higher priority will have access to the 
          queue's resources before jobs with lower priority. However, once a 
          job is running, it will not be preempted for a higher priority job.
        </li>
        
<li>
          In order to prevent one or more users from monopolizing its 
          resources, each queue enforces a limit on the percentage of 
          resources allocated to a user at any given time, if there is 
          competition for them.  
        </li>
        
<li>
          Support for memory-intensive jobs, wherein a job can optionally 
          specify higher memory-requirements than the default, and the tasks 
          of the job will only be run on TaskTrackers that have enough memory 
          to spare.
        </li>
      
</ul>
</div>
    
    
<a name="N1003C"></a><a name="Picking+a+task+to+run"></a>
<h2 class="h3">Picking a task to run</h2>
<div class="section">
<p>Note that many of these steps can be, and will be, enhanced over time
      to provide better algorithms.</p>
<p>Whenever a TaskTracker is free, the Capacity Scheduler first picks a 
      queue that needs to reclaim any resources the earliest (this is a queue
      whose resources were temporarily being used by some other queue and now
      needs access to those resources). If no such queue is found, it then picks
      a queue which has most free space (whose ratio of # of running slots to 
      guaranteed capacity is the lowest).</p>
<p>Once a queue is selected, the scheduler picks a job in the queue. Jobs
      are sorted based on when they're submitted and their priorities (if the 
      queue supports priorities). Jobs are considered in order, and a job is 
      selected if its user is within the user-quota for the queue, i.e., the 
      user is not already using queue resources above his/her limit. The 
      scheduler also makes sure that there is enough free memory in the 
      TaskTracker to tun the job's task, in case the job has special memory
      requirements.</p>
<p>Once a job is selected, the scheduler picks a task to run. This logic 
      to pick a task remains unchanged from earlier versions.</p>
</div>
    
    
<a name="N1004F"></a><a name="Reclaiming+capacity"></a>
<h2 class="h3">Reclaiming capacity</h2>
<div class="section">
<p>Periodically, the scheduler determines:</p>
<ul>
	    
<li>
	      if a queue needs to reclaim capacity. This happens when a queue has
	      at least one task pending and part of its guaranteed capacity is 
	      being used by some other queue. If this happens, the scheduler notes
	      the amount of resources it needs to reclaim for this queue within a 
	      specified period of time (the reclaim time). 
	    </li>
	    
<li>
	      if a queue has not received all the resources it needed to reclaim,
	      and its reclaim time is about to expire. In this case, the scheduler
	      needs to kill tasks from queues running over capacity. This it does
	      by killing the tasks that started the latest.
	    </li>
	  
</ul>
</div>

    
<a name="N10062"></a><a name="Installation"></a>
<h2 class="h3">Installation</h2>
<div class="section">
<p>The capacity scheduler is available as a JAR file in the Hadoop
        tarball under the <em>contrib/capacity-scheduler</em> directory. The name of 
        the JAR file would be on the lines of hadoop-*-capacity-scheduler.jar.</p>
<p>You can also build the scheduler from source by executing
        <em>ant package</em>, in which case it would be available under
        <em>build/contrib/capacity-scheduler</em>.</p>
<p>To run the capacity scheduler in your Hadoop installation, you need 
        to put it on the <em>CLASSPATH</em>. The easiest way is to copy the 
        <span class="codefrag">hadoop-*-capacity-scheduler.jar</span> from 
        to <span class="codefrag">HADOOP_HOME/lib</span>. Alternatively, you can modify 
        <em>HADOOP_CLASSPATH</em> to include this jar, in 
        <span class="codefrag">conf/hadoop-env.sh</span>.</p>
</div>

    
<a name="N1008A"></a><a name="Configuration"></a>
<h2 class="h3">Configuration</h2>
<div class="section">
<a name="N10090"></a><a name="Using+the+capacity+scheduler"></a>
<h3 class="h4">Using the capacity scheduler</h3>
<p>
          To make the Hadoop framework use the capacity scheduler, set up
          the following property in the site configuration:</p>
<table class="ForrestTable" cellspacing="1" cellpadding="4">
            
<tr>
              
<td colspan="1" rowspan="1">Property</td>
              <td colspan="1" rowspan="1">Value</td>
            
</tr>
            
<tr>
              
<td colspan="1" rowspan="1">mapred.jobtracker.taskScheduler</td>
              <td colspan="1" rowspan="1">org.apache.hadoop.mapred.CapacityTaskScheduler</td>
            
</tr>
          
</table>
<a name="N100B7"></a><a name="Setting+up+queues"></a>
<h3 class="h4">Setting up queues</h3>
<p>
          You can define multiple queues to which users can submit jobs with
          the capacity scheduler. To define multiple queues, you should edit
          the site configuration for Hadoop and modify the
          <em>mapred.queue.names</em> property.
        </p>
<p>
          You can also configure ACLs for controlling which users or groups
          have access to the queues.
        </p>
<p>
          For more details, refer to
          <a href="cluster_setup.html#Configuring+the+Hadoop+Daemons">Cluster 
          Setup</a> documentation.
        </p>
<a name="N100CE"></a><a name="Configuring+properties+for+queues"></a>
<h3 class="h4">Configuring properties for queues</h3>
<p>The capacity scheduler can be configured with several properties
        for each queue that control the behavior of the scheduler. This
        configuration is in the <em>conf/capacity-scheduler.xml</em>. By
        default, the configuration is set up for one queue, named 
        <em>default</em>.</p>
<p>To specify a property for a queue that is defined in the site
        configuration, you should use the property name as
        <em>mapred.capacity-scheduler.queue.&lt;queue-name&gt;.&lt;property-name&gt;</em>.
        </p>
<p>For example, to define the property <em>guaranteed-capacity</em>
        for queue named <em>research</em>, you should specify the property
        name as 
        <em>mapred.capacity-scheduler.queue.research.guaranteed-capacity</em>.
        </p>
<p>The properties defined for queues and their descriptions are
        listed in the table below:</p>
<table class="ForrestTable" cellspacing="1" cellpadding="4">
          
<tr>
<th colspan="1" rowspan="1">Name</th><th colspan="1" rowspan="1">Description</th>
</tr>
          
<tr>
<td colspan="1" rowspan="1">mapred.capacity-scheduler.queue.&lt;queue-name&gt;.guaranteed-capacity</td>
          	<td colspan="1" rowspan="1">Percentage of the number of slots in the cluster that are
          	guaranteed to be available for jobs in this queue. 
          	The sum of guaranteed capacities for all queues should be less 
          	than or equal 100.</td>
          
</tr>
          
<tr>
<td colspan="1" rowspan="1">mapred.capacity-scheduler.queue.&lt;queue-name&gt;.reclaim-time-limit</td>
          	<td colspan="1" rowspan="1">The amount of time, in seconds, before which resources 
          	distributed to other queues will be reclaimed.</td>
          
</tr>
          
<tr>
<td colspan="1" rowspan="1">mapred.capacity-scheduler.queue.&lt;queue-name&gt;.supports-priority</td>
          	<td colspan="1" rowspan="1">If true, priorities of jobs will be taken into account in scheduling 
          	decisions.</td>
          
</tr>
          
<tr>
<td colspan="1" rowspan="1">mapred.capacity-scheduler.queue.&lt;queue-name&gt;.minimum-user-limit-percent</td>
          	<td colspan="1" rowspan="1">Each queue enforces a limit on the percentage of resources 
          	allocated to a user at any given time, if there is competition 
          	for them. This user limit can vary between a minimum and maximum 
          	value. The former depends on the number of users who have submitted
          	jobs, and the latter is set to this property value. For example, 
          	suppose the value of this property is 25. If two users have 
          	submitted jobs to a queue, no single user can use more than 50% 
          	of the queue resources. If a third user submits a job, no single 
          	user can use more than 33% of the queue resources. With 4 or more 
          	users, no user can use more than 25% of the queue's resources. A 
          	value of 100 implies no user limits are imposed.</td>
          
</tr>
        
</table>
<a name="N10130"></a><a name="Reviewing+the+configuration+of+the+capacity+scheduler"></a>
<h3 class="h4">Reviewing the configuration of the capacity scheduler</h3>
<p>
          Once the installation and configuration is completed, you can review
          it after starting the Map/Reduce cluster from the admin UI.
        </p>
<ul>
          
<li>Start the Map/Reduce cluster as usual.</li>
          
<li>Open the JobTracker web UI.</li>
          
<li>The queues you have configured should be listed under the <em>Scheduling
              Information</em> section of the page.</li>
          
<li>The properties for the queues should be visible in the <em>Scheduling
              Information</em> column against each queue.</li>
        
</ul>
</div>
  
</div>
<!--+
    |end content
    +-->
<div class="clearboth">&nbsp;</div>
</div>
<div id="footer">
<!--+
    |start bottomstrip
    +-->
<div class="lastmodified">
<script type="text/javascript"><!--
document.write("Last Published: " + document.lastModified);
//  --></script>
</div>
<div class="copyright">
        Copyright &copy;
         2008 <a href="http://www.apache.org/licenses/">The Apache Software Foundation.</a>
</div>
<!--+
    |end bottomstrip
    +-->
</div>
</body>
</html>