|
|
@@ -85,8 +85,9 @@ Hadoop MapReduce Next Generation - Fair Scheduler
|
|
|
cause too much intermediate data to be created or too much context-switching.
|
|
|
Limiting the apps does not cause any subsequently submitted apps to fail,
|
|
|
only to wait in the scheduler's queue until some of the user's earlier apps
|
|
|
- finish. apps to run from each user/queue are chosen in order of priority and
|
|
|
- then submit time, as in the default FIFO scheduler in Hadoop.
|
|
|
+ finish. Apps to run from each user/queue are chosen in the same fair sharing
|
|
|
+ manner, but can alternatively be configured to be chosen in order of submit
|
|
|
+ time, as in the default FIFO scheduler in Hadoop.
|
|
|
|
|
|
Certain add-ons are not yet supported which existed in the original (MR1)
|
|
|
Fair Scheduler. Among them, is the use of a custom policies governing
|
|
|
@@ -142,7 +143,9 @@ Hadoop MapReduce Next Generation - Fair Scheduler
|
|
|
* <<<yarn.scheduler.fair.sizebasedweight>>>
|
|
|
|
|
|
* Whether to assign shares to individual apps based on their size, rather than
|
|
|
- providing an equal share to all apps regardless of size. Defaults to false.
|
|
|
+ providing an equal share to all apps regardless of size. When set to true,
|
|
|
+ apps are weighted by the natural logarithm of one plus the app's total
|
|
|
+ requested memory, divided by the natural logarithm of 2. Defaults to false.
|
|
|
|
|
|
* <<<yarn.scheduler.fair.assignmultiple>>>
|
|
|
|
|
|
@@ -180,16 +183,29 @@ Allocation file format
|
|
|
* <<Queue elements>>, which represent queues. Each may contain the following
|
|
|
properties:
|
|
|
|
|
|
- * minResources: minimum amount of aggregate memory
|
|
|
+ * minResources: minimum MB of aggregate memory the queue expects. If a queue
|
|
|
+ demands resources, and its current allocation is below its configured minimum,
|
|
|
+ it will be assigned available resources before any queue that is not in this
|
|
|
+ situation. If multiple queues are in this situation, resources go to the
|
|
|
+ queue with the smallest ratio between allocation and minimum. Note that it is
|
|
|
+ possible that a queue that is below its minimum may not immediately get up to
|
|
|
+ its minimum when it submits an application, because already-running jobs may
|
|
|
+ be using those resources.
|
|
|
|
|
|
- * maxResources: maximum amount of aggregate memory
|
|
|
+ * maxResources: maximum MB of aggregate memory a queue is allowed. A queue
|
|
|
+ will never be assigned a container that would put it over this limit.
|
|
|
|
|
|
* maxRunningApps: limit the number of apps from the queue to run at once
|
|
|
|
|
|
- * weight: to share the cluster non-proportionally with other queues
|
|
|
+ * weight: to share the cluster non-proportionally with other queues. Weights
|
|
|
+ default to 1, and a queue with weight 2 should receive approximately twice
|
|
|
+ as many resources as a queue with the default weight.
|
|
|
|
|
|
* schedulingMode: either "fifo" or "fair" depending on the in-queue scheduling
|
|
|
- policy desired
|
|
|
+ policy desired. Defaults to "fair". If "fifo", apps with earlier submit
|
|
|
+ times are given preference for containers, but apps submitted later may
|
|
|
+ run concurrently if there is leftover space on the cluster after satisfying
|
|
|
+ the earlier app's requests.
|
|
|
|
|
|
* aclSubmitApps: a list of users that can submit apps to the queue. A (default)
|
|
|
value of "*" means that any users can submit apps. A queue inherits the ACL of
|
Thomas White