MapReduce History Server REST API's.
====================================
Overview
--------
The history server REST API's allow the user to get status on finished applications.
History Server Information API
------------------------------
The history server information resource provides overall information about the history server.
### URI
Both of the following URI's give you the history server information, from an application id identified by the appid value.
* http://history-server-http-address:port/ws/v1/history
* http://history-server-http-address:port/ws/v1/history/info
### HTTP Operations Supported
* GET
### Query Parameters Supported
None
### Elements of the *historyInfo* object
| Item | Data Type | Description |
|:---- |:---- |:---- |
| startedOn | long | The time the history server was started (in ms since epoch) |
| hadoopVersion | string | Version of hadoop common |
| hadoopBuildVersion | string | Hadoop common build string with build version, user, and checksum |
| hadoopVersionBuiltOn | string | Timestamp when hadoop common was built |
### Response Examples
**JSON response**
HTTP Request:
GET http://history-server-http-address:port/ws/v1/history/info
Response Header:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
Response Body:
{
"historyInfo" : {
"startedOn":1353512830963,
"hadoopVersionBuiltOn" : "Wed Jan 11 21:18:36 UTC 2012",
"hadoopBuildVersion" : "0.23.1-SNAPSHOT from 1230253 by user1 source checksum bb6e554c6d50b0397d826081017437a7",
"hadoopVersion" : "0.23.1-SNAPSHOT"
}
}
**XML response**
HTTP Request:
GET http://history-server-http-address:port/ws/v1/history/info
Accept: application/xml
Response Header:
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 330
Server: Jetty(6.1.26)
Response Body:
13535128309630.23.1-SNAPSHOT0.23.1-SNAPSHOT from 1230253 by user1 source checksum bb6e554c6d50b0397d826081017437a7Wed Jan 11 21:18:36 UTC 2012
MapReduce API's
---------------
The following list of resources apply to MapReduce.
### Jobs API
The jobs resource provides a list of the MapReduce jobs that have finished. It does not currently return a full list of parameters
#### URI
* http://history-server-http-address:port/ws/v1/history/mapreduce/jobs
#### HTTP Operations Supported
* GET
#### Query Parameters Supported
Multiple parameters can be specified. The started and finished times have a begin and end parameter to allow you to specify ranges. For example, one could request all jobs that started between 1:00am and 2:00pm on 12/19/2011 with startedTimeBegin=1324256400&startedTimeEnd=1324303200. If the Begin parameter is not specfied, it defaults to 0, and if the End parameter is not specified, it defaults to infinity.
* user - user name
* state - the job state
* queue - queue name
* limit - total number of app objects to be returned
* startedTimeBegin - jobs with start time beginning with this time, specified in ms since epoch
* startedTimeEnd - jobs with start time ending with this time, specified in ms since epoch
* finishedTimeBegin - jobs with finish time beginning with this time, specified in ms since epoch
* finishedTimeEnd - jobs with finish time ending with this time, specified in ms since epoch
#### Elements of the *jobs* object
When you make a request for the list of jobs, the information will be returned as an array of job objects. See also
[Job API](#Job_API)
for syntax of the job object. Except this is a subset of a full job. Only startTime, finishTime, id, name, queue, user, state, mapsTotal, mapsCompleted, reducesTotal, and reducesCompleted are returned.
| Item | Data Type | Description |
|:---- |:---- |:---- |
| job | array of job objects(json)/zero or more job objects(XML) | The collection of job objects |
#### Response Examples
**JSON response**
HTTP Request:
GET http://history-server-http-address:port/ws/v1/history/mapreduce/jobs
Response Header:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
Response Body:
{
"jobs" : {
"job" : [
{
"submitTime" : 1326381344449,
"state" : "SUCCEEDED",
"user" : "user1",
"reducesTotal" : 1,
"mapsCompleted" : 1,
"startTime" : 1326381344489,
"id" : "job_1326381300833_1_1",
"name" : "word count",
"reducesCompleted" : 1,
"mapsTotal" : 1,
"queue" : "default",
"finishTime" : 1326381356010
},
{
"submitTime" : 1326381446500
"state" : "SUCCEEDED",
"user" : "user1",
"reducesTotal" : 1,
"mapsCompleted" : 1,
"startTime" : 1326381446529,
"id" : "job_1326381300833_2_2",
"name" : "Sleep job",
"reducesCompleted" : 1,
"mapsTotal" : 1,
"queue" : "default",
"finishTime" : 1326381582106
}
]
}
}
**XML response**
HTTP Request:
GET http://history-server-http-address:port/ws/v1/history/mapreduce/jobs
Accept: application/xml
Response Header:
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 1922
Server: Jetty(6.1.26)
Response Body:
132638134444913263813444891326381356010job_1326381300833_1_1word countdefaultuser1SUCCEEDED1111132638144650013263814465291326381582106job_1326381300833_2_2Sleep jobdefaultuser1SUCCEEDED1111
### Job API
A Job resource contains information about a particular job identified by jobid.
#### URI
* http://history-server-http-address:port/ws/v1/history/mapreduce/jobs/{jobid}
#### HTTP Operations Supported
* GET
#### Query Parameters Supported
None
#### Elements of the *job* object
| Item | Data Type | Description |
|:---- |:---- |:---- |
| id | string | The job id |
| name | string | The job name |
| queue | string | The queue the job was submitted to |
| user | string | The user name |
| state | string | the job state - valid values are: NEW, INITED, RUNNING, SUCCEEDED, FAILED, KILL\_WAIT, KILLED, ERROR |
| diagnostics | string | A diagnostic message |
| submitTime | long | The time the job submitted (in ms since epoch) |
| startTime | long | The time the job started (in ms since epoch) |
| finishTime | long | The time the job finished (in ms since epoch) |
| mapsTotal | int | The total number of maps |
| mapsCompleted | int | The number of completed maps |
| reducesTotal | int | The total number of reduces |
| reducesCompleted | int | The number of completed reduces |
| uberized | boolean | Indicates if the job was an uber job - ran completely in the application master |
| avgMapTime | long | The average time of a map task (in ms) |
| avgReduceTime | long | The average time of the reduce (in ms) |
| avgShuffleTime | long | The average time of the shuffle (in ms) |
| avgMergeTime | long | The average time of the merge (in ms) |
| failedReduceAttempts | int | The number of failed reduce attempts |
| killedReduceAttempts | int | The number of killed reduce attempts |
| successfulReduceAttempts | int | The number of successful reduce attempts |
| failedMapAttempts | int | The number of failed map attempts |
| killedMapAttempts | int | The number of killed map attempts |
| successfulMapAttempts | int | The number of successful map attempts |
| acls | array of acls(json)/zero or more acls objects(xml) | A collection of acls objects |
### Elements of the *acls* object
| Item | Data Type | Description |
|:---- |:---- |:---- |
| value | string | The acl value |
| name | string | The acl name |
#### Response Examples
**JSON response**
HTTP Request:
GET http://history-server-http-address:port/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2
Response Header:
HTTP/1.1 200 OK
Content-Type: application/json
Server: Jetty(6.1.26)
Content-Length: 720
Response Body:
{
"job" : {
"submitTime": 1326381446500,
"avgReduceTime" : 124961,
"failedReduceAttempts" : 0,
"state" : "SUCCEEDED",
"successfulReduceAttempts" : 1,
"acls" : [
{
"value" : " ",
"name" : "mapreduce.job.acl-modify-job"
},
{
"value" : " ",
"name" : "mapreduce.job.acl-view-job"
}
],
"user" : "user1",
"reducesTotal" : 1,
"mapsCompleted" : 1,
"startTime" : 1326381446529,
"id" : "job_1326381300833_2_2",
"avgMapTime" : 2638,
"successfulMapAttempts" : 1,
"name" : "Sleep job",
"avgShuffleTime" : 2540,
"reducesCompleted" : 1,
"diagnostics" : "",
"failedMapAttempts" : 0,
"avgMergeTime" : 2589,
"killedReduceAttempts" : 0,
"mapsTotal" : 1,
"queue" : "default",
"uberized" : false,
"killedMapAttempts" : 0,
"finishTime" : 1326381582106
}
}
**XML response**
HTTP Request:
GET http://history-server-http-address:port/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2
Accept: application/xml
Response Header:
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 983
Server: Jetty(6.1.26)
Response Body:
132638144650013263814465291326381582106job_1326381300833_2_2Sleep jobdefaultuser1SUCCEEDED1111false263812496125402589001001mapreduce.job.acl-modify-jobmapreduce.job.acl-view-job
### Job Attempts API
With the job attempts API, you can obtain a collection of resources that represent a job attempt. When you run a GET operation on this resource, you obtain a collection of Job Attempt Objects.
#### URI
* http://history-server-http-address:port/ws/v1/history/mapreduce/jobs/{jobid}/jobattempts
#### HTTP Operations Supported
* GET
#### Query Parameters Supported
None
#### Elements of the *jobAttempts* object
When you make a request for the list of job attempts, the information will be returned as an array of job attempt objects.
jobAttempts:
| Item | Data Type | Description |
|:---- |:---- |:---- |
| jobAttempt | array of job attempt objects(JSON)/zero or more job attempt objects(XML) | The collection of job attempt objects |
#### Elements of the *jobAttempt* object
| Item | Data Type | Description |
|:---- |:---- |:---- |
| id | int | The job attempt id |
| nodeId | string | The node id of the node the attempt ran on |
| nodeHttpAddress | string | The node http address of the node the attempt ran on |
| logsLink | string | The http link to the job attempt logs |
| containerId | string | The id of the container for the job attempt |
| startTime | long | The start time of the attempt (in ms since epoch) |
#### Response Examples
**JSON response**
HTTP Request:
GET http://history-server-http-address:port/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/jobattempts
Response Header:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
Response Body:
{
"jobAttempts" : {
"jobAttempt" : [
{
"nodeId" : "host.domain.com:8041",
"nodeHttpAddress" : "host.domain.com:8042",
"startTime" : 1326381444693,
"id" : 1,
"logsLink" : "http://host.domain.com:19888/jobhistory/logs/host.domain.com:8041/container_1326381300833_0002_01_000001/job_1326381300833_2_2/user1",
"containerId" : "container_1326381300833_0002_01_000001"
}
]
}
}
**XML response**
HTTP Request:
GET http://history-server-http-address:port/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/jobattmpts
Accept: application/xml
Response Header:
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 575
Server: Jetty(6.1.26)
Response Body:
host.domain.com:8042host.domain.com:804111326381444693container_1326381300833_0002_01_000001http://host.domain.com:19888/jobhistory/logs/host.domain.com:8041/container_1326381300833_0002_01_000001/job_1326381300833_2_2/user1
### Job Counters API
With the job counters API, you can object a collection of resources that represent al the counters for that job.
#### URI
* http://history-server-http-address:port/ws/v1/history/mapreduce/jobs/{jobid}/counters
#### HTTP Operations Supported
* GET
#### Query Parameters Supported
None
#### Elements of the *jobCounters* object
| Item | Data Type | Description |
|:---- |:---- |:---- |
| id | string | The job id |
| counterGroup | array of counterGroup objects(JSON)/zero or more counterGroup objects(XML) | A collection of counter group objects |
#### Elements of the *counterGroup* objecs
| Item | Data Type | Description |
|:---- |:---- |:---- |
| counterGroupName | string | The name of the counter group |
| counter | array of counter objects(JSON)/zero or more counter objects(XML) | A collection of counter objects |
#### Elements of the *counter* object
| Item | Data Type | Description |
|:---- |:---- |:---- |
| name | string | The name of the counter |
| reduceCounterValue | long | The counter value of reduce tasks |
| mapCounterValue | long | The counter value of map tasks |
| totalCounterValue | long | The counter value of all tasks |
#### Response Examples
**JSON response**
HTTP Request:
GET http://history-server-http-address:port/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/counters
Response Header:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
Response Body:
{
"jobCounters" : {
"id" : "job_1326381300833_2_2",
"counterGroup" : [
{
"counterGroupName" : "Shuffle Errors",
"counter" : [
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "BAD_ID"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "CONNECTION"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "IO_ERROR"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "WRONG_LENGTH"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "WRONG_MAP"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "WRONG_REDUCE"
}
]
},
{
"counterGroupName" : "org.apache.hadoop.mapreduce.FileSystemCounter",
"counter" : [
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 2483,
"name" : "FILE_BYTES_READ"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 108525,
"name" : "FILE_BYTES_WRITTEN"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "FILE_READ_OPS"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "FILE_LARGE_READ_OPS"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "FILE_WRITE_OPS"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 48,
"name" : "HDFS_BYTES_READ"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "HDFS_BYTES_WRITTEN"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 1,
"name" : "HDFS_READ_OPS"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "HDFS_LARGE_READ_OPS"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "HDFS_WRITE_OPS"
}
]
},
{
"counterGroupName" : "org.apache.hadoop.mapreduce.TaskCounter",
"counter" : [
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 1,
"name" : "MAP_INPUT_RECORDS"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 1200,
"name" : "MAP_OUTPUT_RECORDS"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 4800,
"name" : "MAP_OUTPUT_BYTES"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 2235,
"name" : "MAP_OUTPUT_MATERIALIZED_BYTES"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 48,
"name" : "SPLIT_RAW_BYTES"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "COMBINE_INPUT_RECORDS"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "COMBINE_OUTPUT_RECORDS"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 1200,
"name" : "REDUCE_INPUT_GROUPS"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 2235,
"name" : "REDUCE_SHUFFLE_BYTES"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 1200,
"name" : "REDUCE_INPUT_RECORDS"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "REDUCE_OUTPUT_RECORDS"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 2400,
"name" : "SPILLED_RECORDS"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 1,
"name" : "SHUFFLED_MAPS"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "FAILED_SHUFFLE"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 1,
"name" : "MERGED_MAP_OUTPUTS"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 113,
"name" : "GC_TIME_MILLIS"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 1830,
"name" : "CPU_MILLISECONDS"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 478068736,
"name" : "PHYSICAL_MEMORY_BYTES"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 2159284224,
"name" : "VIRTUAL_MEMORY_BYTES"
},
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 378863616,
"name" : "COMMITTED_HEAP_BYTES"
}
]
},
{
"counterGroupName" : "org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter",
"counter" : [
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "BYTES_READ"
}
]
},
{
"counterGroupName" : "org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounter",
"counter" : [
{
"reduceCounterValue" : 0,
"mapCounterValue" : 0,
"totalCounterValue" : 0,
"name" : "BYTES_WRITTEN"
}
]
}
]
}
}
**XML response**
HTTP Request:
GET http://history-server-http-address:port/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/counters
Accept: application/xml
Response Header:
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 7030
Server: Jetty(6.1.26)
Response Body:
job_1326381300833_2_2Shuffle ErrorsBAD_ID000CONNECTION000IO_ERROR000WRONG_LENGTH000WRONG_MAP000WRONG_REDUCE000org.apache.hadoop.mapreduce.FileSystemCounterFILE_BYTES_READ248300FILE_BYTES_WRITTEN10852500FILE_READ_OPS000FILE_LARGE_READ_OPS000FILE_WRITE_OPS000HDFS_BYTES_READ4800HDFS_BYTES_WRITTEN000HDFS_READ_OPS100HDFS_LARGE_READ_OPS000HDFS_WRITE_OPS000org.apache.hadoop.mapreduce.TaskCounterMAP_INPUT_RECORDS100MAP_OUTPUT_RECORDS120000MAP_OUTPUT_BYTES480000MAP_OUTPUT_MATERIALIZED_BYTES223500SPLIT_RAW_BYTES4800COMBINE_INPUT_RECORDS000COMBINE_OUTPUT_RECORDS000REDUCE_INPUT_GROUPS120000REDUCE_SHUFFLE_BYTES223500REDUCE_INPUT_RECORDS120000REDUCE_OUTPUT_RECORDS000SPILLED_RECORDS240000SHUFFLED_MAPS100FAILED_SHUFFLE000MERGED_MAP_OUTPUTS100GC_TIME_MILLIS11300CPU_MILLISECONDS183000PHYSICAL_MEMORY_BYTES47806873600VIRTUAL_MEMORY_BYTES215928422400COMMITTED_HEAP_BYTES37886361600org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounterBYTES_READ000org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounterBYTES_WRITTEN000
### Job Conf API
A job configuration resource contains information about the job configuration for this job.
#### URI
Use the following URI to obtain th job configuration information, from a job identified by the jobid value.
* http://history-server-http-address:port/ws/v1/history/mapreduce/jobs/{jobid}/conf
#### HTTP Operations Supported
* GET
#### Query Parameters Supported
None
#### Elements of the *conf* object
| Item | Data Type | Description |
|:---- |:---- |:---- |
| path | string | The path to the job configuration file |
| property | array of the configuration properties(JSON)/zero or more configuration properties(XML) | Collection of configuration property objects |
#### Elements of the *property* object
| Item | Data Type | Description |
|:---- |:---- |:---- |
| name | string | The name of the configuration property |
| value | string | The value of the configuration property |
| source | string | The location this configuration object came from. If there is more then one of these it shows the history with the latest source at the end of the list. |
#### Response Examples
**JSON response**
HTTP Request:
GET http://history-server-http-address:port/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/conf
Response Header:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
Response Body:
This is a small snippet of the output as the output if very large. The real output contains every property in your job configuration file.
{
"conf" : {
"path" : "hdfs://host.domain.com:9000/user/user1/.staging/job_1326381300833_0002/job.xml",
"property" : [
{
"value" : "/home/hadoop/hdfs/data",
"name" : "dfs.datanode.data.dir"
"source" : ["hdfs-site.xml", "job.xml"]
},
{
"value" : "org.apache.hadoop.yarn.server.webproxy.amfilter.AmFilterInitializer",
"name" : "hadoop.http.filter.initializers"
"source" : ["programmatically", "job.xml"]
},
{
"value" : "/home/hadoop/tmp",
"name" : "mapreduce.cluster.temp.dir"
"source" : ["mapred-site.xml"]
},
...
]
}
}
**XML response**
HTTP Request:
GET http://history-server-http-address:port/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/conf
Accept: application/xml
Response Header:
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 552
Server: Jetty(6.1.26)
Response Body:
hdfs://host.domain.com:9000/user/user1/.staging/job_1326381300833_0002/job.xmldfs.datanode.data.dir/home/hadoop/hdfs/datahdfs-site.xmljob.xmlhadoop.http.filter.initializersorg.apache.hadoop.yarn.server.webproxy.amfilter.AmFilterInitializerprogrammaticallyjob.xmlmapreduce.cluster.temp.dir/home/hadoop/tmpmapred-site.xml
...
### Tasks API
With the tasks API, you can obtain a collection of resources that represent a task within a job. When you run a GET operation on this resource, you obtain a collection of Task Objects.
#### URI
* http://history-server-http-address:port/ws/v1/history/mapreduce/jobs/{jobid}/tasks
#### HTTP Operations Supported
* GET
#### Query Parameters Supported
* type - type of task, valid values are m or r. m for map task or r for reduce task.
#### Elements of the *tasks* object
When you make a request for the list of tasks , the information will be returned as an array of task objects. See also
[Task API](#Task_API)
for syntax of the task object.
| Item | Data Type | Description |
|:---- |:---- |:---- |
| task | array of task objects(JSON)/zero or more task objects(XML) | The collection of task objects. |
#### Response Examples
**JSON response**
HTTP Request:
GET http://history-server-http-address:port/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/tasks
Response Header:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
Response Body:
{
"tasks" : {
"task" : [
{
"progress" : 100,
"elapsedTime" : 6777,
"state" : "SUCCEEDED",
"startTime" : 1326381446541,
"id" : "task_1326381300833_2_2_m_0",
"type" : "MAP",
"successfulAttempt" : "attempt_1326381300833_2_2_m_0_0",
"finishTime" : 1326381453318
},
{
"progress" : 100,
"elapsedTime" : 135559,
"state" : "SUCCEEDED",
"startTime" : 1326381446544,
"id" : "task_1326381300833_2_2_r_0",
"type" : "REDUCE",
"successfulAttempt" : "attempt_1326381300833_2_2_r_0_0",
"finishTime" : 1326381582103
}
]
}
}
**XML response**
HTTP Request:
GET http://history-server-http-address:port/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/tasks
Accept: application/xml
Response Header:
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 653
Server: Jetty(6.1.26)
Response Body:
132638144654113263814533186777task_1326381300833_2_2_m_0SUCCEEDEDMAPattempt_1326381300833_2_2_m_0_013263814465441326381582103135559task_1326381300833_2_2_r_0SUCCEEDEDREDUCEattempt_1326381300833_2_2_r_0_0
### Task API
A Task resource contains information about a particular task within a job.
#### URI
Use the following URI to obtain an Task Object, from a task identified by the taskid value.
* http://history-server-http-address:port/ws/v1/history/mapreduce/jobs/{jobid}/tasks/{taskid}
#### HTTP Operations Supported
* GET
#### Query Parameters Supported
None
#### Elements of the *task* object
| Item | Data Type | Description |
|:---- |:---- |:---- |
| id | string | The task id |
| state | string | The state of the task - valid values are: NEW, SCHEDULED, RUNNING, SUCCEEDED, FAILED, KILL\_WAIT, KILLED |
| type | string | The task type - MAP or REDUCE |
| successfulAttempt | string | The id of the last successful attempt |
| progress | float | The progress of the task as a percent |
| startTime | long | The time in which the task started (in ms since epoch) or -1 if it was never started |
| finishTime | long | The time in which the task finished (in ms since epoch) |
| elapsedTime | long | The elapsed time since the application started (in ms) |
#### Response Examples
**JSON response**
HTTP Request:
GET http://history-server-http-address:port/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/tasks/task_1326381300833_2_2_m_0
Response Header:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
Response Body:
{
"task" : {
"progress" : 100,
"elapsedTime" : 6777,
"state" : "SUCCEEDED",
"startTime" : 1326381446541,
"id" : "task_1326381300833_2_2_m_0",
"type" : "MAP",
"successfulAttempt" : "attempt_1326381300833_2_2_m_0_0",
"finishTime" : 1326381453318
}
}
**XML response**
HTTP Request:
GET http://history-server-http-address:port/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/tasks/task_1326381300833_2_2_m_0
Accept: application/xml
Response Header:
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 299
Server: Jetty(6.1.26)
Response Body:
132638144654113263814533186777task_1326381300833_2_2_m_0SUCCEEDEDMAPattempt_1326381300833_2_2_m_0_0
### Task Counters API
With the task counters API, you can object a collection of resources that represent all the counters for that task.
#### URI
* http://history-server-http-address:port/ws/v1/history/mapreduce/jobs/{jobid}/tasks/{taskid}/counters
#### HTTP Operations Supported
* GET
#### Query Parameters Supported
None
#### Elements of the *jobTaskCounters* object
| Item | Data Type | Description |
|:---- |:---- |:---- |
| id | string | The task id |
| taskcounterGroup | array of counterGroup objects(JSON)/zero or more counterGroup objects(XML) | A collection of counter group objects |
#### Elements of the *counterGroup* object
| Item | Data Type | Description |
|:---- |:---- |:---- |
| counterGroupName | string | The name of the counter group |
| counter | array of counter objects(JSON)/zero or more counter objects(XML) | A collection of counter objects |
#### Elements of the *counter* object
| Item | Data Type | Description |
|:---- |:---- |:---- |
| name | string | The name of the counter |
| value | long | The value of the counter |
#### Response Examples
**JSON response**
HTTP Request:
GET http://history-server-http-address:port/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/tasks/task_1326381300833_2_2_m_0/counters
Response Header:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
Response Body:
{
"jobTaskCounters" : {
"id" : "task_1326381300833_2_2_m_0",
"taskCounterGroup" : [
{
"counterGroupName" : "org.apache.hadoop.mapreduce.FileSystemCounter",
"counter" : [
{
"value" : 2363,
"name" : "FILE_BYTES_READ"
},
{
"value" : 54372,
"name" : "FILE_BYTES_WRITTEN"
},
{
"value" : 0,
"name" : "FILE_READ_OPS"
},
{
"value" : 0,
"name" : "FILE_LARGE_READ_OPS"
},
{
"value" : 0,
"name" : "FILE_WRITE_OPS"
},
{
"value" : 0,
"name" : "HDFS_BYTES_READ"
},
{
"value" : 0,
"name" : "HDFS_BYTES_WRITTEN"
},
{
"value" : 0,
"name" : "HDFS_READ_OPS"
},
{
"value" : 0,
"name" : "HDFS_LARGE_READ_OPS"
},
{
"value" : 0,
"name" : "HDFS_WRITE_OPS"
}
]
},
{
"counterGroupName" : "org.apache.hadoop.mapreduce.TaskCounter",
"counter" : [
{
"value" : 0,
"name" : "COMBINE_INPUT_RECORDS"
},
{
"value" : 0,
"name" : "COMBINE_OUTPUT_RECORDS"
},
{
"value" : 460,
"name" : "REDUCE_INPUT_GROUPS"
},
{
"value" : 2235,
"name" : "REDUCE_SHUFFLE_BYTES"
},
{
"value" : 460,
"name" : "REDUCE_INPUT_RECORDS"
},
{
"value" : 0,
"name" : "REDUCE_OUTPUT_RECORDS"
},
{
"value" : 0,
"name" : "SPILLED_RECORDS"
},
{
"value" : 1,
"name" : "SHUFFLED_MAPS"
},
{
"value" : 0,
"name" : "FAILED_SHUFFLE"
},
{
"value" : 1,
"name" : "MERGED_MAP_OUTPUTS"
},
{
"value" : 26,
"name" : "GC_TIME_MILLIS"
},
{
"value" : 860,
"name" : "CPU_MILLISECONDS"
},
{
"value" : 107839488,
"name" : "PHYSICAL_MEMORY_BYTES"
},
{
"value" : 1123147776,
"name" : "VIRTUAL_MEMORY_BYTES"
},
{
"value" : 57475072,
"name" : "COMMITTED_HEAP_BYTES"
}
]
},
{
"counterGroupName" : "Shuffle Errors",
"counter" : [
{
"value" : 0,
"name" : "BAD_ID"
},
{
"value" : 0,
"name" : "CONNECTION"
},
{
"value" : 0,
"name" : "IO_ERROR"
},
{
"value" : 0,
"name" : "WRONG_LENGTH"
},
{
"value" : 0,
"name" : "WRONG_MAP"
},
{
"value" : 0,
"name" : "WRONG_REDUCE"
}
]
},
{
"counterGroupName" : "org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounter",
"counter" : [
{
"value" : 0,
"name" : "BYTES_WRITTEN"
}
]
}
]
}
}
**XML response**
HTTP Request:
GET http://history-server-http-address:port/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/tasks/task_1326381300833_2_2_m_0/counters
Accept: application/xml
Response Header:
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 2660
Server: Jetty(6.1.26)
Response Body:
task_1326381300833_2_2_m_0org.apache.hadoop.mapreduce.FileSystemCounterFILE_BYTES_READ2363FILE_BYTES_WRITTEN54372FILE_READ_OPS0FILE_LARGE_READ_OPS0FILE_WRITE_OPS0HDFS_BYTES_READ0HDFS_BYTES_WRITTEN0HDFS_READ_OPS0HDFS_LARGE_READ_OPS0HDFS_WRITE_OPS0org.apache.hadoop.mapreduce.TaskCounterCOMBINE_INPUT_RECORDS0COMBINE_OUTPUT_RECORDS0REDUCE_INPUT_GROUPS460REDUCE_SHUFFLE_BYTES2235REDUCE_INPUT_RECORDS460REDUCE_OUTPUT_RECORDS0SPILLED_RECORDS0SHUFFLED_MAPS1FAILED_SHUFFLE0MERGED_MAP_OUTPUTS1GC_TIME_MILLIS26CPU_MILLISECONDS860PHYSICAL_MEMORY_BYTES107839488VIRTUAL_MEMORY_BYTES1123147776COMMITTED_HEAP_BYTES57475072Shuffle ErrorsBAD_ID0CONNECTION0IO_ERROR0WRONG_LENGTH0WRONG_MAP0WRONG_REDUCE0org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounterBYTES_WRITTEN0
### Task Attempts API
With the task attempts API, you can obtain a collection of resources that represent a task attempt within a job. When you run a GET operation on this resource, you obtain a collection of Task Attempt Objects.
#### URI
* http://history-server-http-address:port/ws/v1/history/mapreduce/jobs/{jobid}/tasks/{taskid}/attempts
#### HTTP Operations Supported
* GET
#### Query Parameters Supported
None
#### Elements of the *taskAttempts* object
When you make a request for the list of task attempts, the information will be returned as an array of task attempt objects. See also
[Task Attempt API](#Task_Attempt_API)
for syntax of the task object.
| Item | Data Type | Description |
|:---- |:---- |:---- |
| taskAttempt | array of task attempt objects(JSON)/zero or more task attempt objects(XML) | The collection of task attempt objects |
#### Response Examples
**JSON response**
HTTP Request:
GET http://history-server-http-address:port/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/tasks/task_1326381300833_2_2_m_0/attempts
Response Header:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
Response Body:
{
"taskAttempts" : {
"taskAttempt" : [
{
"assignedContainerId" : "container_1326381300833_0002_01_000002",
"progress" : 100,
"elapsedTime" : 2638,
"state" : "SUCCEEDED",
"diagnostics" : "",
"rack" : "/98.139.92.0",
"nodeHttpAddress" : "host.domain.com:8042",
"startTime" : 1326381450680,
"id" : "attempt_1326381300833_2_2_m_0_0",
"type" : "MAP",
"finishTime" : 1326381453318
}
]
}
}
**XML response**
HTTP Request:
GET http://history-server-http-address:port/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/tasks/task_1326381300833_2_2_m_0/attempts
Accept: application/xml
Response Header:
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 537
Server: Jetty(6.1.26)
Response Body:
132638145068013263814533182638attempt_1326381300833_2_2_m_0_0/98.139.92.0SUCCEEDEDhost.domain.com:8042MAPcontainer_1326381300833_0002_01_000002
### Task Attempt API
A Task Attempt resource contains information about a particular task attempt within a job.
#### URI
Use the following URI to obtain an Task Attempt Object, from a task identified by the attemptid value.
* http://history-server-http-address:port/ws/v1/history/mapreduce/jobs/{jobid}/tasks/{taskid}/attempts/{attemptid}
#### HTTP Operations Supported
* GET
#### Query Parameters Supported
None
#### Elements of the *taskAttempt* object
| Item | Data Type | Description |
|:---- |:---- |:---- |
| id | string | The task id |
| rack | string | The rack |
| state | string | The state of the task attempt - valid values are: NEW, UNASSIGNED, ASSIGNED, RUNNING, COMMIT\_PENDING, SUCCESS\_CONTAINER\_CLEANUP, SUCCEEDED, FAIL\_CONTAINER\_CLEANUP, FAIL\_TASK\_CLEANUP, FAILED, KILL\_CONTAINER\_CLEANUP, KILL\_TASK\_CLEANUP, KILLED |
| type | string | The type of task |
| assignedContainerId | string | The container id this attempt is assigned to |
| nodeHttpAddress | string | The http address of the node this task attempt ran on |
| diagnostics | string | A diagnostics message |
| progress | float | The progress of the task attempt as a percent |
| startTime | long | The time in which the task attempt started (in ms since epoch) |
| finishTime | long | The time in which the task attempt finished (in ms since epoch) |
| elapsedTime | long | The elapsed time since the task attempt started (in ms) |
For reduce task attempts you also have the following fields:
| Item | Data Type | Description |
|:---- |:---- |:---- |
| shuffleFinishTime | long | The time at which shuffle finished (in ms since epoch) |
| mergeFinishTime | long | The time at which merge finished (in ms since epoch) |
| elapsedShuffleTime | long | The time it took for the shuffle phase to complete (time in ms between reduce task start and shuffle finish) |
| elapsedMergeTime | long | The time it took for the merge phase to complete (time in ms between the shuffle finish and merge finish) |
| elapsedReduceTime | long | The time it took for the reduce phase to complete (time in ms between merge finish to end of reduce task) |
#### Response Examples
**JSON response**
HTTP Request:
GET http://history-server-http-address:port/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/tasks/task_1326381300833_2_2_m_0/attempts/attempt_1326381300833_2_2_m_0_0
Response Header:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
Response Body:
{
"taskAttempt" : {
"assignedContainerId" : "container_1326381300833_0002_01_000002",
"progress" : 100,
"elapsedTime" : 2638,
"state" : "SUCCEEDED",
"diagnostics" : "",
"rack" : "/98.139.92.0",
"nodeHttpAddress" : "host.domain.com:8042",
"startTime" : 1326381450680,
"id" : "attempt_1326381300833_2_2_m_0_0",
"type" : "MAP",
"finishTime" : 1326381453318
}
}
**XML response**
HTTP Request:
GET http://history-server-http-address:port/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/tasks/task_1326381300833_2_2_m_0/attempts/attempt_1326381300833_2_2_m_0_0
Accept: application/xml
Response Header:
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 691
Server: Jetty(6.1.26)
Response Body:
132638145068013263814533182638attempt_1326381300833_2_2_m_0_0/98.139.92.0SUCCEEDEDhost.domain.com:8042MAPcontainer_1326381300833_0002_01_000002
### Task Attempt Counters API
With the task attempt counters API, you can object a collection of resources that represent al the counters for that task attempt.
#### URI
* http://history-server-http-address:port/ws/v1/history/mapreduce/jobs/{jobid}/tasks/{taskid}/attempts/{attemptid}/counters
#### HTTP Operations Supported
* GET
#### Query Parameters Supported
None
#### Elements of the *jobTaskAttemptCounters* object
| Item | Data Type | Description |
|:---- |:---- |:---- |
| id | string | The task attempt id |
| taskAttemptcounterGroup | array of task attempt counterGroup objects(JSON)/zero or more task attempt counterGroup objects(XML) | A collection of task attempt counter group objects |
#### Elements of the *taskAttemptCounterGroup* object
| Item | Data Type | Description |
|:---- |:---- |:---- |
| counterGroupName | string | The name of the counter group |
| counter | array of counter objects(JSON)/zero or more counter objects(XML) | A collection of counter objects |
#### Elements of the *counter* object
| Item | Data Type | Description |
|:---- |:---- |:---- |
| name | string | The name of the counter |
| value | long | The value of the counter |
#### Response Examples
**JSON response**
HTTP Request:
GET http://history-server-http-address:port/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/tasks/task_1326381300833_2_2_m_0/attempts/attempt_1326381300833_2_2_m_0_0/counters
Response Header:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
Server: Jetty(6.1.26)
Response Body:
{
"jobTaskAttemptCounters" : {
"taskAttemptCounterGroup" : [
{
"counterGroupName" : "org.apache.hadoop.mapreduce.FileSystemCounter",
"counter" : [
{
"value" : 2363,
"name" : "FILE_BYTES_READ"
},
{
"value" : 54372,
"name" : "FILE_BYTES_WRITTEN"
},
{
"value" : 0,
"name" : "FILE_READ_OPS"
},
{
"value" : 0,
"name" : "FILE_LARGE_READ_OPS"
},
{
"value" : 0,
"name" : "FILE_WRITE_OPS"
},
{
"value" : 0,
"name" : "HDFS_BYTES_READ"
},
{
"value" : 0,
"name" : "HDFS_BYTES_WRITTEN"
},
{
"value" : 0,
"name" : "HDFS_READ_OPS"
},
{
"value" : 0,
"name" : "HDFS_LARGE_READ_OPS"
},
{
"value" : 0,
"name" : "HDFS_WRITE_OPS"
}
]
},
{
"counterGroupName" : "org.apache.hadoop.mapreduce.TaskCounter",
"counter" : [
{
"value" : 0,
"name" : "COMBINE_INPUT_RECORDS"
},
{
"value" : 0,
"name" : "COMBINE_OUTPUT_RECORDS"
},
{
"value" : 460,
"name" : "REDUCE_INPUT_GROUPS"
},
{
"value" : 2235,
"name" : "REDUCE_SHUFFLE_BYTES"
},
{
"value" : 460,
"name" : "REDUCE_INPUT_RECORDS"
},
{
"value" : 0,
"name" : "REDUCE_OUTPUT_RECORDS"
},
{
"value" : 0,
"name" : "SPILLED_RECORDS"
},
{
"value" : 1,
"name" : "SHUFFLED_MAPS"
},
{
"value" : 0,
"name" : "FAILED_SHUFFLE"
},
{
"value" : 1,
"name" : "MERGED_MAP_OUTPUTS"
},
{
"value" : 26,
"name" : "GC_TIME_MILLIS"
},
{
"value" : 860,
"name" : "CPU_MILLISECONDS"
},
{
"value" : 107839488,
"name" : "PHYSICAL_MEMORY_BYTES"
},
{
"value" : 1123147776,
"name" : "VIRTUAL_MEMORY_BYTES"
},
{
"value" : 57475072,
"name" : "COMMITTED_HEAP_BYTES"
}
]
},
{
"counterGroupName" : "Shuffle Errors",
"counter" : [
{
"value" : 0,
"name" : "BAD_ID"
},
{
"value" : 0,
"name" : "CONNECTION"
},
{
"value" : 0,
"name" : "IO_ERROR"
},
{
"value" : 0,
"name" : "WRONG_LENGTH"
},
{
"value" : 0,
"name" : "WRONG_MAP"
},
{
"value" : 0,
"name" : "WRONG_REDUCE"
}
]
},
{
"counterGroupName" : "org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounter",
"counter" : [
{
"value" : 0,
"name" : "BYTES_WRITTEN"
}
]
}
],
"id" : "attempt_1326381300833_2_2_m_0_0"
}
}
**XML response**
HTTP Request:
GET http://history-server-http-address:port/ws/v1/history/mapreduce/jobs/job_1326381300833_2_2/tasks/task_1326381300833_2_2_m_0/attempts/attempt_1326381300833_2_2_m_0_0/counters
Accept: application/xml
Response Header:
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 2735
Server: Jetty(6.1.26)
Response Body:
attempt_1326381300833_2_2_m_0_0org.apache.hadoop.mapreduce.FileSystemCounterFILE_BYTES_READ2363FILE_BYTES_WRITTEN54372FILE_READ_OPS0FILE_LARGE_READ_OPS0FILE_WRITE_OPS0HDFS_BYTES_READ0HDFS_BYTES_WRITTEN0HDFS_READ_OPS0HDFS_LARGE_READ_OPS0HDFS_WRITE_OPS0org.apache.hadoop.mapreduce.TaskCounterCOMBINE_INPUT_RECORDS0COMBINE_OUTPUT_RECORDS0REDUCE_INPUT_GROUPS460REDUCE_SHUFFLE_BYTES2235REDUCE_INPUT_RECORDS460REDUCE_OUTPUT_RECORDS0SPILLED_RECORDS0SHUFFLED_MAPS1FAILED_SHUFFLE0MERGED_MAP_OUTPUTS1GC_TIME_MILLIS26CPU_MILLISECONDS860PHYSICAL_MEMORY_BYTES107839488VIRTUAL_MEMORY_BYTES1123147776COMMITTED_HEAP_BYTES57475072Shuffle ErrorsBAD_ID0CONNECTION0IO_ERROR0WRONG_LENGTH0WRONG_MAP0WRONG_REDUCE0org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounterBYTES_WRITTEN0