|
|
@@ -0,0 +1,224 @@
|
|
|
+~~ Licensed to the Apache Software Foundation (ASF) under one or more
|
|
|
+~~ contributor license agreements. See the NOTICE file distributed with
|
|
|
+~~ this work for additional information regarding copyright ownership.
|
|
|
+~~ The ASF licenses this file to You under the Apache License, Version 2.0
|
|
|
+~~ (the "License"); you may not use this file except in compliance with
|
|
|
+~~ the License. You may obtain a copy of the License at
|
|
|
+~~
|
|
|
+~~ http://www.apache.org/licenses/LICENSE-2.0
|
|
|
+~~
|
|
|
+~~ Unless required by applicable law or agreed to in writing, software
|
|
|
+~~ distributed under the License is distributed on an "AS IS" BASIS,
|
|
|
+~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
|
+~~ See the License for the specific language governing permissions and
|
|
|
+~~ limitations under the License.
|
|
|
+
|
|
|
+ ---
|
|
|
+ MapReduce Commands Guide
|
|
|
+ ---
|
|
|
+ ---
|
|
|
+ ${maven.build.timestamp}
|
|
|
+
|
|
|
+MapReduce Commands Guide
|
|
|
+
|
|
|
+%{toc|section=1|fromDepth=2|toDepth=4}
|
|
|
+
|
|
|
+* Overview
|
|
|
+
|
|
|
+ MapReduce commands are invoked by the <<<bin/mapred>>> script. Running the
|
|
|
+ script without any arguments prints the description for all commands.
|
|
|
+
|
|
|
+ Usage: <<<mapred [--config confdir] COMMAND>>>
|
|
|
+
|
|
|
+ MapReduce has an option parsing framework that employs parsing generic
|
|
|
+ options as well as running classes.
|
|
|
+
|
|
|
+*-------------------------+---------------------------------------------------+
|
|
|
+|| COMMAND_OPTIONS || Description |
|
|
|
+*-------------------------+---------------------------------------------------+
|
|
|
+| --config confdir | Overwrites the default Configuration directory. Default
|
|
|
+| | is $\{HADOOP_PREFIX\}/conf.
|
|
|
+*-------------------------+---------------------------------------------------+
|
|
|
+| COMMAND COMMAND_OPTIONS | Various commands with their options are described
|
|
|
+| | in the following sections. The commands have been
|
|
|
+| | grouped into {{User Commands}} and
|
|
|
+| | {{Administration Commands}}.
|
|
|
+*-------------------------+---------------------------------------------------+
|
|
|
+
|
|
|
+* User Commands
|
|
|
+
|
|
|
+ Commands useful for users of a hadoop cluster.
|
|
|
+
|
|
|
+** <<<pipes>>>
|
|
|
+
|
|
|
+ Runs a pipes job.
|
|
|
+
|
|
|
+ Usage: <<<mapred pipes [-conf <path>] [-jobconf <key=value>, <key=value>,
|
|
|
+ ...] [-input <path>] [-output <path>] [-jar <jar file>] [-inputformat
|
|
|
+ <class>] [-map <class>] [-partitioner <class>] [-reduce <class>] [-writer
|
|
|
+ <class>] [-program <executable>] [-reduces <num>]>>>
|
|
|
+
|
|
|
+*----------------------------------------+------------------------------------+
|
|
|
+|| COMMAND_OPTION || Description
|
|
|
+*----------------------------------------+------------------------------------+
|
|
|
+| -conf <path> | Configuration for job
|
|
|
+*----------------------------------------+------------------------------------+
|
|
|
+| -jobconf <key=value>, <key=value>, ... | Add/override configuration for job
|
|
|
+*----------------------------------------+------------------------------------+
|
|
|
+| -input <path> | Input directory
|
|
|
+*----------------------------------------+------------------------------------+
|
|
|
+| -output <path> | Output directory
|
|
|
+*----------------------------------------+------------------------------------+
|
|
|
+| -jar <jar file> | Jar filename
|
|
|
+*----------------------------------------+------------------------------------+
|
|
|
+| -inputformat <class> | InputFormat class
|
|
|
+*----------------------------------------+------------------------------------+
|
|
|
+| -map <class> | Java Map class
|
|
|
+*----------------------------------------+------------------------------------+
|
|
|
+| -partitioner <class> | Java Partitioner
|
|
|
+*----------------------------------------+------------------------------------+
|
|
|
+| -reduce <class> | Java Reduce class
|
|
|
+*----------------------------------------+------------------------------------+
|
|
|
+| -writer <class> | Java RecordWriter
|
|
|
+*----------------------------------------+------------------------------------+
|
|
|
+| -program <executable> | Executable URI
|
|
|
+*----------------------------------------+------------------------------------+
|
|
|
+| -reduces <num> | Number of reduces
|
|
|
+*----------------------------------------+------------------------------------+
|
|
|
+
|
|
|
+** <<<job>>>
|
|
|
+
|
|
|
+ Command to interact with Map Reduce Jobs.
|
|
|
+
|
|
|
+ Usage: <<<mapred job
|
|
|
+ | [{{{../../hadoop-project-dist/hadoop-common/CommandsManual.html#Generic_Options}GENERIC_OPTIONS}}]
|
|
|
+ | [-submit <job-file>]
|
|
|
+ | [-status <job-id>]
|
|
|
+ | [-counter <job-id> <group-name> <counter-name>]
|
|
|
+ | [-kill <job-id>]
|
|
|
+ | [-events <job-id> <from-event-#> <#-of-events>]
|
|
|
+ | [-history [all] <jobOutputDir>] | [-list [all]]
|
|
|
+ | [-kill-task <task-id>] | [-fail-task <task-id>]
|
|
|
+ | [-set-priority <job-id> <priority>]>>>
|
|
|
+
|
|
|
+*------------------------------+---------------------------------------------+
|
|
|
+|| COMMAND_OPTION || Description
|
|
|
+*------------------------------+---------------------------------------------+
|
|
|
+| -submit <job-file> | Submits the job.
|
|
|
+*------------------------------+---------------------------------------------+
|
|
|
+| -status <job-id> | Prints the map and reduce completion
|
|
|
+ | percentage and all job counters.
|
|
|
+*------------------------------+---------------------------------------------+
|
|
|
+| -counter <job-id> <group-name> <counter-name> | Prints the counter value.
|
|
|
+*------------------------------+---------------------------------------------+
|
|
|
+| -kill <job-id> | Kills the job.
|
|
|
+*------------------------------+---------------------------------------------+
|
|
|
+| -events <job-id> <from-event-#> <#-of-events> | Prints the events' details
|
|
|
+ | received by jobtracker for the given range.
|
|
|
+*------------------------------+---------------------------------------------+
|
|
|
+| -history [all]<jobOutputDir> | Prints job details, failed and killed tip
|
|
|
+ | details. More details about the job such as
|
|
|
+ | successful tasks and task attempts made for
|
|
|
+ | each task can be viewed by specifying the
|
|
|
+ | [all] option.
|
|
|
+*------------------------------+---------------------------------------------+
|
|
|
+| -list [all] | Displays jobs which are yet to complete.
|
|
|
+ | <<<-list all>>> displays all jobs.
|
|
|
+*------------------------------+---------------------------------------------+
|
|
|
+| -kill-task <task-id> | Kills the task. Killed tasks are NOT counted
|
|
|
+ | against failed attempts.
|
|
|
+*------------------------------+---------------------------------------------+
|
|
|
+| -fail-task <task-id> | Fails the task. Failed tasks are counted
|
|
|
+ | against failed attempts.
|
|
|
+*------------------------------+---------------------------------------------+
|
|
|
+| -set-priority <job-id> <priority> | Changes the priority of the job. Allowed
|
|
|
+ | priority values are VERY_HIGH, HIGH, NORMAL,
|
|
|
+ | LOW, VERY_LOW
|
|
|
+*------------------------------+---------------------------------------------+
|
|
|
+
|
|
|
+** <<<queue>>>
|
|
|
+
|
|
|
+ command to interact and view Job Queue information
|
|
|
+
|
|
|
+ Usage: <<<mapred queue [-list] | [-info <job-queue-name> [-showJobs]]
|
|
|
+ | [-showacls]>>>
|
|
|
+
|
|
|
+*-----------------+-----------------------------------------------------------+
|
|
|
+|| COMMAND_OPTION || Description
|
|
|
+*-----------------+-----------------------------------------------------------+
|
|
|
+| -list | Gets list of Job Queues configured in the system.
|
|
|
+ | Along with scheduling information associated with the job
|
|
|
+ | queues.
|
|
|
+*-----------------+-----------------------------------------------------------+
|
|
|
+| -info <job-queue-name> [-showJobs] | Displays the job queue information and
|
|
|
+ | associated scheduling information of particular job queue.
|
|
|
+ | If <<<-showJobs>>> options is present a list of jobs
|
|
|
+ | submitted to the particular job queue is displayed.
|
|
|
+*-----------------+-----------------------------------------------------------+
|
|
|
+| -showacls | Displays the queue name and associated queue operations
|
|
|
+ | allowed for the current user. The list consists of only
|
|
|
+ | those queues to which the user has access.
|
|
|
+*-----------------+-----------------------------------------------------------+
|
|
|
+
|
|
|
+** <<<classpath>>>
|
|
|
+
|
|
|
+ Prints the class path needed to get the Hadoop jar and the required
|
|
|
+ libraries.
|
|
|
+
|
|
|
+ Usage: <<<mapred classpath>>>
|
|
|
+
|
|
|
+** <<<distcp>>>
|
|
|
+
|
|
|
+ Copy file or directories recursively. More information can be found at
|
|
|
+ {{{./DistCp.html}Hadoop DistCp Guide}}.
|
|
|
+
|
|
|
+** <<<archive>>>
|
|
|
+
|
|
|
+ Creates a hadoop archive. More information can be found at
|
|
|
+ {{{./HadoopArchives.html}Hadoop Archives Guide}}.
|
|
|
+
|
|
|
+* Administration Commands
|
|
|
+
|
|
|
+ Commands useful for administrators of a hadoop cluster.
|
|
|
+
|
|
|
+** <<<historyserver>>>
|
|
|
+
|
|
|
+ Start JobHistoryServer.
|
|
|
+
|
|
|
+ Usage: <<<mapred historyserver>>>
|
|
|
+
|
|
|
+** <<<hsadmin>>>
|
|
|
+
|
|
|
+ Runs a MapReduce hsadmin client for execute JobHistoryServer administrative
|
|
|
+ commands.
|
|
|
+
|
|
|
+ Usage: <<<mapred hsadmin
|
|
|
+ [-refreshUserToGroupsMappings] |
|
|
|
+ [-refreshSuperUserGroupsConfiguration] |
|
|
|
+ [-refreshAdminAcls] |
|
|
|
+ [-refreshLoadedJobCache] |
|
|
|
+ [-refreshLogRetentionSettings] |
|
|
|
+ [-refreshJobRetentionSettings] |
|
|
|
+ [-getGroups [username]] | [-help [cmd]]>>>
|
|
|
+
|
|
|
+*-----------------+-----------------------------------------------------------+
|
|
|
+|| COMMAND_OPTION || Description
|
|
|
+*-----------------+-----------------------------------------------------------+
|
|
|
+| -refreshUserToGroupsMappings | Refresh user-to-groups mappings
|
|
|
+*-----------------+-----------------------------------------------------------+
|
|
|
+| -refreshSuperUserGroupsConfiguration| Refresh superuser proxy groups mappings
|
|
|
+*-----------------+-----------------------------------------------------------+
|
|
|
+| -refreshAdminAcls | Refresh acls for administration of Job history server
|
|
|
+*-----------------+-----------------------------------------------------------+
|
|
|
+| -refreshLoadedJobCache | Refresh loaded job cache of Job history server
|
|
|
+*-----------------+-----------------------------------------------------------+
|
|
|
+| -refreshJobRetentionSettings|Refresh job history period, job cleaner settings
|
|
|
+*-----------------+-----------------------------------------------------------+
|
|
|
+| -refreshLogRetentionSettings | Refresh log retention period and log retention
|
|
|
+| | check interval
|
|
|
+*-----------------+-----------------------------------------------------------+
|
|
|
+| -getGroups [username] | Get the groups which given user belongs to
|
|
|
+*-----------------+-----------------------------------------------------------+
|
|
|
+| -help [cmd] | Displays help for the given command or all commands if none is
|
|
|
+| | specified.
|
|
|
+*-----------------+-----------------------------------------------------------+
|
Allen Wittenauer