|
|
@@ -378,7 +378,7 @@ $HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/hadoop-streaming.jar \
|
|
|
Often, you may want to process input data using a map function only. To do this, simply set mapred.reduce.tasks to zero. The Map/Reduce framework will not create any reducer tasks. Rather, the outputs of the mapper tasks will be the final output of the job.
|
|
|
</p>
|
|
|
<p>
|
|
|
-To be backward compatible, Hadoop Streaming also supports the "-reduce NONE" option, which is equivalent to "-jobconf mapred.reduce.tasks=0".
|
|
|
+To be backward compatible, Hadoop Streaming also supports the "-reduce NONE" option, which is equivalent to "-D mapred.reduce.tasks=0".
|
|
|
</p>
|
|
|
<a name="N10083"></a><a name="Specifying+Other+Plugins+for+Jobs"></a>
|
|
|
<h3 class="h4">Specifying Other Plugins for Jobs </h3>
|
|
|
@@ -400,31 +400,31 @@ The class you supply for the output format is expected to take key/value pairs o
|
|
|
<a name="N10096"></a><a name="Large+files+and+archives+in+Hadoop+Streaming"></a>
|
|
|
<h3 class="h4">Large files and archives in Hadoop Streaming </h3>
|
|
|
<p>
|
|
|
-The -cacheFile and -cacheArchive options allow you to make files and archives available to the tasks. The argument is a URI to the file or archive that you have already uploaded to HDFS. These files and archives are cached across jobs. You can retrieve the host and fs_port values from the fs.default.name config variable.
|
|
|
+The -files and -archives options allow you to make files and archives available to the tasks. The argument is a URI to the file or archive that you have already uploaded to HDFS. These files and archives are cached across jobs. You can retrieve the host and fs_port values from the fs.default.name config variable.
|
|
|
</p>
|
|
|
<p>
|
|
|
-Here are examples of the -cacheFile option:
|
|
|
+Here are examples of the -files option:
|
|
|
</p>
|
|
|
<pre class="code">
|
|
|
--cacheFile hdfs://host:fs_port/user/testfile.txt#testlink
|
|
|
+-files hdfs://host:fs_port/user/testfile.txt#testlink
|
|
|
</pre>
|
|
|
<p>
|
|
|
In the above example, the part of the url after # is used as the symlink name that is created in the current working directory of tasks. So the tasks will have a symlink called testlink in the cwd that points to a local copy of testfile.txt. Multiple entries can be specified as:
|
|
|
</p>
|
|
|
<pre class="code">
|
|
|
--cacheFile hdfs://host:fs_port/user/testfile1.txt#testlink1 -cacheFile hdfs://host:fs_port/user/testfile2.txt#testlink2
|
|
|
+-files hdfs://host:fs_port/user/testfile1.txt#testlink1 -files hdfs://host:fs_port/user/testfile2.txt#testlink2
|
|
|
</pre>
|
|
|
<p>
|
|
|
-The -cacheArchive option allows you to copy jars locally to the cwd of tasks and automatically unjar the files. For example:
|
|
|
+The -archives option allows you to copy jars locally to the cwd of tasks and automatically unjar the files. For example:
|
|
|
</p>
|
|
|
<pre class="code">
|
|
|
--cacheArchive hdfs://host:fs_port/user/testfile.jar#testlink3
|
|
|
+-archives hdfs://host:fs_port/user/testfile.jar#testlink3
|
|
|
</pre>
|
|
|
<p>
|
|
|
In the example above, a symlink testlink3 is created in the current working directory of tasks. This symlink points to the directory that stores the unjarred contents of the uploaded jar file.
|
|
|
</p>
|
|
|
<p>
|
|
|
-Here's another example of the -cacheArchive option. Here, the input.txt file has two lines specifying the names of the two files: testlink/cache.txt and testlink/cache2.txt. "testlink" is a symlink to the archived directory, which has the files "cache.txt" and "cache2.txt".
|
|
|
+Here's another example of the -archives option. Here, the input.txt file has two lines specifying the names of the two files: testlink/cache.txt and testlink/cache2.txt. "testlink" is a symlink to the archived directory, which has the files "cache.txt" and "cache2.txt".
|
|
|
</p>
|
|
|
<pre class="code">
|
|
|
$HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/hadoop-streaming.jar \
|
|
|
@@ -432,10 +432,10 @@ $HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/hadoop-streaming.jar \
|
|
|
-mapper "xargs cat" \
|
|
|
-reducer "cat" \
|
|
|
-output "/user/me/samples/cachefile/out" \
|
|
|
- -cacheArchive 'hdfs://hadoop-nn1.example.com/user/me/samples/cachefile/cachedir.jar#testlink' \
|
|
|
- -jobconf mapred.map.tasks=1 \
|
|
|
- -jobconf mapred.reduce.tasks=1 \
|
|
|
- -jobconf mapred.job.name="Experiment"
|
|
|
+ -archives 'hdfs://hadoop-nn1.example.com/user/me/samples/cachefile/cachedir.jar#testlink' \
|
|
|
+ -D mapred.map.tasks=1 \
|
|
|
+ -D mapred.reduce.tasks=1 \
|
|
|
+ -D mapred.job.name="Experiment"
|
|
|
|
|
|
$ ls test_jar/
|
|
|
cache.txt cache2.txt
|
|
|
@@ -469,7 +469,7 @@ This is just the second cache string
|
|
|
<a name="N100BF"></a><a name="Specifying+Additional+Configuration+Variables+for+Jobs"></a>
|
|
|
<h3 class="h4">Specifying Additional Configuration Variables for Jobs </h3>
|
|
|
<p>
|
|
|
-You can specify additional configuration variables by using "-jobconf <n>=<v>". For example:
|
|
|
+You can specify additional configuration variables by using "-D <n>=<v>". For example:
|
|
|
</p>
|
|
|
<pre class="code">
|
|
|
$HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/hadoop-streaming.jar \
|
|
|
@@ -477,10 +477,10 @@ $HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/hadoop-streaming.jar \
|
|
|
-output myOutputDir \
|
|
|
-mapper org.apache.hadoop.mapred.lib.IdentityMapper\
|
|
|
-reducer /bin/wc \
|
|
|
- -jobconf mapred.reduce.tasks=2
|
|
|
+ -D mapred.reduce.tasks=2
|
|
|
</pre>
|
|
|
<p>
|
|
|
-The -jobconf mapred.reduce.tasks=2 in the above example specifies to use two reducers for the job.
|
|
|
+The -D mapred.reduce.tasks=2 in the above example specifies to use two reducers for the job.
|
|
|
</p>
|
|
|
<p>
|
|
|
For more details on the jobconf parameters see:
|
|
|
@@ -497,63 +497,81 @@ Other options you may specify for a streaming job are described here:
|
|
|
<th colspan="1" rowspan="1">Parameter</th><th colspan="1" rowspan="1">Optional/Required </th><th colspan="1" rowspan="1">Description </th>
|
|
|
</tr>
|
|
|
|
|
|
+
|
|
|
<tr>
|
|
|
-<td colspan="1" rowspan="1"> -cluster name </td><td colspan="1" rowspan="1"> Optional </td><td colspan="1" rowspan="1"> Switch between local Hadoop and one or more remote clusters </td>
|
|
|
+<td colspan="1" rowspan="1"> -cmdenv name=value </td><td colspan="1" rowspan="1"> Optional </td><td colspan="1" rowspan="1"> Pass env var to streaming commands </td>
|
|
|
</tr>
|
|
|
|
|
|
|
|
|
<tr>
|
|
|
-<td colspan="1" rowspan="1"> -dfs host:port or local </td><td colspan="1" rowspan="1"> Optional </td><td colspan="1" rowspan="1"> Override the HDFS configuration for the job </td>
|
|
|
+<td colspan="1" rowspan="1"> -inputreader JavaClassName </td><td colspan="1" rowspan="1"> Optional </td><td colspan="1" rowspan="1"> For backwards-compatibility: specifies a record reader class (instead of an input format class) </td>
|
|
|
</tr>
|
|
|
|
|
|
<tr>
|
|
|
-<td colspan="1" rowspan="1"> -jt host:port or local </td><td colspan="1" rowspan="1"> Optional </td><td colspan="1" rowspan="1"> Override the JobTracker configuration for the job </td>
|
|
|
+<td colspan="1" rowspan="1"> -verbose </td><td colspan="1" rowspan="1"> Optional </td><td colspan="1" rowspan="1"> Verbose output </td>
|
|
|
</tr>
|
|
|
|
|
|
+</table>
|
|
|
+<p>
|
|
|
+Streaming support Hadoop generic command line options.
|
|
|
+
|
|
|
+Supported parameters are :
|
|
|
+The general command line syntax is :
|
|
|
+<br> bin/hadoop command [genericOptions] [commandOptions]
|
|
|
+</p>
|
|
|
+<table class="ForrestTable" cellspacing="1" cellpadding="4">
|
|
|
+
|
|
|
<tr>
|
|
|
-<td colspan="1" rowspan="1"> -additionalconfspec specfile </td><td colspan="1" rowspan="1"> Optional </td><td colspan="1" rowspan="1"> Specifies a set of configuration variables in an XML file like hadoop-site.xml, instead of using multiple options of type "-jobconf name=value" </td>
|
|
|
+<th colspan="1" rowspan="1">Parameter</th><th colspan="1" rowspan="1">Optional/Required </th><th colspan="1" rowspan="1">Description </th>
|
|
|
</tr>
|
|
|
|
|
|
|
|
|
<tr>
|
|
|
-<td colspan="1" rowspan="1"> -cmdenv name=value </td><td colspan="1" rowspan="1"> Optional </td><td colspan="1" rowspan="1"> Pass env var to streaming commands </td>
|
|
|
+<td colspan="1" rowspan="1"> -conf configuration_file </td><td colspan="1" rowspan="1"> Optional </td><td colspan="1" rowspan="1"> specify an application configuration file </td>
|
|
|
</tr>
|
|
|
|
|
|
<tr>
|
|
|
-<td colspan="1" rowspan="1"> -cacheFile fileNameURI </td><td colspan="1" rowspan="1"> Optional </td><td colspan="1" rowspan="1"> Specify a file to be uploaded to the HDFS </td>
|
|
|
+<td colspan="1" rowspan="1"> -D property=value </td><td colspan="1" rowspan="1"> Optional </td><td colspan="1" rowspan="1"> use value for given property </td>
|
|
|
</tr>
|
|
|
|
|
|
<tr>
|
|
|
-<td colspan="1" rowspan="1"> -cacheArchive fileNameURI </td><td colspan="1" rowspan="1"> Optional </td><td colspan="1" rowspan="1"> Specify a jar file to be uploaded to the HDFS. This jar file is unjarred automatically in the cwd of the task </td>
|
|
|
+<td colspan="1" rowspan="1"> -fs host:port or local </td><td colspan="1" rowspan="1"> Optional </td><td colspan="1" rowspan="1"> specify a namenode </td>
|
|
|
</tr>
|
|
|
|
|
|
+<tr>
|
|
|
+<td colspan="1" rowspan="1"> -jt host:port or local </td><td colspan="1" rowspan="1"> Optional </td><td colspan="1" rowspan="1"> specify a job tracker </td>
|
|
|
+</tr>
|
|
|
|
|
|
<tr>
|
|
|
-<td colspan="1" rowspan="1"> -inputreader JavaClassName </td><td colspan="1" rowspan="1"> Optional </td><td colspan="1" rowspan="1"> For backwards-compatibility: specifies a record reader class (instead of an input format class) </td>
|
|
|
+<td colspan="1" rowspan="1"> -files </td><td colspan="1" rowspan="1"> Optional </td><td colspan="1" rowspan="1"> specify comma separated files to be copied to the map reduce cluster </td>
|
|
|
</tr>
|
|
|
|
|
|
<tr>
|
|
|
-<td colspan="1" rowspan="1"> -verbose </td><td colspan="1" rowspan="1"> Optional </td><td colspan="1" rowspan="1"> Verbose output </td>
|
|
|
+<td colspan="1" rowspan="1"> -archives </td><td colspan="1" rowspan="1"> Optional </td><td colspan="1" rowspan="1"> specify comma separated archives to be unarchived on the compute machines </td>
|
|
|
+</tr>
|
|
|
+
|
|
|
+<tr>
|
|
|
+<td colspan="1" rowspan="1"> </td><td colspan="1" rowspan="1"> Optional </td><td colspan="1" rowspan="1"> </td>
|
|
|
+</tr>
|
|
|
+
|
|
|
+<tr>
|
|
|
+<td colspan="1" rowspan="1"> -jt host:port or local </td><td colspan="1" rowspan="1"> Optional </td><td colspan="1" rowspan="1"> </td>
|
|
|
</tr>
|
|
|
|
|
|
</table>
|
|
|
<p>
|
|
|
-To switch between "local" Hadoop and one or more remote Hadoop clusters use -cluster <name>.
|
|
|
-By default, hadoop-default.xml and hadoop-site.xml are used. The -cluster <name> option will cause $HADOOP_HOME/conf/hadoop-<name>.xml to be used instead.
|
|
|
-</p>
|
|
|
-<p>
|
|
|
To change the local temp directory use:
|
|
|
</p>
|
|
|
<pre class="code">
|
|
|
- -jobconf dfs.data.dir=/tmp
|
|
|
+ -D dfs.data.dir=/tmp
|
|
|
</pre>
|
|
|
<p>
|
|
|
To specify additional local temp directories use:
|
|
|
</p>
|
|
|
<pre class="code">
|
|
|
- -jobconf mapred.local.dir=/tmp/local
|
|
|
- -jobconf mapred.system.dir=/tmp/system
|
|
|
- -jobconf mapred.temp.dir=/tmp/temp
|
|
|
+ -D mapred.local.dir=/tmp/local
|
|
|
+ -D mapred.system.dir=/tmp/system
|
|
|
+ -D mapred.temp.dir=/tmp/temp
|
|
|
</pre>
|
|
|
<p>
|
|
|
For more details on jobconf parameters see:
|
|
|
@@ -568,10 +586,10 @@ To set an environment variable in a streaming command use:
|
|
|
</div>
|
|
|
|
|
|
|
|
|
-<a name="N1018E"></a><a name="More+usage+examples"></a>
|
|
|
+<a name="N101BD"></a><a name="More+usage+examples"></a>
|
|
|
<h2 class="h3">More usage examples </h2>
|
|
|
<div class="section">
|
|
|
-<a name="N10194"></a><a name="Customizing+the+Way+to+Split+Lines+into+Key%2FValue+Pairs"></a>
|
|
|
+<a name="N101C3"></a><a name="Customizing+the+Way+to+Split+Lines+into+Key%2FValue+Pairs"></a>
|
|
|
<h3 class="h4">Customizing the Way to Split Lines into Key/Value Pairs </h3>
|
|
|
<p>
|
|
|
As noted earlier, when the Map/Reduce framework reads a line from the stdout of the mapper, it splits the line into a key/value pair. By default, the prefix of the line up to the first tab character is the key and the the rest of the line (excluding the tab character) is the value.
|
|
|
@@ -585,16 +603,16 @@ $HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/hadoop-streaming.jar \
|
|
|
-output myOutputDir \
|
|
|
-mapper org.apache.hadoop.mapred.lib.IdentityMapper \
|
|
|
-reducer org.apache.hadoop.mapred.lib.IdentityReducer \
|
|
|
- -jobconf stream.map.output.field.separator=. \
|
|
|
- -jobconf stream.num.map.output.key.fields=4
|
|
|
+ -D stream.map.output.field.separator=. \
|
|
|
+ -D stream.num.map.output.key.fields=4
|
|
|
</pre>
|
|
|
<p>
|
|
|
-In the above example, "-jobconf stream.map.output.field.separator=." specifies "." as the field separator for the map outputs, and the prefix up to the fourth "." in a line will be the key and the rest of the line (excluding the fourth ".") will be the value. If a line has less than four "."s, then the whole line will be the key and the value will be an empty Text object (like the one created by new Text("")).
|
|
|
+In the above example, "-D stream.map.output.field.separator=." specifies "." as the field separator for the map outputs, and the prefix up to the fourth "." in a line will be the key and the rest of the line (excluding the fourth ".") will be the value. If a line has less than four "."s, then the whole line will be the key and the value will be an empty Text object (like the one created by new Text("")).
|
|
|
</p>
|
|
|
<p>
|
|
|
-Similarly, you can use "-jobconf stream.reduce.output.field.separator=SEP" and "-jobconf stream.num.reduce.output.fields=NUM" to specify the nth field separator in a line of the reduce outputs as the separator between the key and the value.
|
|
|
+Similarly, you can use "-D stream.reduce.output.field.separator=SEP" and "-D stream.num.reduce.output.fields=NUM" to specify the nth field separator in a line of the reduce outputs as the separator between the key and the value.
|
|
|
</p>
|
|
|
-<a name="N101AA"></a><a name="A+Useful+Partitioner+Class+%28secondary+sort%2C+the+-partitioner+org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner+option%29"></a>
|
|
|
+<a name="N101D9"></a><a name="A+Useful+Partitioner+Class+%28secondary+sort%2C+the+-partitioner+org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner+option%29"></a>
|
|
|
<h3 class="h4">A Useful Partitioner Class (secondary sort, the -partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner option) </h3>
|
|
|
<p>
|
|
|
Hadoop has a library class, org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner, that is useful for many applications. This class allows the Map/Reduce framework to partition the map outputs based on prefixes of keys, not the whole keys. For example:
|
|
|
@@ -606,17 +624,17 @@ $HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/hadoop-streaming.jar \
|
|
|
-mapper org.apache.hadoop.mapred.lib.IdentityMapper \
|
|
|
-reducer org.apache.hadoop.mapred.lib.IdentityReducer \
|
|
|
-partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner \
|
|
|
- -jobconf stream.map.output.field.separator=. \
|
|
|
- -jobconf stream.num.map.output.key.fields=4 \
|
|
|
- -jobconf map.output.key.field.separator=. \
|
|
|
- -jobconf num.key.fields.for.partition=2 \
|
|
|
- -jobconf mapred.reduce.tasks=12
|
|
|
+ -D stream.map.output.field.separator=. \
|
|
|
+ -D stream.num.map.output.key.fields=4 \
|
|
|
+ -D map.output.key.field.separator=. \
|
|
|
+ -D num.key.fields.for.partition=2 \
|
|
|
+ -D mapred.reduce.tasks=12
|
|
|
</pre>
|
|
|
<p>
|
|
|
-Here, <em>-jobconf stream.map.output.field.separator=.</em> and <em>-jobconf stream.num.map.output.key.fields=4</em> are as explained in previous example. The two variables are used by streaming to identify the key/value pair of mapper.
|
|
|
+Here, <em>-D stream.map.output.field.separator=.</em> and <em>-D stream.num.map.output.key.fields=4</em> are as explained in previous example. The two variables are used by streaming to identify the key/value pair of mapper.
|
|
|
</p>
|
|
|
<p>
|
|
|
-The map output keys of the above Map/Reduce job normally have four fields separated by ".". However, the Map/Reduce framework will partition the map outputs by the first two fields of the keys using the <em>-jobconf num.key.fields.for.partition=2</em> option. Here, <em>-jobconf map.output.key.field.separator=.</em> specifies the separator for the partition. This guarantees that all the key/value pairs with the same first two fields in the keys will be partitioned into the same reducer.
|
|
|
+The map output keys of the above Map/Reduce job normally have four fields separated by ".". However, the Map/Reduce framework will partition the map outputs by the first two fields of the keys using the <em>-D num.key.fields.for.partition=2</em> option. Here, <em>-D map.output.key.field.separator=.</em> specifies the separator for the partition. This guarantees that all the key/value pairs with the same first two fields in the keys will be partitioned into the same reducer.
|
|
|
</p>
|
|
|
<p>
|
|
|
|
|
|
@@ -654,7 +672,7 @@ Sorting within each partition for the reducer(all 4 fields used for sorting)</p>
|
|
|
11.14.2.2
|
|
|
11.14.2.3
|
|
|
</pre>
|
|
|
-<a name="N101E0"></a><a name="Working+with+the+Hadoop+Aggregate+Package+%28the+-reduce+aggregate+option%29"></a>
|
|
|
+<a name="N1020F"></a><a name="Working+with+the+Hadoop+Aggregate+Package+%28the+-reduce+aggregate+option%29"></a>
|
|
|
<h3 class="h4">Working with the Hadoop Aggregate Package (the -reduce aggregate option) </h3>
|
|
|
<p>
|
|
|
Hadoop has a library package called "Aggregate" (
|
|
|
@@ -677,7 +695,7 @@ $HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/hadoop-streaming.jar \
|
|
|
-mapper myAggregatorForKeyCount.py \
|
|
|
-reducer aggregate \
|
|
|
-file myAggregatorForKeyCount.py \
|
|
|
- -jobconf mapred.reduce.tasks=12
|
|
|
+ -D mapred.reduce.tasks=12
|
|
|
</pre>
|
|
|
<p>
|
|
|
The python program myAggregatorForKeyCount.py looks like:
|
|
|
@@ -703,7 +721,7 @@ def main(argv):
|
|
|
if __name__ == "__main__":
|
|
|
main(sys.argv)
|
|
|
</pre>
|
|
|
-<a name="N101FB"></a><a name="Field+Selection+%28+similar+to+unix+%27cut%27+command%29"></a>
|
|
|
+<a name="N1022A"></a><a name="Field+Selection+%28+similar+to+unix+%27cut%27+command%29"></a>
|
|
|
<h3 class="h4">Field Selection ( similar to unix 'cut' command) </h3>
|
|
|
<p>
|
|
|
Hadoop has a library class, org.apache.hadoop.mapred.lib.FieldSelectionMapReduce, that effectively allows you to process text data like the unix "cut" utility. The map function defined in the class treats each input key/value pair as a list of fields. You can specify the field separator (the default is the tab character). You can select an arbitrary list of fields as the map output key, and an arbitrary list of fields as the map output value. Similarly, the reduce function defined in the class treats each input key/value pair as a list of fields. You can select an arbitrary list of fields as the reduce output key, and an arbitrary list of fields as the reduce output value. For example:
|
|
|
@@ -715,32 +733,32 @@ $HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/hadoop-streaming.jar \
|
|
|
-mapper org.apache.hadoop.mapred.lib.FieldSelectionMapReduce\
|
|
|
-reducer org.apache.hadoop.mapred.lib.FieldSelectionMapReduce\
|
|
|
-partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner \
|
|
|
- -jobconf map.output.key.field.separa=. \
|
|
|
- -jobconf num.key.fields.for.partition=2 \
|
|
|
- -jobconf mapred.data.field.separator=. \
|
|
|
- -jobconf map.output.key.value.fields.spec=6,5,1-3:0- \
|
|
|
- -jobconf reduce.output.key.value.fields.spec=0-2:5- \
|
|
|
- -jobconf mapred.reduce.tasks=12
|
|
|
+ -D map.output.key.field.separa=. \
|
|
|
+ -D num.key.fields.for.partition=2 \
|
|
|
+ -D mapred.data.field.separator=. \
|
|
|
+ -D map.output.key.value.fields.spec=6,5,1-3:0- \
|
|
|
+ -D reduce.output.key.value.fields.spec=0-2:5- \
|
|
|
+ -D mapred.reduce.tasks=12
|
|
|
</pre>
|
|
|
<p>
|
|
|
-The option "-jobconf map.output.key.value.fields.spec=6,5,1-3:0-" specifies key/value selection for the map outputs. Key selection spec and value selection spec are separated by ":". In this case, the map output key will consist of fields 6, 5, 1, 2, and 3. The map output value will consist of all fields (0- means field 0 and all
|
|
|
+The option "-D map.output.key.value.fields.spec=6,5,1-3:0-" specifies key/value selection for the map outputs. Key selection spec and value selection spec are separated by ":". In this case, the map output key will consist of fields 6, 5, 1, 2, and 3. The map output value will consist of all fields (0- means field 0 and all
|
|
|
the subsequent fields).
|
|
|
</p>
|
|
|
<p>
|
|
|
-The option "-jobconf reduce.output.key.value.fields.spec=0-2:0-" specifies key/value selection for the reduce outputs. In this case, the reduce output key will consist of fields 0, 1, 2 (corresponding to the original fields 6, 5, 1). The reduce output value will consist of all fields starting from field 5 (corresponding to all the original fields).
|
|
|
+The option "-D reduce.output.key.value.fields.spec=0-2:0-" specifies key/value selection for the reduce outputs. In this case, the reduce output key will consist of fields 0, 1, 2 (corresponding to the original fields 6, 5, 1). The reduce output value will consist of all fields starting from field 5 (corresponding to all the original fields).
|
|
|
</p>
|
|
|
</div>
|
|
|
|
|
|
|
|
|
-<a name="N1020F"></a><a name="Frequently+Asked+Questions"></a>
|
|
|
+<a name="N1023E"></a><a name="Frequently+Asked+Questions"></a>
|
|
|
<h2 class="h3">Frequently Asked Questions </h2>
|
|
|
<div class="section">
|
|
|
-<a name="N10215"></a><a name="How+do+I+use+Hadoop+Streaming+to+run+an+arbitrary+set+of+%28semi-%29independent+tasks%3F"></a>
|
|
|
+<a name="N10244"></a><a name="How+do+I+use+Hadoop+Streaming+to+run+an+arbitrary+set+of+%28semi-%29independent+tasks%3F"></a>
|
|
|
<h3 class="h4">How do I use Hadoop Streaming to run an arbitrary set of (semi-)independent tasks? </h3>
|
|
|
<p>
|
|
|
Often you do not need the full power of Map Reduce, but only need to run multiple instances of the same program - either on different parts of the data, or on the same data, but with different parameters. You can use Hadoop Streaming to do this.
|
|
|
</p>
|
|
|
-<a name="N1021F"></a><a name="How+do+I+process+files%2C+one+per+map%3F"></a>
|
|
|
+<a name="N1024E"></a><a name="How+do+I+process+files%2C+one+per+map%3F"></a>
|
|
|
<h3 class="h4">How do I process files, one per map? </h3>
|
|
|
<p>
|
|
|
As an example, consider the problem of zipping (compressing) a set of files across the hadoop cluster. You can achieve this using either of these methods:
|
|
|
@@ -784,13 +802,13 @@ As an example, consider the problem of zipping (compressing) a set of files acro
|
|
|
</li>
|
|
|
|
|
|
</ol>
|
|
|
-<a name="N1024A"></a><a name="How+many+reducers+should+I+use%3F"></a>
|
|
|
+<a name="N10279"></a><a name="How+many+reducers+should+I+use%3F"></a>
|
|
|
<h3 class="h4">How many reducers should I use? </h3>
|
|
|
<p>
|
|
|
See the Hadoop Wiki for details: <a href="mapred_tutorial.html#Reducer">Reducer</a>
|
|
|
|
|
|
</p>
|
|
|
-<a name="N10258"></a><a name="If+I+set+up+an+alias+in+my+shell+script%2C+will+that+work+after+-mapper%2C+i.e.+say+I+do%3A+alias+c1%3D%27cut+-f1%27.+Will+-mapper+%22c1%22+work%3F"></a>
|
|
|
+<a name="N10287"></a><a name="If+I+set+up+an+alias+in+my+shell+script%2C+will+that+work+after+-mapper%2C+i.e.+say+I+do%3A+alias+c1%3D%27cut+-f1%27.+Will+-mapper+%22c1%22+work%3F"></a>
|
|
|
<h3 class="h4">If I set up an alias in my shell script, will that work after -mapper, i.e. say I do: alias c1='cut -f1'. Will -mapper "c1" work? </h3>
|
|
|
<p>
|
|
|
Using an alias will not work, but variable substitution is allowed as shown in this example:
|
|
|
@@ -806,7 +824,7 @@ $ c2='cut -f2'; $HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/hadoop-streaming.jar \
|
|
|
-input /user/me/samples/student_marks
|
|
|
-mapper \"$c2\" -reducer 'cat'
|
|
|
-output /user/me/samples/student_out
|
|
|
- -jobconf mapred.job.name='Experiment'
|
|
|
+ -D mapred.job.name='Experiment'
|
|
|
|
|
|
$ hadoop dfs -ls samples/student_out
|
|
|
Found 1 items/user/me/samples/student_out/part-00000 <r 3> 16
|
|
|
@@ -817,20 +835,20 @@ $ hadoop dfs -cat samples/student_out/part-00000
|
|
|
75
|
|
|
80
|
|
|
</pre>
|
|
|
-<a name="N10266"></a><a name="Can+I+use+UNIX+pipes%3F+For+example%2C+will+-mapper+%22cut+-f1+%7C+sed+s%2Ffoo%2Fbar%2Fg%22+work%3F"></a>
|
|
|
+<a name="N10295"></a><a name="Can+I+use+UNIX+pipes%3F+For+example%2C+will+-mapper+%22cut+-f1+%7C+sed+s%2Ffoo%2Fbar%2Fg%22+work%3F"></a>
|
|
|
<h3 class="h4">Can I use UNIX pipes? For example, will -mapper "cut -f1 | sed s/foo/bar/g" work?</h3>
|
|
|
<p>
|
|
|
Currently this does not work and gives an "java.io.IOException: Broken pipe" error. This is probably a bug that needs to be investigated.
|
|
|
</p>
|
|
|
-<a name="N10270"></a><a name="When+I+run+a+streaming+job+by"></a>
|
|
|
+<a name="N1029F"></a><a name="When+I+run+a+streaming+job+by"></a>
|
|
|
<h3 class="h4">When I run a streaming job by distributing large executables (for example, 3.6G) through the -file option, I get a "No space left on device" error. What do I do? </h3>
|
|
|
<p>
|
|
|
The jar packaging happens in a directory pointed to by the configuration variable stream.tmpdir. The default value of stream.tmpdir is /tmp. Set the value to a directory with more space:
|
|
|
</p>
|
|
|
<pre class="code">
|
|
|
--jobconf stream.tmpdir=/export/bigspace/...
|
|
|
+-D stream.tmpdir=/export/bigspace/...
|
|
|
</pre>
|
|
|
-<a name="N10281"></a><a name="How+do+I+specify+multiple+input+directories%3F"></a>
|
|
|
+<a name="N102B0"></a><a name="How+do+I+specify+multiple+input+directories%3F"></a>
|
|
|
<h3 class="h4">How do I specify multiple input directories? </h3>
|
|
|
<p>
|
|
|
You can specify multiple input directories with multiple '-input' options:
|
|
|
@@ -838,17 +856,17 @@ You can specify multiple input directories with multiple '-input' options:
|
|
|
<pre class="code">
|
|
|
hadoop jar hadoop-streaming.jar -input '/user/foo/dir1' -input '/user/foo/dir2'
|
|
|
</pre>
|
|
|
-<a name="N1028E"></a><a name="How+do+I+generate+output+files+with+gzip+format%3F"></a>
|
|
|
+<a name="N102BD"></a><a name="How+do+I+generate+output+files+with+gzip+format%3F"></a>
|
|
|
<h3 class="h4">How do I generate output files with gzip format? </h3>
|
|
|
<p>
|
|
|
-Instead of plain text files, you can generate gzip files as your generated output. Pass '-jobconf mapred.output.compress=true -jobconf mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCode' as option to your streaming job.
|
|
|
+Instead of plain text files, you can generate gzip files as your generated output. Pass '-D mapred.output.compress=true -D mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCode' as option to your streaming job.
|
|
|
</p>
|
|
|
-<a name="N10298"></a><a name="How+do+I+provide+my+own+input%2Foutput+format+with+streaming%3F"></a>
|
|
|
+<a name="N102C7"></a><a name="How+do+I+provide+my+own+input%2Foutput+format+with+streaming%3F"></a>
|
|
|
<h3 class="h4">How do I provide my own input/output format with streaming? </h3>
|
|
|
<p>
|
|
|
At least as late as version 0.14, Hadoop does not support multiple jar files. So, when specifying your own custom classes you will have to pack them along with the streaming jar and use the custom jar instead of the default hadoop streaming jar.
|
|
|
</p>
|
|
|
-<a name="N102A2"></a><a name="How+do+I+parse+XML+documents+using+streaming%3F"></a>
|
|
|
+<a name="N102D1"></a><a name="How+do+I+parse+XML+documents+using+streaming%3F"></a>
|
|
|
<h3 class="h4">How do I parse XML documents using streaming? </h3>
|
|
|
<p>
|
|
|
You can use the record reader StreamXmlRecordReader to process XML documents.
|
|
|
@@ -859,14 +877,14 @@ hadoop jar hadoop-streaming.jar -inputreader "StreamXmlRecord,begin=BEGIN_STRING
|
|
|
<p>
|
|
|
Anything found between BEGIN_STRING and END_STRING would be treated as one record for map tasks.
|
|
|
</p>
|
|
|
-<a name="N102B3"></a><a name="How+do+I+update+counters+in+streaming+applications%3F"></a>
|
|
|
+<a name="N102E2"></a><a name="How+do+I+update+counters+in+streaming+applications%3F"></a>
|
|
|
<h3 class="h4">How do I update counters in streaming applications? </h3>
|
|
|
<p>
|
|
|
A streaming process can use the stderr to emit counter information.
|
|
|
<span class="codefrag">reporter:counter:<group>,<counter>,<amount></span>
|
|
|
should be sent to stderr to update the counter.
|
|
|
</p>
|
|
|
-<a name="N102C0"></a><a name="How+do+I+update+status+in+streaming+applications%3F"></a>
|
|
|
+<a name="N102EF"></a><a name="How+do+I+update+status+in+streaming+applications%3F"></a>
|
|
|
<h3 class="h4">How do I update status in streaming applications? </h3>
|
|
|
<p>
|
|
|
A streaming process can use the stderr to emit status information.
|
Arun Murthy