|
@@ -289,7 +289,7 @@ document.write("Last Published: " + document.lastModified);
|
|
|
<a href="#Example%3A+WordCount+v2.0">Example: WordCount v2.0</a>
|
|
|
<ul class="minitoc">
|
|
|
<li>
|
|
|
-<a href="#Source+Code-N10C11">Source Code</a>
|
|
|
+<a href="#Source+Code-N10C14">Source Code</a>
|
|
|
</li>
|
|
|
<li>
|
|
|
<a href="#Sample+Runs">Sample Runs</a>
|
|
@@ -1798,11 +1798,15 @@ document.write("Last Published: " + document.lastModified);
|
|
|
<p>The application-writer can take advantage of this feature by
|
|
|
creating any side-files required in <span class="codefrag">${mapred.output.dir}</span>
|
|
|
during execution of a task via
|
|
|
- <a href="api/org/apache/hadoop/mapred/JobConf.html#getOutputPath()">
|
|
|
- JobConf.getOutputPath()</a>, and the framework will promote them
|
|
|
+ <a href="api/org/apache/hadoop/mapred/JobConf.html#getCurrentOutputPath()">
|
|
|
+ JobConf.getCurrentOutputPath()</a>, and the framework will promote them
|
|
|
similarly for succesful task-attempts, thus eliminating the need to
|
|
|
- pick unique paths per task-attempt.</p>
|
|
|
-<a name="N10A84"></a><a name="RecordWriter"></a>
|
|
|
+ pick unique paths per task-attempt. she can get the actual configured
|
|
|
+ path (final output path) via
|
|
|
+ <a href="api/org/apache/hadoop/mapred/JobConf.html#getFinalOutputPath()">
|
|
|
+ JobConf.getFinalOutputPath()</a>
|
|
|
+</p>
|
|
|
+<a name="N10A87"></a><a name="RecordWriter"></a>
|
|
|
<h4>RecordWriter</h4>
|
|
|
<p>
|
|
|
<a href="api/org/apache/hadoop/mapred/RecordWriter.html">
|
|
@@ -1810,9 +1814,9 @@ document.write("Last Published: " + document.lastModified);
|
|
|
pairs to an output file.</p>
|
|
|
<p>RecordWriter implementations write the job outputs to the
|
|
|
<span class="codefrag">FileSystem</span>.</p>
|
|
|
-<a name="N10A9B"></a><a name="Other+Useful+Features"></a>
|
|
|
+<a name="N10A9E"></a><a name="Other+Useful+Features"></a>
|
|
|
<h3 class="h4">Other Useful Features</h3>
|
|
|
-<a name="N10AA1"></a><a name="Counters"></a>
|
|
|
+<a name="N10AA4"></a><a name="Counters"></a>
|
|
|
<h4>Counters</h4>
|
|
|
<p>
|
|
|
<span class="codefrag">Counters</span> represent global counters, defined either by
|
|
@@ -1826,7 +1830,7 @@ document.write("Last Published: " + document.lastModified);
|
|
|
Reporter.incrCounter(Enum, long)</a> in the <span class="codefrag">map</span> and/or
|
|
|
<span class="codefrag">reduce</span> methods. These counters are then globally
|
|
|
aggregated by the framework.</p>
|
|
|
-<a name="N10ACC"></a><a name="DistributedCache"></a>
|
|
|
+<a name="N10ACF"></a><a name="DistributedCache"></a>
|
|
|
<h4>DistributedCache</h4>
|
|
|
<p>
|
|
|
<a href="api/org/apache/hadoop/filecache/DistributedCache.html">
|
|
@@ -1859,7 +1863,7 @@ document.write("Last Published: " + document.lastModified);
|
|
|
<a href="api/org/apache/hadoop/filecache/DistributedCache.html#createSymlink(org.apache.hadoop.conf.Configuration)">
|
|
|
DistributedCache.createSymlink(Path, Configuration)</a> api. Files
|
|
|
have <em>execution permissions</em> set.</p>
|
|
|
-<a name="N10B0A"></a><a name="Tool"></a>
|
|
|
+<a name="N10B0D"></a><a name="Tool"></a>
|
|
|
<h4>Tool</h4>
|
|
|
<p>The <a href="api/org/apache/hadoop/util/Tool.html">Tool</a>
|
|
|
interface supports the handling of generic Hadoop command-line options.
|
|
@@ -1899,7 +1903,7 @@ document.write("Last Published: " + document.lastModified);
|
|
|
</span>
|
|
|
|
|
|
</p>
|
|
|
-<a name="N10B3C"></a><a name="IsolationRunner"></a>
|
|
|
+<a name="N10B3F"></a><a name="IsolationRunner"></a>
|
|
|
<h4>IsolationRunner</h4>
|
|
|
<p>
|
|
|
<a href="api/org/apache/hadoop/mapred/IsolationRunner.html">
|
|
@@ -1923,13 +1927,13 @@ document.write("Last Published: " + document.lastModified);
|
|
|
<p>
|
|
|
<span class="codefrag">IsolationRunner</span> will run the failed task in a single
|
|
|
jvm, which can be in the debugger, over precisely the same input.</p>
|
|
|
-<a name="N10B6F"></a><a name="JobControl"></a>
|
|
|
+<a name="N10B72"></a><a name="JobControl"></a>
|
|
|
<h4>JobControl</h4>
|
|
|
<p>
|
|
|
<a href="api/org/apache/hadoop/mapred/jobcontrol/package-summary.html">
|
|
|
JobControl</a> is a utility which encapsulates a set of Map-Reduce jobs
|
|
|
and their dependencies.</p>
|
|
|
-<a name="N10B7C"></a><a name="Data+Compression"></a>
|
|
|
+<a name="N10B7F"></a><a name="Data+Compression"></a>
|
|
|
<h4>Data Compression</h4>
|
|
|
<p>Hadoop Map-Reduce provides facilities for the application-writer to
|
|
|
specify compression for both intermediate map-outputs and the
|
|
@@ -1943,7 +1947,7 @@ document.write("Last Published: " + document.lastModified);
|
|
|
codecs for reasons of both performance (zlib) and non-availability of
|
|
|
Java libraries (lzo). More details on their usage and availability are
|
|
|
available <a href="native_libraries.html">here</a>.</p>
|
|
|
-<a name="N10B9C"></a><a name="Intermediate+Outputs"></a>
|
|
|
+<a name="N10B9F"></a><a name="Intermediate+Outputs"></a>
|
|
|
<h5>Intermediate Outputs</h5>
|
|
|
<p>Applications can control compression of intermediate map-outputs
|
|
|
via the
|
|
@@ -1964,7 +1968,7 @@ document.write("Last Published: " + document.lastModified);
|
|
|
<a href="api/org/apache/hadoop/mapred/JobConf.html#setMapOutputCompressionType(org.apache.hadoop.io.SequenceFile.CompressionType)">
|
|
|
JobConf.setMapOutputCompressionType(SequenceFile.CompressionType)</a>
|
|
|
api.</p>
|
|
|
-<a name="N10BC8"></a><a name="Job+Outputs"></a>
|
|
|
+<a name="N10BCB"></a><a name="Job+Outputs"></a>
|
|
|
<h5>Job Outputs</h5>
|
|
|
<p>Applications can control compression of job-outputs via the
|
|
|
<a href="api/org/apache/hadoop/mapred/OutputFormatBase.html#setCompressOutput(org.apache.hadoop.mapred.JobConf,%20boolean)">
|
|
@@ -1984,7 +1988,7 @@ document.write("Last Published: " + document.lastModified);
|
|
|
</div>
|
|
|
|
|
|
|
|
|
-<a name="N10BF7"></a><a name="Example%3A+WordCount+v2.0"></a>
|
|
|
+<a name="N10BFA"></a><a name="Example%3A+WordCount+v2.0"></a>
|
|
|
<h2 class="h3">Example: WordCount v2.0</h2>
|
|
|
<div class="section">
|
|
|
<p>Here is a more complete <span class="codefrag">WordCount</span> which uses many of the
|
|
@@ -1994,7 +1998,7 @@ document.write("Last Published: " + document.lastModified);
|
|
|
<a href="quickstart.html#SingleNodeSetup">pseudo-distributed</a> or
|
|
|
<a href="quickstart.html#Fully-Distributed+Operation">fully-distributed</a>
|
|
|
Hadoop installation.</p>
|
|
|
-<a name="N10C11"></a><a name="Source+Code-N10C11"></a>
|
|
|
+<a name="N10C14"></a><a name="Source+Code-N10C14"></a>
|
|
|
<h3 class="h4">Source Code</h3>
|
|
|
<table class="ForrestTable" cellspacing="1" cellpadding="4">
|
|
|
|
|
@@ -3204,7 +3208,7 @@ document.write("Last Published: " + document.lastModified);
|
|
|
</tr>
|
|
|
|
|
|
</table>
|
|
|
-<a name="N11373"></a><a name="Sample+Runs"></a>
|
|
|
+<a name="N11376"></a><a name="Sample+Runs"></a>
|
|
|
<h3 class="h4">Sample Runs</h3>
|
|
|
<p>Sample text-files as input:</p>
|
|
|
<p>
|
|
@@ -3372,7 +3376,7 @@ document.write("Last Published: " + document.lastModified);
|
|
|
<br>
|
|
|
|
|
|
</p>
|
|
|
-<a name="N11447"></a><a name="Highlights"></a>
|
|
|
+<a name="N1144A"></a><a name="Highlights"></a>
|
|
|
<h3 class="h4">Highlights</h3>
|
|
|
<p>The second version of <span class="codefrag">WordCount</span> improves upon the
|
|
|
previous one by using some features offered by the Map-Reduce framework:
|