HADOOP-3942. Update distcp documentation to include features introduced in

HADOOP-3873, HADOOP-3939. Contributed by Tsz Wo (Nicholas), SZE.


git-svn-id: https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.19@700998 13f79535-47bb-0310-9956-ffa450edef68

CHANGES.txt

@@ -778,6 +778,9 @@ Release 0.19.0 - Unreleased
     HADOOP-4302. Fix a race condition in TestReduceFetch that can yield false
     negatvies. (cdouglas)
 
+    HADOOP-3942. Update distcp documentation to include features introduced in
+    HADOOP-3873, HADOOP-3939. (Tsz Wo (Nicholas), SZE via cdouglas)
+
 Release 0.18.2 - Unreleased
 
   BUG FIXES

docs/SLG_user_guide.html

@@ -156,6 +156,9 @@ document.write("Last Published: " + document.lastModified);
 <a href="api/index.html">API Docs</a>
 </div>
 <div class="menuitem">
+<a href="jdiff/changes.html">API Changes</a>
+</div>
+<div class="menuitem">
 <a href="http://wiki.apache.org/hadoop/">Wiki</a>
 </div>
 <div class="menuitem">

docs/cluster_setup.html

@@ -156,6 +156,9 @@ document.write("Last Published: " + document.lastModified);
 <a href="api/index.html">API Docs</a>
 </div>
 <div class="menuitem">
+<a href="jdiff/changes.html">API Changes</a>
+</div>
+<div class="menuitem">
 <a href="http://wiki.apache.org/hadoop/">Wiki</a>
 </div>
 <div class="menuitem">

docs/commands_manual.html

@@ -156,6 +156,9 @@ document.write("Last Published: " + document.lastModified);
 <a href="api/index.html">API Docs</a>
 </div>
 <div class="menuitem">
+<a href="jdiff/changes.html">API Changes</a>
+</div>
+<div class="menuitem">
 <a href="http://wiki.apache.org/hadoop/">Wiki</a>
 </div>
 <div class="menuitem">

docs/distcp.html

@@ -156,6 +156,9 @@ document.write("Last Published: " + document.lastModified);
 <a href="api/index.html">API Docs</a>
 </div>
 <div class="menuitem">
+<a href="jdiff/changes.html">API Changes</a>
+</div>
+<div class="menuitem">
 <a href="http://wiki.apache.org/hadoop/">Wiki</a>
 </div>
 <div class="menuitem">
@@ -209,6 +212,9 @@ document.write("Last Published: " + document.lastModified);
 <a href="#Option+Index">Option Index</a>
 </li>
 <li>
+<a href="#Symbolic-Representations">Symbolic Representations</a>
+</li>
+<li>
 <a href="#uo">Update and Overwrite</a>
 </li>
 </ul>
@@ -408,10 +414,48 @@ document.write("Last Published: " + document.lastModified);
               qualified URI.
               </td>
 </tr>
+          
+<tr>
+<td colspan="1" rowspan="1"><span class="codefrag">-filelimit &lt;n&gt;</span></td>
+              <td colspan="1" rowspan="1">Limit the total number of files to be &lt;= n</td>
+              <td colspan="1" rowspan="1">See also <a href="#Symbolic-Representations">Symbolic
+                  Representations</a>.
+              </td>
+</tr>
+          
+<tr>
+<td colspan="1" rowspan="1"><span class="codefrag">-sizelimit &lt;n&gt;</span></td>
+              <td colspan="1" rowspan="1">Limit the total size to be &lt;= n bytes</td>
+              <td colspan="1" rowspan="1">See also <a href="#Symbolic-Representations">Symbolic
+                  Representations</a>.
+              </td>
+</tr>
+          
+<tr>
+<td colspan="1" rowspan="1"><span class="codefrag">-delete</span></td>
+              <td colspan="1" rowspan="1">Delete the files existing in the dst but not in src</td>
+              <td colspan="1" rowspan="1">The deletion is done by FS Shell.  So the trash will be used,
+                  if it is enable.
+              </td>
+</tr>
 
         
 </table>
-<a name="N10136"></a><a name="uo"></a>
+<a name="N10171"></a><a name="Symbolic-Representations"></a>
+<h4>Symbolic Representations</h4>
+<p>
+        The parameter &lt;n&gt; in <span class="codefrag">-filelimit</span>
+        and <span class="codefrag">-sizelimit</span> can be specified with symbolic
+        representation.  For examples,
+        </p>
+<ul>
+          
+<li>1230k = 1230 * 1024 = 1259520</li>
+          
+<li>891g = 891 * 1024^3 = 956703965184</li>
+        
+</ul>
+<a name="N1018A"></a><a name="uo"></a>
 <h4>Update and Overwrite</h4>
 <p>It's worth giving some examples of <span class="codefrag">-update</span> and
         <span class="codefrag">-overwrite</span>. Consider a copy from <span class="codefrag">/foo/a</span> and
@@ -510,10 +554,10 @@ document.write("Last Published: " + document.lastModified);
 </div> <!-- Usage -->
 
     
-<a name="N101E7"></a><a name="etc"></a>
+<a name="N1023B"></a><a name="etc"></a>
 <h2 class="h3">Appendix</h2>
 <div class="section">
-<a name="N101ED"></a><a name="Map+sizing"></a>
+<a name="N10241"></a><a name="Map+sizing"></a>
 <h3 class="h4">Map sizing</h3>
 <p>DistCp makes a faint attempt to size each map comparably so that
           each copies roughly the same number of bytes. Note that files are the
@@ -527,7 +571,7 @@ document.write("Last Published: " + document.lastModified);
 <p>Tuning the number of maps to the size of the source and
           destination clusters, the size of the copy, and the available
           bandwidth is recommended for long-running and regularly run jobs.</p>
-<a name="N10206"></a><a name="cpver"></a>
+<a name="N1025A"></a><a name="cpver"></a>
 <h3 class="h4">Copying between versions of HDFS</h3>
 <p>For copying between two different versions of Hadoop, one will
         usually use HftpFileSystem. This is a read-only FileSystem, so DistCp
@@ -536,7 +580,7 @@ document.write("Last Published: " + document.lastModified);
         specified as <span class="codefrag">hftp://&lt;dfs.http.address&gt;/&lt;path&gt;</span>
         (the default <span class="codefrag">dfs.http.address</span> is
         &lt;namenode&gt;:50070).</p>
-<a name="N10216"></a><a name="Map%2FReduce+and+other+side-effects"></a>
+<a name="N1026A"></a><a name="Map%2FReduce+and+other+side-effects"></a>
 <h3 class="h4">Map/Reduce and other side-effects</h3>
 <p>As has been mentioned in the preceding, should a map fail to copy
         one of its inputs, there will be several side-effects.</p>

docs/hadoop_archives.html

@@ -156,6 +156,9 @@ document.write("Last Published: " + document.lastModified);
 <a href="api/index.html">API Docs</a>
 </div>
 <div class="menuitem">
+<a href="jdiff/changes.html">API Changes</a>
+</div>
+<div class="menuitem">
 <a href="http://wiki.apache.org/hadoop/">Wiki</a>
 </div>
 <div class="menuitem">

docs/hdfs_design.html

@@ -158,6 +158,9 @@ document.write("Last Published: " + document.lastModified);
 <a href="api/index.html">API Docs</a>
 </div>
 <div class="menuitem">
+<a href="jdiff/changes.html">API Changes</a>
+</div>
+<div class="menuitem">
 <a href="http://wiki.apache.org/hadoop/">Wiki</a>
 </div>
 <div class="menuitem">

docs/hdfs_permissions_guide.html

@@ -158,6 +158,9 @@ document.write("Last Published: " + document.lastModified);
 <a href="api/index.html">API Docs</a>
 </div>
 <div class="menuitem">
+<a href="jdiff/changes.html">API Changes</a>
+</div>
+<div class="menuitem">
 <a href="http://wiki.apache.org/hadoop/">Wiki</a>
 </div>
 <div class="menuitem">

docs/hdfs_quota_admin_guide.html

@@ -156,6 +156,9 @@ document.write("Last Published: " + document.lastModified);
 <a href="api/index.html">API Docs</a>
 </div>
 <div class="menuitem">
+<a href="jdiff/changes.html">API Changes</a>
+</div>
+<div class="menuitem">
 <a href="http://wiki.apache.org/hadoop/">Wiki</a>
 </div>
 <div class="menuitem">

docs/hdfs_shell.html

@@ -156,6 +156,9 @@ document.write("Last Published: " + document.lastModified);
 <a href="api/index.html">API Docs</a>
 </div>
 <div class="menuitem">
+<a href="jdiff/changes.html">API Changes</a>
+</div>
+<div class="menuitem">
 <a href="http://wiki.apache.org/hadoop/">Wiki</a>
 </div>
 <div class="menuitem">

docs/hdfs_user_guide.html

@@ -158,6 +158,9 @@ document.write("Last Published: " + document.lastModified);
 <a href="api/index.html">API Docs</a>
 </div>
 <div class="menuitem">
+<a href="jdiff/changes.html">API Changes</a>
+</div>
+<div class="menuitem">
 <a href="http://wiki.apache.org/hadoop/">Wiki</a>
 </div>
 <div class="menuitem">

docs/hod.html

@@ -158,6 +158,9 @@ document.write("Last Published: " + document.lastModified);
 <a href="api/index.html">API Docs</a>
 </div>
 <div class="menuitem">
+<a href="jdiff/changes.html">API Changes</a>
+</div>
+<div class="menuitem">
 <a href="http://wiki.apache.org/hadoop/">Wiki</a>
 </div>
 <div class="menuitem">

docs/hod_admin_guide.html

@@ -158,6 +158,9 @@ document.write("Last Published: " + document.lastModified);
 <a href="api/index.html">API Docs</a>
 </div>
 <div class="menuitem">
+<a href="jdiff/changes.html">API Changes</a>
+</div>
+<div class="menuitem">
 <a href="http://wiki.apache.org/hadoop/">Wiki</a>
 </div>
 <div class="menuitem">

docs/hod_config_guide.html

@@ -158,6 +158,9 @@ document.write("Last Published: " + document.lastModified);
 <a href="api/index.html">API Docs</a>
 </div>
 <div class="menuitem">
+<a href="jdiff/changes.html">API Changes</a>
+</div>
+<div class="menuitem">
 <a href="http://wiki.apache.org/hadoop/">Wiki</a>
 </div>
 <div class="menuitem">

docs/hod_user_guide.html

@@ -158,6 +158,9 @@ document.write("Last Published: " + document.lastModified);
 <a href="api/index.html">API Docs</a>
 </div>
 <div class="menuitem">
+<a href="jdiff/changes.html">API Changes</a>
+</div>
+<div class="menuitem">
 <a href="http://wiki.apache.org/hadoop/">Wiki</a>
 </div>
 <div class="menuitem">

docs/index.html

@@ -156,6 +156,9 @@ document.write("Last Published: " + document.lastModified);
 <a href="api/index.html">API Docs</a>
 </div>
 <div class="menuitem">
+<a href="jdiff/changes.html">API Changes</a>
+</div>
+<div class="menuitem">
 <a href="http://wiki.apache.org/hadoop/">Wiki</a>
 </div>
 <div class="menuitem">

docs/linkmap.html

@@ -156,6 +156,9 @@ document.write("Last Published: " + document.lastModified);
 <a href="api/index.html">API Docs</a>
 </div>
 <div class="menuitem">
+<a href="jdiff/changes.html">API Changes</a>
+</div>
+<div class="menuitem">
 <a href="http://wiki.apache.org/hadoop/">Wiki</a>
 </div>
 <div class="menuitem">
@@ -315,6 +318,12 @@ document.write("Last Published: " + document.lastModified);
 </li>
 </ul>
     
+<ul>
+<li>
+<a href="jdiff/changes.html">API Changes</a>&nbsp;&nbsp;___________________&nbsp;&nbsp;<em>jdiff</em>
+</li>
+</ul>
+    
 <ul>
 <li>
 <a href="http://wiki.apache.org/hadoop/">Wiki</a>&nbsp;&nbsp;___________________&nbsp;&nbsp;<em>wiki</em>

docs/linkmap.pdf

@@ -5,10 +5,10 @@
 /Producer (FOP 0.20.5) >>
 endobj
 5 0 obj
-<< /Length 1192 /Filter [ /ASCII85Decode /FlateDecode ]
+<< /Length 1185 /Filter [ /ASCII85Decode /FlateDecode ]
  >>
 stream
-Gatn'9lJcG&;KZQ'fr4IE;?JoURQ)R>r6CsHa/@n@_kZY9LKL6M/psm^Qb*hU^L4Xli]jM$C^_JE/#A&m<\<W*JlHR0NrB1YlC%C@.^ro&N%6Nb5/B169sNe>$'*9="C8hS*1b<WHiOW6.SgHhi._RriL:6X3K;2;7.1IDVmk^7dE9u15c^Pb+9SG&eIEFjjmYhp*NoAjN;?2Z2%""4''e<oi@$B>ZJC6BP4-+DNED'USJp0<]=<e]1.;Q3sr3u$tSTQqcAa00.)TbmhQ[+5_%#O4YN&bkJH."r85gb#D;PB#'#5[Q=8[HSRWK<a#sPo!(.:o)Bh3<>EsDt=Z<L[Wop)g8h?D]<2f^r_1:NoJ78%.JG*\1R_9CgZ`+cB"NA*03Z+D3K,CBbZA23;:T`c=T$PEXa'=mBa6WpL:!&bk7s$\$f4r]l!PT2XRiQ[jKO+%0:U.;CV]]J17d:THqqJa58@WL,Xbl*^$YP*()2N"lX)<j?F:_?mK)+AHrmeT/IFlU#$sX-K(Am*hiV#ik'(0D[<3g@8B"&SNVSp+M;kN#*C)@3IaesUOPi`JFZD_:cbq@8K\s2r'&d*_aTB`4tO>I8VSuO8]c8b1b]^C#)S'nr;#kg.IK?+a@S-IHgp%N2Ob48G-SRd7Il>*$O<LO(iE:pn&-bYiAaS8&@k;7Z9C2jq-(JIeCS=c3;oajoedZ>r0;qH]oc0f%)a!ODf?a)3kq=GFs&.E144l9Op&Oi$J<ReT9DO0HRFMCCG?e6!:N@6,VEJB(5pX.JDH\_2hCAeDTq5rV8'^]^UbQ'BW*lP&9Aq<:*\C-ApAq,.iVip9<#Z%6:X\Q-h+t7d1*BA+!V>'S085'Ui&GA!W_<;R^U#BMMM"FV/[VnTGC]Be'b:L!#^/ZO33hjI7>(6Igi>_(C\VlN(1=?ir4#olf1)A*1^ZCZL%Fji4^@di/g_>f0O+V)qo&37^fD((b:XcNnQk]:"Q8W"Z>>)$[044j:o?pPgXi2_[Op:O:M^SaISc/&QdV%WX6AH(#:U(LCD2f%orOuT#%sJE!LGFGf86A7bE<n7V)A-g%%u[pqSHLjs_<[X[MA+.&.ga^T(r\&K#[iKh\l$*.NI9OHnZq+(An^O>;is+b'O0aT@_(O0gL$q>Ic68BEi6L[(qtHMo9q7eChM+4S\?es`Ej6dSGN7`',dn~>
+Gatn'9lnc;&;KZL'g@pNL%T0W(fJMs:nT'B#@JBW/%'&,Vm`**ruTiCYnTYej.r5[-$d/lUXcNSer/)Fq)&Wn!F`39cbf*W3X^SP)%8k1$\P5o@jD2E`PJP[^0W@e'$l/&'ZKpj`B.5Z(OS:QXSC,$ot4H=m6t#);6&b]n3'mee]K_!a=',j1l;8@M32$(ZKY'0m`C1=k!S`s^iYO2BrbJIS[9lT)kj?q7bU"3AsSUcU`*7BWG62Lp?^/D:(r`-:[HsMUApW4$p1pp('=&N[]asW<S^nnfF.`SpLeE+rF6C-&0qR&9q"*=+V<r^#QuPtYV..VPJ=-G[a=?[Q)r--BUE1[Ad`0MqS-EgX;dg's7N%5fSOo%`T(JYM?/IUeYr12a2gRM@Lq.//(=AbnQJY"UW-fPb(F8KKT<uqaJ&dJ`MRL:W.t@[$Iql]406?&o:D`0(lMmcN!thp!;ZKVH8uP^lSR4Ugh5SVcm/XN`3*jHXiH'U&F.LNQh,9.qt3r>ZP.9a@$J4]a5M25I2)*>;&?NPFCmIEU*a!W'ZMcCERZ0'k*&GdgK'29_"4XhV:<+6=4Ojo:EfusMI!6Wk(2R6T6XF6k]=aQ^YrbI2j5]`9,JmT1YmtJOg<W/I^["X\_(`D("H._0HmUm0jZ/_2[,P_97>5=R1>2E.W1g>Y='fLK&_fH9!n1'SZ!%X0iR@9`H1>0E*dA,k&[>nKXl]&<_19?BFI7H<o3?5jX61uWDuBh>4#pI8jmlm1i[Y\JQIbGPSGLkRsFIFrN`ahpSHHuJ<<'3?;&[>#"KA8/abkD$a(;eQ\\:7$a065]2L29DNAa)R_e8+RkBl:BA\7Rqbqf"h1o)Kd24'/POFKkO^d<7p'sVXPXG=?'W1Sr(pMLDgZj',`1fWYc)g`]2\ob--Vn#qe<c;X(uNJ53g``^/ie$3S<nFI/322Tc<RneR9Zr9$2"Vl9:@O)?;;j4G"L<^@rH7DLIU,g\Xq]MLgqB2o7q<(*.*n.L<GS0;"gAcUSb5&'_WocUUW#oqj1WlAe4jB6N;lN_[&Xe+MDe$&.N@dXCo#K(Yrp2da,bYD+'=,6ESE:fR4Ae)C-0HaN,p4mI+68j_D;;pkC1:H:ZY^f'PkCh+c,&4LP(0C98P&D2he%D41(S]tes=PX;7rCH#:$kd"YF1,)E#%jZfDir~>
 endstream
 endobj
 6 0 obj
@@ -20,10 +20,10 @@ endobj
 >>
 endobj
 7 0 obj
-<< /Length 388 /Filter [ /ASCII85Decode /FlateDecode ]
+<< /Length 433 /Filter [ /ASCII85Decode /FlateDecode ]
  >>
 stream
-Gar?-5uWCi&;BTNMESBQA$TX/E,7W0i-<Tt2sYQp)^o2(Q(j<=h>jY\L"#d(p[-_+VpXmp5To*KAeBe\'#=Gr5c@^_UmU-SdN^kMeqc/:)^1%7:QsJji!Fa4ltr0&ZPSFA_WLkSKbm;K]qbFH]Xl(>Qr7tk\%VLJPT^kj_^^Ma%*"r/bQK8@eZ*:f4."MLd0a=Q:m>lgHQ$UtI6EPhWj33hWj6VUF#C^*U\AXh8K1EDbqt"<U2u%l[=^#VMWX:i`SWDlNu+9&'$6*H9_iAnYs/FI"i&j1VpggGWt;O(&K'a7Mn[`93doUK^2LT3VHs0>8IU9BRt9#$)dmipgQ5K:epQ.3:mTX%\[,"-Pq4?L:Z"P&I1Fo#?2a#2CtRPS]SMSi~>
+GatU._,>n>&A@ZMHYa4q8]W5tCch@K30%0[eo>.<-mOEQW8ld)!Yqs"a`)Qe]R=^B1QT2l,lnh'+Gr;dM8t1XNrrHR;NX;b,Xd!(S?.F;.9pgebl*tmfZ]f,3_$KpD,K'16"1:N5!MtjK':p6WZTQNT<BR;>`BasS)nM7:.Js./uPmMrfM8fAUVS;cK*)3;CY#0jOeBlWnGJs"=T$T.0Kjl#L.\X,&C8[Zu0KR2259@R,"M5leWUN>\%0HnZ<uV_*sL<2Qop==-<7.X>)gqa-fCk+Igq"$A-\mm+m$[j50V#R,m9&JIeHm5^+[G9JZ;XK/PaUhMphr9,bMc+q,2P+C2mt/[Zu:?Y,\mlD@m,*<'+Qmu_!/kY\^n4d[86Ar'MW>^r2_^atE<Hu&#SaDi><=5SVL],G']g%*$uZ(Ki2ng(Z=~>
 endstream
 endobj
 8 0 obj
@@ -87,19 +87,19 @@ endobj
 xref
 0 14
 0000000000 65535 f 
-0000002603 00000 n 
-0000002667 00000 n 
-0000002717 00000 n 
+0000002641 00000 n 
+0000002705 00000 n 
+0000002755 00000 n 
 0000000015 00000 n 
 0000000071 00000 n 
-0000001355 00000 n 
-0000001461 00000 n 
-0000001940 00000 n 
-0000002046 00000 n 
-0000002158 00000 n 
-0000002268 00000 n 
-0000002379 00000 n 
-0000002487 00000 n 
+0000001348 00000 n 
+0000001454 00000 n 
+0000001978 00000 n 
+0000002084 00000 n 
+0000002196 00000 n 
+0000002306 00000 n 
+0000002417 00000 n 
+0000002525 00000 n 
 trailer
 <<
 /Size 14
@@ -107,5 +107,5 @@ trailer
 /Info 4 0 R
 >>
 startxref
-2839
+2877
 %%EOF

docs/mapred_tutorial.html

@@ -156,6 +156,9 @@ document.write("Last Published: " + document.lastModified);
 <a href="api/index.html">API Docs</a>
 </div>
 <div class="menuitem">
+<a href="jdiff/changes.html">API Changes</a>
+</div>
+<div class="menuitem">
 <a href="http://wiki.apache.org/hadoop/">Wiki</a>
 </div>
 <div class="menuitem">

docs/native_libraries.html

@@ -156,6 +156,9 @@ document.write("Last Published: " + document.lastModified);
 <a href="api/index.html">API Docs</a>
 </div>
 <div class="menuitem">
+<a href="jdiff/changes.html">API Changes</a>
+</div>
+<div class="menuitem">
 <a href="http://wiki.apache.org/hadoop/">Wiki</a>
 </div>
 <div class="menuitem">

docs/quickstart.html

@@ -156,6 +156,9 @@ document.write("Last Published: " + document.lastModified);
 <a href="api/index.html">API Docs</a>
 </div>
 <div class="menuitem">
+<a href="jdiff/changes.html">API Changes</a>
+</div>
+<div class="menuitem">
 <a href="http://wiki.apache.org/hadoop/">Wiki</a>
 </div>
 <div class="menuitem">

docs/streaming.html

@@ -159,6 +159,9 @@ document.write("Last Published: " + document.lastModified);
 <a href="api/index.html">API Docs</a>
 </div>
 <div class="menuitem">
+<a href="jdiff/changes.html">API Changes</a>
+</div>
+<div class="menuitem">
 <a href="http://wiki.apache.org/hadoop/">Wiki</a>
 </div>
 <div class="menuitem">

src/docs/src/documentation/content/xdocs/distcp.xml

@@ -168,11 +168,39 @@
               line. The <code>urilist_uri</code> list should be a fully
               qualified URI.
               </td></tr>
+          <tr><td><code>-filelimit &lt;n&gt;</code></td>
+              <td>Limit the total number of files to be &lt;= n</td>
+              <td>See also <a href="#Symbolic-Representations">Symbolic
+                  Representations</a>.
+              </td></tr>
+          <tr><td><code>-sizelimit &lt;n&gt;</code></td>
+              <td>Limit the total size to be &lt;= n bytes</td>
+              <td>See also <a href="#Symbolic-Representations">Symbolic
+                  Representations</a>.
+              </td></tr>
+          <tr><td><code>-delete</code></td>
+              <td>Delete the files existing in the dst but not in src</td>
+              <td>The deletion is done by FS Shell.  So the trash will be used,
+                  if it is enable.
+              </td></tr>
 
         </table>
 
       </section>
 
+      <section id="Symbolic-Representations">
+        <title>Symbolic Representations</title>
+        <p>
+        The parameter &lt;n&gt; in <code>-filelimit</code>
+        and <code>-sizelimit</code> can be specified with symbolic
+        representation.  For examples,
+        </p>
+        <ul>
+          <li>1230k = 1230 * 1024 = 1259520</li>
+          <li>891g = 891 * 1024^3 = 956703965184</li>
+        </ul>
+      </section>
+
       <section id="uo">
         <title>Update and Overwrite</title>