|
|
@@ -0,0 +1,283 @@
|
|
|
+<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
|
|
|
+<html>
|
|
|
+<head>
|
|
|
+<META http-equiv="Content-Type" content="text/html; charset=UTF-8">
|
|
|
+<meta content="Apache Forrest" name="Generator">
|
|
|
+<meta name="Forrest-version" content="0.8">
|
|
|
+<meta name="Forrest-skin-name" content="pelt">
|
|
|
+<title>Hadoop Archives</title>
|
|
|
+<link type="text/css" href="skin/basic.css" rel="stylesheet">
|
|
|
+<link media="screen" type="text/css" href="skin/screen.css" rel="stylesheet">
|
|
|
+<link media="print" type="text/css" href="skin/print.css" rel="stylesheet">
|
|
|
+<link type="text/css" href="skin/profile.css" rel="stylesheet">
|
|
|
+<script src="skin/getBlank.js" language="javascript" type="text/javascript"></script><script src="skin/getMenu.js" language="javascript" type="text/javascript"></script><script src="skin/fontsize.js" language="javascript" type="text/javascript"></script>
|
|
|
+<link rel="shortcut icon" href="images/favicon.ico">
|
|
|
+</head>
|
|
|
+<body onload="init()">
|
|
|
+<script type="text/javascript">ndeSetTextSize();</script>
|
|
|
+<div id="top">
|
|
|
+<!--+
|
|
|
+ |breadtrail
|
|
|
+ +-->
|
|
|
+<div class="breadtrail">
|
|
|
+<a href="http://www.apache.org/">Apache</a> > <a href="http://hadoop.apache.org/">Hadoop</a> > <a href="http://hadoop.apache.org/core/">Core</a><script src="skin/breadcrumbs.js" language="JavaScript" type="text/javascript"></script>
|
|
|
+</div>
|
|
|
+<!--+
|
|
|
+ |header
|
|
|
+ +-->
|
|
|
+<div class="header">
|
|
|
+<!--+
|
|
|
+ |start group logo
|
|
|
+ +-->
|
|
|
+<div class="grouplogo">
|
|
|
+<a href="http://hadoop.apache.org/"><img class="logoImage" alt="Hadoop" src="images/hadoop-logo.jpg" title="Apache Hadoop"></a>
|
|
|
+</div>
|
|
|
+<!--+
|
|
|
+ |end group logo
|
|
|
+ +-->
|
|
|
+<!--+
|
|
|
+ |start Project Logo
|
|
|
+ +-->
|
|
|
+<div class="projectlogo">
|
|
|
+<a href="http://hadoop.apache.org/core/"><img class="logoImage" alt="Hadoop" src="images/core-logo.gif" title="Scalable Computing Platform"></a>
|
|
|
+</div>
|
|
|
+<!--+
|
|
|
+ |end Project Logo
|
|
|
+ +-->
|
|
|
+<!--+
|
|
|
+ |start Search
|
|
|
+ +-->
|
|
|
+<div class="searchbox">
|
|
|
+<form action="http://www.google.com/search" method="get" class="roundtopsmall">
|
|
|
+<input value="hadoop.apache.org" name="sitesearch" type="hidden"><input onFocus="getBlank (this, 'Search the site with google');" size="25" name="q" id="query" type="text" value="Search the site with google">
|
|
|
+ <input name="Search" value="Search" type="submit">
|
|
|
+</form>
|
|
|
+</div>
|
|
|
+<!--+
|
|
|
+ |end search
|
|
|
+ +-->
|
|
|
+<!--+
|
|
|
+ |start Tabs
|
|
|
+ +-->
|
|
|
+<ul id="tabs">
|
|
|
+<li>
|
|
|
+<a class="unselected" href="http://hadoop.apache.org/core/">Project</a>
|
|
|
+</li>
|
|
|
+<li>
|
|
|
+<a class="unselected" href="http://wiki.apache.org/hadoop">Wiki</a>
|
|
|
+</li>
|
|
|
+<li class="current">
|
|
|
+<a class="selected" href="index.html">Hadoop 0.18 Documentation</a>
|
|
|
+</li>
|
|
|
+</ul>
|
|
|
+<!--+
|
|
|
+ |end Tabs
|
|
|
+ +-->
|
|
|
+</div>
|
|
|
+</div>
|
|
|
+<div id="main">
|
|
|
+<div id="publishedStrip">
|
|
|
+<!--+
|
|
|
+ |start Subtabs
|
|
|
+ +-->
|
|
|
+<div id="level2tabs"></div>
|
|
|
+<!--+
|
|
|
+ |end Endtabs
|
|
|
+ +-->
|
|
|
+<script type="text/javascript"><!--
|
|
|
+document.write("Last Published: " + document.lastModified);
|
|
|
+// --></script>
|
|
|
+</div>
|
|
|
+<!--+
|
|
|
+ |breadtrail
|
|
|
+ +-->
|
|
|
+<div class="breadtrail">
|
|
|
+
|
|
|
+
|
|
|
+ </div>
|
|
|
+<!--+
|
|
|
+ |start Menu, mainarea
|
|
|
+ +-->
|
|
|
+<!--+
|
|
|
+ |start Menu
|
|
|
+ +-->
|
|
|
+<div id="menu">
|
|
|
+<div onclick="SwitchMenu('menu_selected_1.1', 'skin/')" id="menu_selected_1.1Title" class="menutitle" style="background-image: url('skin/images/chapter_open.gif');">Documentation</div>
|
|
|
+<div id="menu_selected_1.1" class="selectedmenuitemgroup" style="display: block;">
|
|
|
+<div class="menuitem">
|
|
|
+<a href="index.html">Overview</a>
|
|
|
+</div>
|
|
|
+<div class="menuitem">
|
|
|
+<a href="quickstart.html">Quickstart</a>
|
|
|
+</div>
|
|
|
+<div class="menuitem">
|
|
|
+<a href="cluster_setup.html">Cluster Setup</a>
|
|
|
+</div>
|
|
|
+<div class="menuitem">
|
|
|
+<a href="hdfs_design.html">HDFS Architecture</a>
|
|
|
+</div>
|
|
|
+<div class="menuitem">
|
|
|
+<a href="hdfs_user_guide.html">HDFS User Guide</a>
|
|
|
+</div>
|
|
|
+<div class="menuitem">
|
|
|
+<a href="hdfs_permissions_guide.html">HDFS Permissions Guide</a>
|
|
|
+</div>
|
|
|
+<div class="menuitem">
|
|
|
+<a href="hdfs_quota_admin_guide.html">HDFS Quotas Administrator Guide</a>
|
|
|
+</div>
|
|
|
+<div class="menuitem">
|
|
|
+<a href="hdfs_shell.html">FS Shell Guide</a>
|
|
|
+</div>
|
|
|
+<div class="menuitem">
|
|
|
+<a href="mapred_tutorial.html">Map-Reduce Tutorial</a>
|
|
|
+</div>
|
|
|
+<div class="menuitem">
|
|
|
+<a href="native_libraries.html">Native Hadoop Libraries</a>
|
|
|
+</div>
|
|
|
+<div class="menuitem">
|
|
|
+<a href="streaming.html">Streaming</a>
|
|
|
+</div>
|
|
|
+<div class="menupage">
|
|
|
+<div class="menupagetitle">Hadoop Archives</div>
|
|
|
+</div>
|
|
|
+<div class="menuitem">
|
|
|
+<a href="hod.html">Hadoop On Demand</a>
|
|
|
+</div>
|
|
|
+<div class="menuitem">
|
|
|
+<a href="api/index.html">API Docs</a>
|
|
|
+</div>
|
|
|
+<div class="menuitem">
|
|
|
+<a href="http://wiki.apache.org/hadoop/">Wiki</a>
|
|
|
+</div>
|
|
|
+<div class="menuitem">
|
|
|
+<a href="http://wiki.apache.org/hadoop/FAQ">FAQ</a>
|
|
|
+</div>
|
|
|
+<div class="menuitem">
|
|
|
+<a href="http://hadoop.apache.org/core/mailing_lists.html">Mailing Lists</a>
|
|
|
+</div>
|
|
|
+<div class="menuitem">
|
|
|
+<a href="releasenotes.html">Release Notes</a>
|
|
|
+</div>
|
|
|
+<div class="menuitem">
|
|
|
+<a href="changes.html">All Changes</a>
|
|
|
+</div>
|
|
|
+</div>
|
|
|
+<div id="credit"></div>
|
|
|
+<div id="roundbottom">
|
|
|
+<img style="display: none" class="corner" height="15" width="15" alt="" src="skin/images/rc-b-l-15-1body-2menu-3menu.png"></div>
|
|
|
+<!--+
|
|
|
+ |alternative credits
|
|
|
+ +-->
|
|
|
+<div id="credit2"></div>
|
|
|
+</div>
|
|
|
+<!--+
|
|
|
+ |end Menu
|
|
|
+ +-->
|
|
|
+<!--+
|
|
|
+ |start content
|
|
|
+ +-->
|
|
|
+<div id="content">
|
|
|
+<div title="Portable Document Format" class="pdflink">
|
|
|
+<a class="dida" href="hadoop_archives.pdf"><img alt="PDF -icon" src="skin/images/pdfdoc.gif" class="skin"><br>
|
|
|
+ PDF</a>
|
|
|
+</div>
|
|
|
+<h1>Hadoop Archives</h1>
|
|
|
+<div id="minitoc-area">
|
|
|
+<ul class="minitoc">
|
|
|
+<li>
|
|
|
+<a href="#What+are+Hadoop+archives%3F"> What are Hadoop archives? </a>
|
|
|
+</li>
|
|
|
+<li>
|
|
|
+<a href="#How+to+create+an+archive%3F"> How to create an archive? </a>
|
|
|
+</li>
|
|
|
+<li>
|
|
|
+<a href="#How+to+look+up+files+in+archives%3F"> How to look up files in archives? </a>
|
|
|
+</li>
|
|
|
+</ul>
|
|
|
+</div>
|
|
|
+
|
|
|
+<a name="N1000D"></a><a name="What+are+Hadoop+archives%3F"></a>
|
|
|
+<h2 class="h3"> What are Hadoop archives? </h2>
|
|
|
+<div class="section">
|
|
|
+<p>
|
|
|
+ Hadoop archives are special format archives. A Hadoop archive
|
|
|
+ maps to a FileSystem directory. A Hadoop archive always has a *.har
|
|
|
+ extension. A Hadoop archive directory contains metadata (in the form
|
|
|
+ of _index and _masterindex) and data (part-*) files. The _index file contains
|
|
|
+ the name of the files that are part of the archive and there location
|
|
|
+ within the part files.
|
|
|
+ </p>
|
|
|
+</div>
|
|
|
+
|
|
|
+<a name="N10017"></a><a name="How+to+create+an+archive%3F"></a>
|
|
|
+<h2 class="h3"> How to create an archive? </h2>
|
|
|
+<div class="section">
|
|
|
+<p>
|
|
|
+
|
|
|
+<span class="codefrag">Usage: hadoop archive -archiveName name <src>* <dest></span>
|
|
|
+
|
|
|
+</p>
|
|
|
+<p>
|
|
|
+ -archiveName is the name of the archive you would like to create. An example would be
|
|
|
+ foo.har. The name should have a *.har extension. The inputs are filesystem pathnames which
|
|
|
+ work as usual with regular expressions. The destination directory would contain the archive.
|
|
|
+ Note that this is a Map Reduce job that creates the archives. You would need a map reduce cluster
|
|
|
+ to run this. The following is an example:</p>
|
|
|
+<p>
|
|
|
+
|
|
|
+<span class="codefrag">hadoop -archiveName foo.har /user/hadoop/dir1 /user/hadoop/dir2 /user/zoo/</span>
|
|
|
+
|
|
|
+</p>
|
|
|
+<p>
|
|
|
+ In the above example /user/hadoop/dir1 and /user/hadoop/dir2 will be archived in the following
|
|
|
+ filesystem directory -- /user/zoo/foo.har. The sources are not changed or removed when an archive
|
|
|
+ is created.
|
|
|
+ </p>
|
|
|
+</div>
|
|
|
+
|
|
|
+<a name="N1002F"></a><a name="How+to+look+up+files+in+archives%3F"></a>
|
|
|
+<h2 class="h3"> How to look up files in archives? </h2>
|
|
|
+<div class="section">
|
|
|
+<p>
|
|
|
+ The archives exposes itself as a filesystem layer. So all the fs shell commands in the archives work but
|
|
|
+ with a different URI. Also, note that archives are immutable. So, rename's, deletes and creates return an error.
|
|
|
+ URI for Hadoop Archives is
|
|
|
+ </p>
|
|
|
+<p>
|
|
|
+<span class="codefrag">har://scheme-hostname:port/archivepath/fileinarchive</span>
|
|
|
+</p>
|
|
|
+<p>
|
|
|
+ If no scheme is provided it assumes the underlying filesystem.
|
|
|
+ In that case the URI would look like
|
|
|
+ </p>
|
|
|
+<p>
|
|
|
+<span class="codefrag">
|
|
|
+ har:///archivepath/fileinarchive</span>
|
|
|
+</p>
|
|
|
+</div>
|
|
|
+
|
|
|
+</div>
|
|
|
+<!--+
|
|
|
+ |end content
|
|
|
+ +-->
|
|
|
+<div class="clearboth"> </div>
|
|
|
+</div>
|
|
|
+<div id="footer">
|
|
|
+<!--+
|
|
|
+ |start bottomstrip
|
|
|
+ +-->
|
|
|
+<div class="lastmodified">
|
|
|
+<script type="text/javascript"><!--
|
|
|
+document.write("Last Published: " + document.lastModified);
|
|
|
+// --></script>
|
|
|
+</div>
|
|
|
+<div class="copyright">
|
|
|
+ Copyright ©
|
|
|
+ 2007 <a href="http://www.apache.org/licenses/">The Apache Software Foundation.</a>
|
|
|
+</div>
|
|
|
+<!--+
|
|
|
+ |end bottomstrip
|
|
|
+ +-->
|
|
|
+</div>
|
|
|
+</body>
|
|
|
+</html>
|
Hairong Kuang