zookeeperStarted.html 19 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631
  1. <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
  2. <html>
  3. <head>
  4. <META http-equiv="Content-Type" content="text/html; charset=UTF-8">
  5. <meta content="Apache Forrest" name="Generator">
  6. <meta name="Forrest-version" content="0.9">
  7. <meta name="Forrest-skin-name" content="pelt">
  8. <title>ZooKeeper Getting Started Guide</title>
  9. <link type="text/css" href="skin/basic.css" rel="stylesheet">
  10. <link media="screen" type="text/css" href="skin/screen.css" rel="stylesheet">
  11. <link media="print" type="text/css" href="skin/print.css" rel="stylesheet">
  12. <link type="text/css" href="skin/profile.css" rel="stylesheet">
  13. <script src="skin/getBlank.js" language="javascript" type="text/javascript"></script><script src="skin/getMenu.js" language="javascript" type="text/javascript"></script><script src="skin/fontsize.js" language="javascript" type="text/javascript"></script>
  14. <link rel="shortcut icon" href="images/favicon.ico">
  15. </head>
  16. <body onload="init()">
  17. <script type="text/javascript">ndeSetTextSize();</script>
  18. <div id="top">
  19. <!--+
  20. |breadtrail
  21. +-->
  22. <div class="breadtrail">
  23. <a href="http://www.apache.org/">Apache</a> &gt; <a href="http://zookeeper.apache.org/">ZooKeeper</a> &gt; <a href="http://zookeeper.apache.org/">ZooKeeper</a><script src="skin/breadcrumbs.js" language="JavaScript" type="text/javascript"></script>
  24. </div>
  25. <!--+
  26. |header
  27. +-->
  28. <div class="header">
  29. <!--+
  30. |start group logo
  31. +-->
  32. <div class="grouplogo">
  33. <a href="http://hadoop.apache.org/"><img class="logoImage" alt="Hadoop" src="images/hadoop-logo.jpg" title="Apache Hadoop"></a>
  34. </div>
  35. <!--+
  36. |end group logo
  37. +-->
  38. <!--+
  39. |start Project Logo
  40. +-->
  41. <div class="projectlogo">
  42. <a href="http://zookeeper.apache.org/"><img class="logoImage" alt="ZooKeeper" src="images/zookeeper_small.gif" title="ZooKeeper: distributed coordination"></a>
  43. </div>
  44. <!--+
  45. |end Project Logo
  46. +-->
  47. <!--+
  48. |start Search
  49. +-->
  50. <div class="searchbox">
  51. <form action="http://www.google.com/search" method="get" class="roundtopsmall">
  52. <input value="zookeeper.apache.org" name="sitesearch" type="hidden"><input onFocus="getBlank (this, 'Search the site with google');" size="25" name="q" id="query" type="text" value="Search the site with google">&nbsp;
  53. <input name="Search" value="Search" type="submit">
  54. </form>
  55. </div>
  56. <!--+
  57. |end search
  58. +-->
  59. <!--+
  60. |start Tabs
  61. +-->
  62. <ul id="tabs">
  63. <li>
  64. <a class="unselected" href="http://zookeeper.apache.org/">Project</a>
  65. </li>
  66. <li>
  67. <a class="unselected" href="https://cwiki.apache.org/confluence/display/ZOOKEEPER/">Wiki</a>
  68. </li>
  69. <li class="current">
  70. <a class="selected" href="index.html">ZooKeeper 3.4 Documentation</a>
  71. </li>
  72. </ul>
  73. <!--+
  74. |end Tabs
  75. +-->
  76. </div>
  77. </div>
  78. <div id="main">
  79. <div id="publishedStrip">
  80. <!--+
  81. |start Subtabs
  82. +-->
  83. <div id="level2tabs"></div>
  84. <!--+
  85. |end Endtabs
  86. +-->
  87. <script type="text/javascript"><!--
  88. document.write("Last Published: " + document.lastModified);
  89. // --></script>
  90. </div>
  91. <!--+
  92. |breadtrail
  93. +-->
  94. <div class="breadtrail">
  95. &nbsp;
  96. </div>
  97. <!--+
  98. |start Menu, mainarea
  99. +-->
  100. <!--+
  101. |start Menu
  102. +-->
  103. <div id="menu">
  104. <div onclick="SwitchMenu('menu_selected_1.1', 'skin/')" id="menu_selected_1.1Title" class="menutitle" style="background-image: url('skin/images/chapter_open.gif');">Overview</div>
  105. <div id="menu_selected_1.1" class="selectedmenuitemgroup" style="display: block;">
  106. <div class="menuitem">
  107. <a href="index.html">Welcome</a>
  108. </div>
  109. <div class="menuitem">
  110. <a href="zookeeperOver.html">Overview</a>
  111. </div>
  112. <div class="menupage">
  113. <div class="menupagetitle">Getting Started</div>
  114. </div>
  115. <div class="menuitem">
  116. <a href="releasenotes.html">Release Notes</a>
  117. </div>
  118. </div>
  119. <div onclick="SwitchMenu('menu_1.2', 'skin/')" id="menu_1.2Title" class="menutitle">Developer</div>
  120. <div id="menu_1.2" class="menuitemgroup">
  121. <div class="menuitem">
  122. <a href="api/index.html">API Docs</a>
  123. </div>
  124. <div class="menuitem">
  125. <a href="zookeeperProgrammers.html">Programmer's Guide</a>
  126. </div>
  127. <div class="menuitem">
  128. <a href="javaExample.html">Java Example</a>
  129. </div>
  130. <div class="menuitem">
  131. <a href="zookeeperTutorial.html">Barrier and Queue Tutorial</a>
  132. </div>
  133. <div class="menuitem">
  134. <a href="recipes.html">Recipes</a>
  135. </div>
  136. </div>
  137. <div onclick="SwitchMenu('menu_1.3', 'skin/')" id="menu_1.3Title" class="menutitle">Admin &amp; Ops</div>
  138. <div id="menu_1.3" class="menuitemgroup">
  139. <div class="menuitem">
  140. <a href="zookeeperAdmin.html">Administrator's Guide</a>
  141. </div>
  142. <div class="menuitem">
  143. <a href="zookeeperQuotas.html">Quota Guide</a>
  144. </div>
  145. <div class="menuitem">
  146. <a href="zookeeperJMX.html">JMX</a>
  147. </div>
  148. <div class="menuitem">
  149. <a href="zookeeperObservers.html">Observers Guide</a>
  150. </div>
  151. </div>
  152. <div onclick="SwitchMenu('menu_1.4', 'skin/')" id="menu_1.4Title" class="menutitle">Contributor</div>
  153. <div id="menu_1.4" class="menuitemgroup">
  154. <div class="menuitem">
  155. <a href="zookeeperInternals.html">ZooKeeper Internals</a>
  156. </div>
  157. </div>
  158. <div onclick="SwitchMenu('menu_1.5', 'skin/')" id="menu_1.5Title" class="menutitle">Miscellaneous</div>
  159. <div id="menu_1.5" class="menuitemgroup">
  160. <div class="menuitem">
  161. <a href="https://cwiki.apache.org/confluence/display/ZOOKEEPER">Wiki</a>
  162. </div>
  163. <div class="menuitem">
  164. <a href="https://cwiki.apache.org/confluence/display/ZOOKEEPER/FAQ">FAQ</a>
  165. </div>
  166. <div class="menuitem">
  167. <a href="http://zookeeper.apache.org/mailing_lists.html">Mailing Lists</a>
  168. </div>
  169. </div>
  170. <div id="credit"></div>
  171. <div id="roundbottom">
  172. <img style="display: none" class="corner" height="15" width="15" alt="" src="skin/images/rc-b-l-15-1body-2menu-3menu.png"></div>
  173. <!--+
  174. |alternative credits
  175. +-->
  176. <div id="credit2"></div>
  177. </div>
  178. <!--+
  179. |end Menu
  180. +-->
  181. <!--+
  182. |start content
  183. +-->
  184. <div id="content">
  185. <div title="Portable Document Format" class="pdflink">
  186. <a class="dida" href="zookeeperStarted.pdf"><img alt="PDF -icon" src="skin/images/pdfdoc.gif" class="skin"><br>
  187. PDF</a>
  188. </div>
  189. <h1>ZooKeeper Getting Started Guide</h1>
  190. <div id="front-matter">
  191. <div id="minitoc-area">
  192. <ul class="minitoc">
  193. <li>
  194. <a href="#ch_GettingStarted">Getting Started: Coordinating Distributed Applications with
  195. ZooKeeper</a>
  196. <ul class="minitoc">
  197. <li>
  198. <a href="#sc_Prerequisites">Pre-requisites</a>
  199. </li>
  200. <li>
  201. <a href="#sc_Download">Download</a>
  202. </li>
  203. <li>
  204. <a href="#sc_InstallingSingleMode">Standalone Operation</a>
  205. </li>
  206. <li>
  207. <a href="#sc_FileManagement">Managing ZooKeeper Storage</a>
  208. </li>
  209. <li>
  210. <a href="#sc_ConnectingToZooKeeper">Connecting to ZooKeeper</a>
  211. </li>
  212. <li>
  213. <a href="#sc_ProgrammingToZooKeeper">Programming to ZooKeeper</a>
  214. </li>
  215. <li>
  216. <a href="#sc_RunningReplicatedZooKeeper">Running Replicated ZooKeeper</a>
  217. </li>
  218. <li>
  219. <a href="#Other+Optimizations">Other Optimizations</a>
  220. </li>
  221. </ul>
  222. </li>
  223. </ul>
  224. </div>
  225. </div>
  226. <a name="ch_GettingStarted"></a>
  227. <h2 class="h3">Getting Started: Coordinating Distributed Applications with
  228. ZooKeeper</h2>
  229. <div class="section">
  230. <p>This document contains information to get you started quickly with
  231. ZooKeeper. It is aimed primarily at developers hoping to try it out, and
  232. contains simple installation instructions for a single ZooKeeper server, a
  233. few commands to verify that it is running, and a simple programming
  234. example. Finally, as a convenience, there are a few sections regarding
  235. more complicated installations, for example running replicated
  236. deployments, and optimizing the transaction log. However for the complete
  237. instructions for commercial deployments, please refer to the <a href="zookeeperAdmin.html">ZooKeeper
  238. Administrator's Guide</a>.</p>
  239. <a name="sc_Prerequisites"></a>
  240. <h3 class="h4">Pre-requisites</h3>
  241. <p>See <a href="zookeeperAdmin.html#sc_systemReq">
  242. System Requirements</a> in the Admin guide.</p>
  243. <a name="sc_Download"></a>
  244. <h3 class="h4">Download</h3>
  245. <p>To get a ZooKeeper distribution, download a recent
  246. <a href="http://zookeeper.apache.org/releases.html">
  247. stable</a> release from one of the Apache Download
  248. Mirrors.</p>
  249. <a name="sc_InstallingSingleMode"></a>
  250. <h3 class="h4">Standalone Operation</h3>
  251. <p>Setting up a ZooKeeper server in standalone mode is
  252. straightforward. The server is contained in a single JAR file,
  253. so installation consists of creating a configuration.</p>
  254. <p>Once you've downloaded a stable ZooKeeper release unpack
  255. it and cd to the root</p>
  256. <p>To start ZooKeeper you need a configuration file. Here is a sample,
  257. create it in <strong>conf/zoo.cfg</strong>:</p>
  258. <pre class="code">
  259. tickTime=2000
  260. dataDir=/var/lib/zookeeper
  261. clientPort=2181
  262. </pre>
  263. <p>This file can be called anything, but for the sake of this
  264. discussion call
  265. it <strong>conf/zoo.cfg</strong>. Change the
  266. value of <strong>dataDir</strong> to specify an
  267. existing (empty to start with) directory. Here are the meanings
  268. for each of the fields:</p>
  269. <dl>
  270. <dt>
  271. <term>
  272. <strong>tickTime</strong>
  273. </term>
  274. </dt>
  275. <dd>
  276. <p>the basic time unit in milliseconds used by ZooKeeper. It is
  277. used to do heartbeats and the minimum session timeout will be
  278. twice the tickTime.</p>
  279. </dd>
  280. </dl>
  281. <dl>
  282. <dt>
  283. <term>
  284. <strong>dataDir</strong>
  285. </term>
  286. </dt>
  287. <dd>
  288. <p>the location to store the in-memory database snapshots and,
  289. unless specified otherwise, the transaction log of updates to the
  290. database.</p>
  291. </dd>
  292. <dt>
  293. <term>
  294. <strong>clientPort</strong>
  295. </term>
  296. </dt>
  297. <dd>
  298. <p>the port to listen for client connections</p>
  299. </dd>
  300. </dl>
  301. <p>Now that you created the configuration file, you can start
  302. ZooKeeper:</p>
  303. <pre class="code">bin/zkServer.sh start</pre>
  304. <p>ZooKeeper logs messages using log4j -- more detail
  305. available in the
  306. <a href="zookeeperProgrammers.html#Logging">Logging</a>
  307. section of the Programmer's Guide. You will see log messages
  308. coming to the console (default) and/or a log file depending on
  309. the log4j configuration.</p>
  310. <p>The steps outlined here run ZooKeeper in standalone mode. There is
  311. no replication, so if ZooKeeper process fails, the service will go down.
  312. This is fine for most development situations, but to run ZooKeeper in
  313. replicated mode, please see <a href="#sc_RunningReplicatedZooKeeper">Running Replicated
  314. ZooKeeper</a>.</p>
  315. <a name="sc_FileManagement"></a>
  316. <h3 class="h4">Managing ZooKeeper Storage</h3>
  317. <p>For long running production systems ZooKeeper storage must
  318. be managed externally (dataDir and logs). See the section on
  319. <a href="zookeeperAdmin.html#sc_maintenance">maintenance</a> for
  320. more details.</p>
  321. <a name="sc_ConnectingToZooKeeper"></a>
  322. <h3 class="h4">Connecting to ZooKeeper</h3>
  323. <p>Once ZooKeeper is running, you have several options for connection
  324. to it:</p>
  325. <ul>
  326. <li>
  327. <p>
  328. <strong>Java</strong>: Use</p>
  329. <pre class="code">bin/zkCli.sh -server 127.0.0.1:2181</pre>
  330. <p>This lets you perform simple, file-like operations.</p>
  331. </li>
  332. <li>
  333. <p>
  334. <strong>C</strong>: compile cli_mt
  335. (multi-threaded) or cli_st (single-threaded) by running
  336. <span class="codefrag command">make cli_mt</span> or <span class="codefrag command">make
  337. cli_st</span> in
  338. the <strong>src/c</strong> subdirectory in
  339. the ZooKeeper sources. See the README contained within
  340. <strong>src/c</strong> for full details.</p>
  341. <p>You can run the program
  342. from <strong>src/c</strong> using:</p>
  343. <pre class="code">LD_LIBRARY_PATH=. cli_mt 127.0.0.1:2181</pre>
  344. <p>or</p>
  345. <pre class="code">LD_LIBRARY_PATH=. cli_st 127.0.0.1:2181</pre>
  346. <p>This will give you a simple shell to execute file
  347. system like operations on ZooKeeper.</p>
  348. </li>
  349. </ul>
  350. <p>Once you have connected, you should see something like:
  351. </p>
  352. <pre class="code">
  353. Connecting to localhost:2181
  354. log4j:WARN No appenders could be found for logger (org.apache.zookeeper.ZooKeeper).
  355. log4j:WARN Please initialize the log4j system properly.
  356. Welcome to ZooKeeper!
  357. JLine support is enabled
  358. [zkshell: 0]
  359. </pre>
  360. <p>
  361. From the shell, type <span class="codefrag command">help</span> to get a listing of commands that can be executed from the client, as in:
  362. </p>
  363. <pre class="code">
  364. [zkshell: 0] help
  365. ZooKeeper host:port cmd args
  366. get path [watch]
  367. ls path [watch]
  368. set path data [version]
  369. delquota [-n|-b] path
  370. quit
  371. printwatches on|off
  372. create path data acl
  373. stat path [watch]
  374. listquota path
  375. history
  376. setAcl path acl
  377. getAcl path
  378. sync path
  379. redo cmdno
  380. addauth scheme auth
  381. delete path [version]
  382. deleteall path
  383. setquota -n|-b val path
  384. </pre>
  385. <p>From here, you can try a few simple commands to get a feel for this simple command line interface. First, start by issuing the list command, as
  386. in <span class="codefrag command">ls</span>, yielding:
  387. </p>
  388. <pre class="code">
  389. [zkshell: 8] ls /
  390. [zookeeper]
  391. </pre>
  392. <p>Next, create a new znode by running <span class="codefrag command">create /zk_test my_data</span>. This creates a new znode and associates the string "my_data" with the node.
  393. You should see:</p>
  394. <pre class="code">
  395. [zkshell: 9] create /zk_test my_data
  396. Created /zk_test
  397. </pre>
  398. <p> Issue another <span class="codefrag command">ls /</span> command to see what the directory looks like:
  399. </p>
  400. <pre class="code">
  401. [zkshell: 11] ls /
  402. [zookeeper, zk_test]
  403. </pre>
  404. <p>
  405. Notice that the zk_test directory has now been created.
  406. </p>
  407. <p>Next, verify that the data was associated with the znode by running the <span class="codefrag command">get</span> command, as in:
  408. </p>
  409. <pre class="code">
  410. [zkshell: 12] get /zk_test
  411. my_data
  412. cZxid = 5
  413. ctime = Fri Jun 05 13:57:06 PDT 2009
  414. mZxid = 5
  415. mtime = Fri Jun 05 13:57:06 PDT 2009
  416. pZxid = 5
  417. cversion = 0
  418. dataVersion = 0
  419. aclVersion = 0
  420. ephemeralOwner = 0
  421. dataLength = 7
  422. numChildren = 0
  423. </pre>
  424. <p>We can change the data associated with zk_test by issuing the <span class="codefrag command">set</span> command, as in:
  425. </p>
  426. <pre class="code">
  427. [zkshell: 14] set /zk_test junk
  428. cZxid = 5
  429. ctime = Fri Jun 05 13:57:06 PDT 2009
  430. mZxid = 6
  431. mtime = Fri Jun 05 14:01:52 PDT 2009
  432. pZxid = 5
  433. cversion = 0
  434. dataVersion = 1
  435. aclVersion = 0
  436. ephemeralOwner = 0
  437. dataLength = 4
  438. numChildren = 0
  439. [zkshell: 15] get /zk_test
  440. junk
  441. cZxid = 5
  442. ctime = Fri Jun 05 13:57:06 PDT 2009
  443. mZxid = 6
  444. mtime = Fri Jun 05 14:01:52 PDT 2009
  445. pZxid = 5
  446. cversion = 0
  447. dataVersion = 1
  448. aclVersion = 0
  449. ephemeralOwner = 0
  450. dataLength = 4
  451. numChildren = 0
  452. </pre>
  453. <p>
  454. (Notice we did a <span class="codefrag command">get</span> after setting the data and it did, indeed, change.</p>
  455. <p>Finally, let's <span class="codefrag command">delete</span> the node by issuing:
  456. </p>
  457. <pre class="code">
  458. [zkshell: 16] delete /zk_test
  459. [zkshell: 17] ls /
  460. [zookeeper]
  461. [zkshell: 18]
  462. </pre>
  463. <p>That's it for now. To explore more, continue with the rest of this document and see the <a href="zookeeperProgrammers.html">Programmer's Guide</a>. </p>
  464. <a name="sc_ProgrammingToZooKeeper"></a>
  465. <h3 class="h4">Programming to ZooKeeper</h3>
  466. <p>ZooKeeper has a Java bindings and C bindings. They are
  467. functionally equivalent. The C bindings exist in two variants: single
  468. threaded and multi-threaded. These differ only in how the messaging loop
  469. is done. For more information, see the <a href="zookeeperProgrammers.html#ch_programStructureWithExample">Programming
  470. Examples in the ZooKeeper Programmer's Guide</a> for
  471. sample code using of the different APIs.</p>
  472. <a name="sc_RunningReplicatedZooKeeper"></a>
  473. <h3 class="h4">Running Replicated ZooKeeper</h3>
  474. <p>Running ZooKeeper in standalone mode is convenient for evaluation,
  475. some development, and testing. But in production, you should run
  476. ZooKeeper in replicated mode. A replicated group of servers in the same
  477. application is called a <em>quorum</em>, and in replicated
  478. mode, all servers in the quorum have copies of the same configuration
  479. file. The file is similar to the one used in standalone mode, but with a
  480. few differences. Here is an example:</p>
  481. <pre class="code">
  482. tickTime=2000
  483. dataDir=/var/lib/zookeeper
  484. clientPort=2181
  485. initLimit=5
  486. syncLimit=2
  487. server.1=zoo1:2888:3888
  488. server.2=zoo2:2888:3888
  489. server.3=zoo3:2888:3888
  490. </pre>
  491. <p>The new entry, <strong>initLimit</strong> is
  492. timeouts ZooKeeper uses to limit the length of time the ZooKeeper
  493. servers in quorum have to connect to a leader. The entry <strong>syncLimit</strong> limits how far out of date a server can
  494. be from a leader.</p>
  495. <p>With both of these timeouts, you specify the unit of time using
  496. <strong>tickTime</strong>. In this example, the timeout
  497. for initLimit is 5 ticks at 2000 milleseconds a tick, or 10
  498. seconds.</p>
  499. <p>The entries of the form <em>server.X</em> list the
  500. servers that make up the ZooKeeper service. When the server starts up,
  501. it knows which server it is by looking for the file
  502. <em>myid</em> in the data directory. That file has the
  503. contains the server number, in ASCII.</p>
  504. <p>Finally, note the two port numbers after each server
  505. name: " 2888" and "3888". Peers use the former port to connect
  506. to other peers. Such a connection is necessary so that peers
  507. can communicate, for example, to agree upon the order of
  508. updates. More specifically, a ZooKeeper server uses this port
  509. to connect followers to the leader. When a new leader arises, a
  510. follower opens a TCP connection to the leader using this
  511. port. Because the default leader election also uses TCP, we
  512. currently require another port for leader election. This is the
  513. second port in the server entry.
  514. </p>
  515. <div class="note">
  516. <div class="label">Note</div>
  517. <div class="content">
  518. <p>If you want to test multiple servers on a single
  519. machine, specify the servername
  520. as <em>localhost</em> with unique quorum &amp;
  521. leader election ports (i.e. 2888:3888, 2889:3889, 2890:3890 in
  522. the example above) for each server.X in that server's config
  523. file. Of course separate <em>dataDir</em>s and
  524. distinct <em>clientPort</em>s are also necessary
  525. (in the above replicated example, running on a
  526. single <em>localhost</em>, you would still have
  527. three config files).</p>
  528. </div>
  529. </div>
  530. <a name="Other+Optimizations"></a>
  531. <h3 class="h4">Other Optimizations</h3>
  532. <p>There are a couple of other configuration parameters that can
  533. greatly increase performance:</p>
  534. <ul>
  535. <li>
  536. <p>To get low latencies on updates it is important to
  537. have a dedicated transaction log directory. By default
  538. transaction logs are put in the same directory as the data
  539. snapshots and <em>myid</em> file. The dataLogDir
  540. parameters indicates a different directory to use for the
  541. transaction logs.</p>
  542. </li>
  543. <li>
  544. <p>
  545. <em>[tbd: what is the other config param?]</em>
  546. </p>
  547. </li>
  548. </ul>
  549. </div>
  550. <p align="right">
  551. <font size="-2"></font>
  552. </p>
  553. </div>
  554. <!--+
  555. |end content
  556. +-->
  557. <div class="clearboth">&nbsp;</div>
  558. </div>
  559. <div id="footer">
  560. <!--+
  561. |start bottomstrip
  562. +-->
  563. <div class="lastmodified">
  564. <script type="text/javascript"><!--
  565. document.write("Last Published: " + document.lastModified);
  566. // --></script>
  567. </div>
  568. <div class="copyright">
  569. Copyright &copy;
  570. 2008-2013 <a href="http://www.apache.org/licenses/">The Apache Software Foundation.</a>
  571. </div>
  572. <!--+
  573. |end bottomstrip
  574. +-->
  575. </div>
  576. </body>
  577. </html>