zookeeperAdmin.html 94 KB

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677787980818283848586878889909192939495969798991001011021031041051061071081091101111121131141151161171181191201211221231241251261271281291301311321331341351361371381391401411421431441451461471481491501511521531541551561571581591601611621631641651661671681691701711721731741751761771781791801811821831841851861871881891901911921931941951961971981992002012022032042052062072082092102112122132142152162172182192202212222232242252262272282292302312322332342352362372382392402412422432442452462472482492502512522532542552562572582592602612622632642652662672682692702712722732742752762772782792802812822832842852862872882892902912922932942952962972982993003013023033043053063073083093103113123133143153163173183193203213223233243253263273283293303313323333343353363373383393403413423433443453463473483493503513523533543553563573583593603613623633643653663673683693703713723733743753763773783793803813823833843853863873883893903913923933943953963973983994004014024034044054064074084094104114124134144154164174184194204214224234244254264274284294304314324334344354364374384394404414424434444454464474484494504514524534544554564574584594604614624634644654664674684694704714724734744754764774784794804814824834844854864874884894904914924934944954964974984995005015025035045055065075085095105115125135145155165175185195205215225235245255265275285295305315325335345355365375385395405415425435445455465475485495505515525535545555565575585595605615625635645655665675685695705715725735745755765775785795805815825835845855865875885895905915925935945955965975985996006016026036046056066076086096106116126136146156166176186196206216226236246256266276286296306316326336346356366376386396406416426436446456466476486496506516526536546556566576586596606616626636646656666676686696706716726736746756766776786796806816826836846856866876886896906916926936946956966976986997007017027037047057067077087097107117127137147157167177187197207217227237247257267277287297307317327337347357367377387397407417427437447457467477487497507517527537547557567577587597607617627637647657667677687697707717727737747757767777787797807817827837847857867877887897907917927937947957967977987998008018028038048058068078088098108118128138148158168178188198208218228238248258268278288298308318328338348358368378388398408418428438448458468478488498508518528538548558568578588598608618628638648658668678688698708718728738748758768778788798808818828838848858868878888898908918928938948958968978988999009019029039049059069079089099109119129139149159169179189199209219229239249259269279289299309319329339349359369379389399409419429439449459469479489499509519529539549559569579589599609619629639649659669679689699709719729739749759769779789799809819829839849859869879889899909919929939949959969979989991000100110021003100410051006100710081009101010111012101310141015101610171018101910201021102210231024102510261027102810291030103110321033103410351036103710381039104010411042104310441045104610471048104910501051105210531054105510561057105810591060106110621063106410651066106710681069107010711072107310741075107610771078107910801081108210831084108510861087108810891090109110921093109410951096109710981099110011011102110311041105110611071108110911101111111211131114111511161117111811191120112111221123112411251126112711281129113011311132113311341135113611371138113911401141114211431144114511461147114811491150115111521153115411551156115711581159116011611162116311641165116611671168116911701171117211731174117511761177117811791180118111821183118411851186118711881189119011911192119311941195119611971198119912001201120212031204120512061207120812091210121112121213121412151216121712181219122012211222122312241225122612271228122912301231123212331234123512361237123812391240124112421243124412451246124712481249125012511252125312541255125612571258125912601261126212631264126512661267126812691270127112721273127412751276127712781279128012811282128312841285128612871288128912901291129212931294129512961297129812991300130113021303130413051306130713081309131013111312131313141315131613171318131913201321132213231324132513261327132813291330133113321333133413351336133713381339134013411342134313441345134613471348134913501351135213531354135513561357135813591360136113621363136413651366136713681369137013711372137313741375137613771378137913801381138213831384138513861387138813891390139113921393139413951396139713981399140014011402140314041405140614071408140914101411141214131414141514161417141814191420142114221423142414251426142714281429143014311432143314341435143614371438143914401441144214431444144514461447144814491450145114521453145414551456145714581459146014611462146314641465146614671468146914701471147214731474147514761477147814791480148114821483148414851486148714881489149014911492149314941495149614971498149915001501150215031504150515061507150815091510151115121513151415151516151715181519152015211522152315241525152615271528152915301531153215331534153515361537153815391540154115421543154415451546154715481549155015511552155315541555155615571558155915601561156215631564156515661567156815691570157115721573157415751576157715781579158015811582158315841585158615871588158915901591159215931594159515961597159815991600160116021603160416051606160716081609161016111612161316141615161616171618161916201621162216231624162516261627162816291630163116321633163416351636163716381639164016411642164316441645164616471648164916501651165216531654165516561657165816591660166116621663166416651666166716681669167016711672167316741675167616771678167916801681168216831684168516861687168816891690169116921693169416951696169716981699170017011702170317041705170617071708170917101711171217131714171517161717171817191720172117221723172417251726172717281729173017311732173317341735173617371738173917401741174217431744174517461747174817491750175117521753175417551756175717581759176017611762176317641765176617671768176917701771177217731774177517761777177817791780178117821783178417851786178717881789179017911792179317941795179617971798179918001801180218031804180518061807180818091810181118121813181418151816181718181819182018211822182318241825182618271828182918301831183218331834183518361837183818391840184118421843184418451846184718481849185018511852185318541855185618571858185918601861186218631864186518661867186818691870187118721873187418751876187718781879188018811882188318841885188618871888188918901891189218931894189518961897189818991900190119021903190419051906190719081909191019111912191319141915191619171918191919201921192219231924192519261927192819291930193119321933193419351936193719381939194019411942194319441945194619471948194919501951195219531954195519561957195819591960196119621963196419651966196719681969197019711972197319741975197619771978197919801981198219831984198519861987198819891990199119921993199419951996199719981999200020012002200320042005200620072008200920102011201220132014201520162017201820192020202120222023202420252026202720282029203020312032203320342035203620372038203920402041204220432044204520462047204820492050205120522053205420552056205720582059206020612062206320642065206620672068206920702071207220732074207520762077207820792080208120822083208420852086208720882089209020912092209320942095209620972098209921002101210221032104210521062107210821092110211121122113211421152116211721182119212021212122212321242125212621272128212921302131213221332134213521362137213821392140214121422143214421452146214721482149215021512152215321542155215621572158215921602161216221632164216521662167216821692170217121722173217421752176217721782179218021812182218321842185218621872188218921902191219221932194219521962197219821992200220122022203220422052206220722082209221022112212221322142215221622172218221922202221222222232224222522262227222822292230223122322233223422352236223722382239224022412242224322442245224622472248224922502251225222532254225522562257225822592260226122622263226422652266226722682269227022712272227322742275227622772278227922802281228222832284228522862287228822892290229122922293229422952296229722982299230023012302230323042305230623072308230923102311231223132314231523162317231823192320232123222323232423252326232723282329233023312332233323342335233623372338233923402341234223432344234523462347234823492350235123522353235423552356235723582359236023612362236323642365236623672368236923702371237223732374237523762377237823792380238123822383238423852386238723882389239023912392239323942395239623972398239924002401240224032404240524062407240824092410241124122413241424152416241724182419242024212422242324242425242624272428242924302431243224332434243524362437243824392440244124422443244424452446244724482449245024512452245324542455245624572458245924602461246224632464246524662467246824692470247124722473247424752476247724782479248024812482248324842485248624872488248924902491249224932494249524962497249824992500250125022503250425052506250725082509251025112512251325142515251625172518251925202521252225232524252525262527252825292530253125322533253425352536253725382539254025412542254325442545254625472548254925502551255225532554255525562557255825592560256125622563256425652566256725682569257025712572257325742575257625772578257925802581258225832584258525862587258825892590259125922593259425952596259725982599260026012602260326042605260626072608260926102611261226132614261526162617261826192620262126222623262426252626262726282629263026312632263326342635263626372638263926402641264226432644264526462647264826492650265126522653265426552656265726582659266026612662266326642665266626672668266926702671267226732674267526762677267826792680
  1. <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
  2. <html>
  3. <head>
  4. <META http-equiv="Content-Type" content="text/html; charset=UTF-8">
  5. <meta content="Apache Forrest" name="Generator">
  6. <meta name="Forrest-version" content="0.9">
  7. <meta name="Forrest-skin-name" content="pelt">
  8. <title>ZooKeeper Administrator's Guide</title>
  9. <link type="text/css" href="skin/basic.css" rel="stylesheet">
  10. <link media="screen" type="text/css" href="skin/screen.css" rel="stylesheet">
  11. <link media="print" type="text/css" href="skin/print.css" rel="stylesheet">
  12. <link type="text/css" href="skin/profile.css" rel="stylesheet">
  13. <script src="skin/getBlank.js" language="javascript" type="text/javascript"></script><script src="skin/getMenu.js" language="javascript" type="text/javascript"></script><script src="skin/fontsize.js" language="javascript" type="text/javascript"></script>
  14. <link rel="shortcut icon" href="images/favicon.ico">
  15. </head>
  16. <body onload="init()">
  17. <script type="text/javascript">ndeSetTextSize();</script>
  18. <div id="top">
  19. <!--+
  20. |breadtrail
  21. +-->
  22. <div class="breadtrail">
  23. <a href="http://www.apache.org/">Apache</a> &gt; <a href="http://zookeeper.apache.org/">ZooKeeper</a> &gt; <a href="http://zookeeper.apache.org/">ZooKeeper</a><script src="skin/breadcrumbs.js" language="JavaScript" type="text/javascript"></script>
  24. </div>
  25. <!--+
  26. |header
  27. +-->
  28. <div class="header">
  29. <!--+
  30. |start group logo
  31. +-->
  32. <div class="grouplogo">
  33. <a href="http://hadoop.apache.org/"><img class="logoImage" alt="Hadoop" src="images/hadoop-logo.jpg" title="Apache Hadoop"></a>
  34. </div>
  35. <!--+
  36. |end group logo
  37. +-->
  38. <!--+
  39. |start Project Logo
  40. +-->
  41. <div class="projectlogo">
  42. <a href="http://zookeeper.apache.org/"><img class="logoImage" alt="ZooKeeper" src="images/zookeeper_small.gif" title="ZooKeeper: distributed coordination"></a>
  43. </div>
  44. <!--+
  45. |end Project Logo
  46. +-->
  47. <!--+
  48. |start Search
  49. +-->
  50. <div class="searchbox">
  51. <form action="http://www.google.com/search" method="get" class="roundtopsmall">
  52. <input value="zookeeper.apache.org" name="sitesearch" type="hidden"><input onFocus="getBlank (this, 'Search the site with google');" size="25" name="q" id="query" type="text" value="Search the site with google">&nbsp;
  53. <input name="Search" value="Search" type="submit">
  54. </form>
  55. </div>
  56. <!--+
  57. |end search
  58. +-->
  59. <!--+
  60. |start Tabs
  61. +-->
  62. <ul id="tabs">
  63. <li>
  64. <a class="unselected" href="http://zookeeper.apache.org/">Project</a>
  65. </li>
  66. <li>
  67. <a class="unselected" href="https://cwiki.apache.org/confluence/display/ZOOKEEPER/">Wiki</a>
  68. </li>
  69. <li class="current">
  70. <a class="selected" href="index.html">ZooKeeper 3.6 Documentation</a>
  71. </li>
  72. </ul>
  73. <!--+
  74. |end Tabs
  75. +-->
  76. </div>
  77. </div>
  78. <div id="main">
  79. <div id="publishedStrip">
  80. <!--+
  81. |start Subtabs
  82. +-->
  83. <div id="level2tabs"></div>
  84. <!--+
  85. |end Endtabs
  86. +-->
  87. <script type="text/javascript"><!--
  88. document.write("Last Published: " + document.lastModified);
  89. // --></script>
  90. </div>
  91. <!--+
  92. |breadtrail
  93. +-->
  94. <div class="breadtrail">
  95. &nbsp;
  96. </div>
  97. <!--+
  98. |start Menu, mainarea
  99. +-->
  100. <!--+
  101. |start Menu
  102. +-->
  103. <div id="menu">
  104. <div onclick="SwitchMenu('menu_1.1', 'skin/')" id="menu_1.1Title" class="menutitle">Overview</div>
  105. <div id="menu_1.1" class="menuitemgroup">
  106. <div class="menuitem">
  107. <a href="index.html">Welcome</a>
  108. </div>
  109. <div class="menuitem">
  110. <a href="zookeeperOver.html">Overview</a>
  111. </div>
  112. <div class="menuitem">
  113. <a href="zookeeperStarted.html">Getting Started</a>
  114. </div>
  115. <div class="menuitem">
  116. <a href="releasenotes.html">Release Notes</a>
  117. </div>
  118. </div>
  119. <div onclick="SwitchMenu('menu_1.2', 'skin/')" id="menu_1.2Title" class="menutitle">Developer</div>
  120. <div id="menu_1.2" class="menuitemgroup">
  121. <div class="menuitem">
  122. <a href="api/index.html">API Docs</a>
  123. </div>
  124. <div class="menuitem">
  125. <a href="zookeeperProgrammers.html">Programmer's Guide</a>
  126. </div>
  127. <div class="menuitem">
  128. <a href="javaExample.html">Java Example</a>
  129. </div>
  130. <div class="menuitem">
  131. <a href="zookeeperTutorial.html">Barrier and Queue Tutorial</a>
  132. </div>
  133. <div class="menuitem">
  134. <a href="recipes.html">Recipes</a>
  135. </div>
  136. </div>
  137. <div onclick="SwitchMenu('menu_selected_1.3', 'skin/')" id="menu_selected_1.3Title" class="menutitle" style="background-image: url('skin/images/chapter_open.gif');">Admin &amp; Ops</div>
  138. <div id="menu_selected_1.3" class="selectedmenuitemgroup" style="display: block;">
  139. <div class="menupage">
  140. <div class="menupagetitle">Administrator's Guide</div>
  141. </div>
  142. <div class="menuitem">
  143. <a href="zookeeperQuotas.html">Quota Guide</a>
  144. </div>
  145. <div class="menuitem">
  146. <a href="zookeeperJMX.html">JMX</a>
  147. </div>
  148. <div class="menuitem">
  149. <a href="zookeeperObservers.html">Observers Guide</a>
  150. </div>
  151. <div class="menuitem">
  152. <a href="zookeeperReconfig.html">Dynamic Reconfiguration</a>
  153. </div>
  154. </div>
  155. <div onclick="SwitchMenu('menu_1.4', 'skin/')" id="menu_1.4Title" class="menutitle">Contributor</div>
  156. <div id="menu_1.4" class="menuitemgroup">
  157. <div class="menuitem">
  158. <a href="zookeeperInternals.html">ZooKeeper Internals</a>
  159. </div>
  160. </div>
  161. <div onclick="SwitchMenu('menu_1.5', 'skin/')" id="menu_1.5Title" class="menutitle">Miscellaneous</div>
  162. <div id="menu_1.5" class="menuitemgroup">
  163. <div class="menuitem">
  164. <a href="https://cwiki.apache.org/confluence/display/ZOOKEEPER">Wiki</a>
  165. </div>
  166. <div class="menuitem">
  167. <a href="https://cwiki.apache.org/confluence/display/ZOOKEEPER/FAQ">FAQ</a>
  168. </div>
  169. <div class="menuitem">
  170. <a href="http://zookeeper.apache.org/mailing_lists.html">Mailing Lists</a>
  171. </div>
  172. </div>
  173. <div id="credit"></div>
  174. <div id="roundbottom">
  175. <img style="display: none" class="corner" height="15" width="15" alt="" src="skin/images/rc-b-l-15-1body-2menu-3menu.png"></div>
  176. <!--+
  177. |alternative credits
  178. +-->
  179. <div id="credit2"></div>
  180. </div>
  181. <!--+
  182. |end Menu
  183. +-->
  184. <!--+
  185. |start content
  186. +-->
  187. <div id="content">
  188. <div title="Portable Document Format" class="pdflink">
  189. <a class="dida" href="zookeeperAdmin.pdf"><img alt="PDF -icon" src="skin/images/pdfdoc.gif" class="skin"><br>
  190. PDF</a>
  191. </div>
  192. <h1>ZooKeeper Administrator's Guide</h1>
  193. <h3>A Guide to Deployment and Administration</h3>
  194. <div id="front-matter">
  195. <div id="minitoc-area">
  196. <ul class="minitoc">
  197. <li>
  198. <a href="#ch_deployment">Deployment</a>
  199. <ul class="minitoc">
  200. <li>
  201. <a href="#sc_systemReq">System Requirements</a>
  202. <ul class="minitoc">
  203. <li>
  204. <a href="#sc_supportedPlatforms">Supported Platforms</a>
  205. </li>
  206. <li>
  207. <a href="#sc_requiredSoftware">Required Software </a>
  208. </li>
  209. </ul>
  210. </li>
  211. <li>
  212. <a href="#sc_zkMulitServerSetup">Clustered (Multi-Server) Setup</a>
  213. </li>
  214. <li>
  215. <a href="#sc_singleAndDevSetup">Single Server and Developer Setup</a>
  216. </li>
  217. </ul>
  218. </li>
  219. <li>
  220. <a href="#ch_administration">Administration</a>
  221. <ul class="minitoc">
  222. <li>
  223. <a href="#sc_designing">Designing a ZooKeeper Deployment</a>
  224. <ul class="minitoc">
  225. <li>
  226. <a href="#sc_CrossMachineRequirements">Cross Machine Requirements</a>
  227. </li>
  228. <li>
  229. <a href="#Single+Machine+Requirements">Single Machine Requirements</a>
  230. </li>
  231. </ul>
  232. </li>
  233. <li>
  234. <a href="#sc_provisioning">Provisioning</a>
  235. </li>
  236. <li>
  237. <a href="#sc_strengthsAndLimitations">Things to Consider: ZooKeeper Strengths and Limitations</a>
  238. </li>
  239. <li>
  240. <a href="#sc_administering">Administering</a>
  241. </li>
  242. <li>
  243. <a href="#sc_maintenance">Maintenance</a>
  244. <ul class="minitoc">
  245. <li>
  246. <a href="#Ongoing+Data+Directory+Cleanup">Ongoing Data Directory Cleanup</a>
  247. </li>
  248. <li>
  249. <a href="#Debug+Log+Cleanup+%28log4j%29">Debug Log Cleanup (log4j)</a>
  250. </li>
  251. </ul>
  252. </li>
  253. <li>
  254. <a href="#sc_supervision">Supervision</a>
  255. </li>
  256. <li>
  257. <a href="#sc_monitoring">Monitoring</a>
  258. </li>
  259. <li>
  260. <a href="#sc_logging">Logging</a>
  261. </li>
  262. <li>
  263. <a href="#sc_troubleshooting">Troubleshooting</a>
  264. </li>
  265. <li>
  266. <a href="#sc_configuration">Configuration Parameters</a>
  267. <ul class="minitoc">
  268. <li>
  269. <a href="#sc_minimumConfiguration">Minimum Configuration</a>
  270. </li>
  271. <li>
  272. <a href="#sc_advancedConfiguration">Advanced Configuration</a>
  273. </li>
  274. <li>
  275. <a href="#sc_clusterOptions">Cluster Options</a>
  276. </li>
  277. <li>
  278. <a href="#sc_authOptions">Encryption, Authentication, Authorization Options</a>
  279. </li>
  280. <li>
  281. <a href="#Experimental+Options%2FFeatures">Experimental Options/Features</a>
  282. </li>
  283. <li>
  284. <a href="#Unsafe+Options">Unsafe Options</a>
  285. </li>
  286. <li>
  287. <a href="#Disabling+data+directory+autocreation">Disabling data directory autocreation</a>
  288. </li>
  289. <li>
  290. <a href="#sc_db_existence_validation">Enabling db existence validation</a>
  291. </li>
  292. <li>
  293. <a href="#sc_performance_options">Performance Tuning Options</a>
  294. </li>
  295. <li>
  296. <a href="#Communication+using+the+Netty+framework">Communication using the Netty framework</a>
  297. </li>
  298. <li>
  299. <a href="#sc_adminserver_config">AdminServer configuration</a>
  300. </li>
  301. </ul>
  302. </li>
  303. <li>
  304. <a href="#sc_zkCommands">ZooKeeper Commands</a>
  305. <ul class="minitoc">
  306. <li>
  307. <a href="#sc_4lw">The Four Letter Words</a>
  308. </li>
  309. <li>
  310. <a href="#sc_adminserver">The AdminServer</a>
  311. </li>
  312. </ul>
  313. </li>
  314. <li>
  315. <a href="#sc_dataFileManagement">Data File Management</a>
  316. <ul class="minitoc">
  317. <li>
  318. <a href="#The+Data+Directory">The Data Directory</a>
  319. </li>
  320. <li>
  321. <a href="#The+Log+Directory">The Log Directory</a>
  322. </li>
  323. <li>
  324. <a href="#sc_filemanagement">File Management</a>
  325. </li>
  326. <li>
  327. <a href="#Recovery+-+TxnLogToolkit">Recovery - TxnLogToolkit</a>
  328. </li>
  329. </ul>
  330. </li>
  331. <li>
  332. <a href="#sc_commonProblems">Things to Avoid</a>
  333. </li>
  334. <li>
  335. <a href="#sc_bestPractices">Best Practices</a>
  336. </li>
  337. </ul>
  338. </li>
  339. </ul>
  340. </div>
  341. </div>
  342. <a name="ch_deployment"></a>
  343. <h2 class="h3">Deployment</h2>
  344. <div class="section">
  345. <p>This section contains information about deploying Zookeeper and
  346. covers these topics:</p>
  347. <ul>
  348. <li>
  349. <p>
  350. <a href="#sc_systemReq">System Requirements</a>
  351. </p>
  352. </li>
  353. <li>
  354. <p>
  355. <a href="#sc_zkMulitServerSetup">Clustered (Multi-Server) Setup</a>
  356. </p>
  357. </li>
  358. <li>
  359. <p>
  360. <a href="#sc_singleAndDevSetup">Single Server and Developer Setup</a>
  361. </p>
  362. </li>
  363. </ul>
  364. <p>The first two sections assume you are interested in installing
  365. ZooKeeper in a production environment such as a datacenter. The final
  366. section covers situations in which you are setting up ZooKeeper on a
  367. limited basis - for evaluation, testing, or development - but not in a
  368. production environment.</p>
  369. <a name="sc_systemReq"></a>
  370. <h3 class="h4">System Requirements</h3>
  371. <a name="sc_supportedPlatforms"></a>
  372. <h4>Supported Platforms</h4>
  373. <p>ZooKeeper consists of multiple components. Some components are
  374. supported broadly, and other components are supported only on a smaller
  375. set of platforms.</p>
  376. <ul>
  377. <li>
  378. <p>
  379. <strong>Client</strong> is the Java client
  380. library, used by applications to connect to a ZooKeeper ensemble.
  381. </p>
  382. </li>
  383. <li>
  384. <p>
  385. <strong>Server</strong> is the Java server
  386. that runs on the ZooKeeper ensemble nodes.</p>
  387. </li>
  388. <li>
  389. <p>
  390. <strong>Native Client</strong> is a client
  391. implemented in C, similar to the Java client, used by applications
  392. to connect to a ZooKeeper ensemble.</p>
  393. </li>
  394. <li>
  395. <p>
  396. <strong>Contrib</strong> refers to multiple
  397. optional add-on components.</p>
  398. </li>
  399. </ul>
  400. <p>The following matrix describes the level of support committed for
  401. running each component on different operating system platforms.</p>
  402. <table class="ForrestTable" cellspacing="1" cellpadding="4">
  403. <caption>Support Matrix</caption>
  404. <title>Support Matrix</title>
  405. <tr>
  406. <th>Operating System</th>
  407. <th>Client</th>
  408. <th>Server</th>
  409. <th>Native Client</th>
  410. <th>Contrib</th>
  411. </tr>
  412. <tr>
  413. <td>GNU/Linux</td>
  414. <td>Development and Production</td>
  415. <td>Development and Production</td>
  416. <td>Development and Production</td>
  417. <td>Development and Production</td>
  418. </tr>
  419. <tr>
  420. <td>Solaris</td>
  421. <td>Development and Production</td>
  422. <td>Development and Production</td>
  423. <td>Not Supported</td>
  424. <td>Not Supported</td>
  425. </tr>
  426. <tr>
  427. <td>FreeBSD</td>
  428. <td>Development and Production</td>
  429. <td>Development and Production</td>
  430. <td>Not Supported</td>
  431. <td>Not Supported</td>
  432. </tr>
  433. <tr>
  434. <td>Windows</td>
  435. <td>Development and Production</td>
  436. <td>Development and Production</td>
  437. <td>Not Supported</td>
  438. <td>Not Supported</td>
  439. </tr>
  440. <tr>
  441. <td>Mac OS X</td>
  442. <td>Development Only</td>
  443. <td>Development Only</td>
  444. <td>Not Supported</td>
  445. <td>Not Supported</td>
  446. </tr>
  447. </table>
  448. <p>For any operating system not explicitly mentioned as supported in
  449. the matrix, components may or may not work. The ZooKeeper community
  450. will fix obvious bugs that are reported for other platforms, but there
  451. is no full support.</p>
  452. <a name="sc_requiredSoftware"></a>
  453. <h4>Required Software </h4>
  454. <p>ZooKeeper runs in Java, release 1.8 or greater (JDK 8 or
  455. greater, FreeBSD support requires openjdk8). It runs as an
  456. <em>ensemble</em> of ZooKeeper servers. Three
  457. ZooKeeper servers is the minimum recommended size for an
  458. ensemble, and we also recommend that they run on separate
  459. machines. At Yahoo!, ZooKeeper is usually deployed on
  460. dedicated RHEL boxes, with dual-core processors, 2GB of RAM,
  461. and 80GB IDE hard drives.</p>
  462. <a name="sc_zkMulitServerSetup"></a>
  463. <h3 class="h4">Clustered (Multi-Server) Setup</h3>
  464. <p>For reliable ZooKeeper service, you should deploy ZooKeeper in a
  465. cluster known as an <em>ensemble</em>. As long as a majority
  466. of the ensemble are up, the service will be available. Because Zookeeper
  467. requires a majority, it is best to use an
  468. odd number of machines. For example, with four machines ZooKeeper can
  469. only handle the failure of a single machine; if two machines fail, the
  470. remaining two machines do not constitute a majority. However, with five
  471. machines ZooKeeper can handle the failure of two machines. </p>
  472. <div class="note">
  473. <div class="label">Note</div>
  474. <div class="content">
  475. <p>
  476. As mentioned in the
  477. <a href="zookeeperStarted.html">ZooKeeper Getting Started Guide</a>
  478. , a minimum of three servers are required for a fault tolerant
  479. clustered setup, and it is strongly recommended that you have an
  480. odd number of servers.
  481. </p>
  482. <p>Usually three servers is more than enough for a production
  483. install, but for maximum reliability during maintenance, you may
  484. wish to install five servers. With three servers, if you perform
  485. maintenance on one of them, you are vulnerable to a failure on one
  486. of the other two servers during that maintenance. If you have five
  487. of them running, you can take one down for maintenance, and know
  488. that you're still OK if one of the other four suddenly fails.
  489. </p>
  490. <p>Your redundancy considerations should include all aspects of
  491. your environment. If you have three ZooKeeper servers, but their
  492. network cables are all plugged into the same network switch, then
  493. the failure of that switch will take down your entire ensemble.
  494. </p>
  495. </div>
  496. </div>
  497. <p>Here are the steps to setting a server that will be part of an
  498. ensemble. These steps should be performed on every host in the
  499. ensemble:</p>
  500. <ol>
  501. <li>
  502. <p>Install the Java JDK. You can use the native packaging system
  503. for your system, or download the JDK from:</p>
  504. <p>
  505. <a href="http://java.sun.com/javase/downloads/index.jsp">http://java.sun.com/javase/downloads/index.jsp</a>
  506. </p>
  507. </li>
  508. <li>
  509. <p>Set the Java heap size. This is very important to avoid
  510. swapping, which will seriously degrade ZooKeeper performance. To
  511. determine the correct value, use load tests, and make sure you are
  512. well below the usage limit that would cause you to swap. Be
  513. conservative - use a maximum heap size of 3GB for a 4GB
  514. machine.</p>
  515. </li>
  516. <li>
  517. <p>Install the ZooKeeper Server Package. It can be downloaded
  518. from:
  519. </p>
  520. <p>
  521. <a href="http://zookeeper.apache.org/releases.html">
  522. http://zookeeper.apache.org/releases.html
  523. </a>
  524. </p>
  525. </li>
  526. <li>
  527. <p>Create a configuration file. This file can be called anything.
  528. Use the following settings as a starting point:</p>
  529. <pre class="code">
  530. tickTime=2000
  531. dataDir=/var/lib/zookeeper/
  532. clientPort=2181
  533. initLimit=5
  534. syncLimit=2
  535. server.1=zoo1:2888:3888
  536. server.2=zoo2:2888:3888
  537. server.3=zoo3:2888:3888</pre>
  538. <p>You can find the meanings of these and other configuration
  539. settings in the section <a href="#sc_configuration">Configuration Parameters</a>. A word
  540. though about a few here:</p>
  541. <p>Every machine that is part of the ZooKeeper ensemble should know
  542. about every other machine in the ensemble. You accomplish this with
  543. the series of lines of the form <strong>server.id=host:port:port</strong>. The parameters <strong>host</strong> and <strong>port</strong> are straightforward. You attribute the
  544. server id to each machine by creating a file named
  545. <span class="codefrag filename">myid</span>, one for each server, which resides in
  546. that server's data directory, as specified by the configuration file
  547. parameter <strong>dataDir</strong>.</p>
  548. </li>
  549. <li>
  550. <p>The myid file
  551. consists of a single line containing only the text of that machine's
  552. id. So <span class="codefrag filename">myid</span> of server 1 would contain the text
  553. "1" and nothing else. The id must be unique within the
  554. ensemble and should have a value between 1 and 255.</p>
  555. </li>
  556. <li>
  557. <p>Create an initialization marker file <span class="codefrag filename">initialize</span>
  558. in the same directory as <span class="codefrag filename">myid</span>. This file indicates
  559. that an empty data directory is expected. When present, an empty data base
  560. is created and the marker file deleted. When not present, an empty data
  561. directory will mean this peer will not have voting rights and it will not
  562. populate the data directory until it communicates with an active leader.
  563. Intended use is to only create this file when bringing up a new
  564. ensemble. </p>
  565. </li>
  566. <li>
  567. <p>If your configuration file is set up, you can start a
  568. ZooKeeper server:</p>
  569. <p>
  570. <span class="codefrag computeroutput">$ java -cp zookeeper.jar:lib/slf4j-api-1.7.5.jar:lib/slf4j-log4j12-1.7.5.jar:lib/log4j-1.2.17.jar:conf \
  571. org.apache.zookeeper.server.quorum.QuorumPeerMain zoo.cfg
  572. </span>
  573. </p>
  574. <p>QuorumPeerMain starts a ZooKeeper server,
  575. <a href="http://java.sun.com/javase/technologies/core/mntr-mgmt/javamanagement/">JMX</a>
  576. management beans are also registered which allows
  577. management through a JMX management console.
  578. The <a href="zookeeperJMX.html">ZooKeeper JMX
  579. document</a> contains details on managing ZooKeeper with JMX.
  580. </p>
  581. <p>See the script <em>bin/zkServer.sh</em>,
  582. which is included in the release, for an example
  583. of starting server instances.</p>
  584. </li>
  585. <li>
  586. <p>Test your deployment by connecting to the hosts:</p>
  587. <p>In Java, you can run the following command to execute
  588. simple operations:</p>
  589. <p>
  590. <span class="codefrag computeroutput">$ bin/zkCli.sh -server 127.0.0.1:2181</span>
  591. </p>
  592. </li>
  593. </ol>
  594. <a name="sc_singleAndDevSetup"></a>
  595. <h3 class="h4">Single Server and Developer Setup</h3>
  596. <p>If you want to setup ZooKeeper for development purposes, you will
  597. probably want to setup a single server instance of ZooKeeper, and then
  598. install either the Java or C client-side libraries and bindings on your
  599. development machine.</p>
  600. <p>The steps to setting up a single server instance are the similar
  601. to the above, except the configuration file is simpler. You can find the
  602. complete instructions in the <a href="zookeeperStarted.html#sc_InstallingSingleMode">Installing and
  603. Running ZooKeeper in Single Server Mode</a> section of the <a href="zookeeperStarted.html">ZooKeeper Getting Started
  604. Guide</a>.</p>
  605. <p>For information on installing the client side libraries, refer to
  606. the <a href="zookeeperProgrammers.html#Bindings">Bindings</a>
  607. section of the <a href="zookeeperProgrammers.html">ZooKeeper
  608. Programmer's Guide</a>.</p>
  609. </div>
  610. <a name="ch_administration"></a>
  611. <h2 class="h3">Administration</h2>
  612. <div class="section">
  613. <p>This section contains information about running and maintaining
  614. ZooKeeper and covers these topics: </p>
  615. <ul>
  616. <li>
  617. <p>
  618. <a href="#sc_designing">Designing a ZooKeeper Deployment</a>
  619. </p>
  620. </li>
  621. <li>
  622. <p>
  623. <a href="#sc_provisioning">Provisioning</a>
  624. </p>
  625. </li>
  626. <li>
  627. <p>
  628. <a href="#sc_strengthsAndLimitations">Things to Consider: ZooKeeper Strengths and Limitations</a>
  629. </p>
  630. </li>
  631. <li>
  632. <p>
  633. <a href="#sc_administering">Administering</a>
  634. </p>
  635. </li>
  636. <li>
  637. <p>
  638. <a href="#sc_maintenance">Maintenance</a>
  639. </p>
  640. </li>
  641. <li>
  642. <p>
  643. <a href="#sc_supervision">Supervision</a>
  644. </p>
  645. </li>
  646. <li>
  647. <p>
  648. <a href="#sc_monitoring">Monitoring</a>
  649. </p>
  650. </li>
  651. <li>
  652. <p>
  653. <a href="#sc_logging">Logging</a>
  654. </p>
  655. </li>
  656. <li>
  657. <p>
  658. <a href="#sc_troubleshooting">Troubleshooting</a>
  659. </p>
  660. </li>
  661. <li>
  662. <p>
  663. <a href="#sc_configuration">Configuration Parameters</a>
  664. </p>
  665. </li>
  666. <li>
  667. <p>
  668. <a href="#sc_zkCommands">ZooKeeper Commands</a>
  669. </p>
  670. </li>
  671. <li>
  672. <p>
  673. <a href="#sc_dataFileManagement">Data File Management</a>
  674. </p>
  675. </li>
  676. <li>
  677. <p>
  678. <a href="#sc_commonProblems">Things to Avoid</a>
  679. </p>
  680. </li>
  681. <li>
  682. <p>
  683. <a href="#sc_bestPractices">Best Practices</a>
  684. </p>
  685. </li>
  686. </ul>
  687. <a name="sc_designing"></a>
  688. <h3 class="h4">Designing a ZooKeeper Deployment</h3>
  689. <p>The reliability of ZooKeeper rests on two basic assumptions.</p>
  690. <ol>
  691. <li>
  692. <p> Only a minority of servers in a deployment
  693. will fail. <em>Failure</em> in this context
  694. means a machine crash, or some error in the network that
  695. partitions a server off from the majority.</p>
  696. </li>
  697. <li>
  698. <p> Deployed machines operate correctly. To
  699. operate correctly means to execute code correctly, to have
  700. clocks that work properly, and to have storage and network
  701. components that perform consistently.</p>
  702. </li>
  703. </ol>
  704. <p>The sections below contain considerations for ZooKeeper
  705. administrators to maximize the probability for these assumptions
  706. to hold true. Some of these are cross-machines considerations,
  707. and others are things you should consider for each and every
  708. machine in your deployment.</p>
  709. <a name="sc_CrossMachineRequirements"></a>
  710. <h4>Cross Machine Requirements</h4>
  711. <p>For the ZooKeeper service to be active, there must be a
  712. majority of non-failing machines that can communicate with
  713. each other. To create a deployment that can tolerate the
  714. failure of F machines, you should count on deploying 2xF+1
  715. machines. Thus, a deployment that consists of three machines
  716. can handle one failure, and a deployment of five machines can
  717. handle two failures. Note that a deployment of six machines
  718. can only handle two failures since three machines is not a
  719. majority. For this reason, ZooKeeper deployments are usually
  720. made up of an odd number of machines.</p>
  721. <p>To achieve the highest probability of tolerating a failure
  722. you should try to make machine failures independent. For
  723. example, if most of the machines share the same switch,
  724. failure of that switch could cause a correlated failure and
  725. bring down the service. The same holds true of shared power
  726. circuits, cooling systems, etc.</p>
  727. <a name="Single+Machine+Requirements"></a>
  728. <h4>Single Machine Requirements</h4>
  729. <p>If ZooKeeper has to contend with other applications for
  730. access to resources like storage media, CPU, network, or
  731. memory, its performance will suffer markedly. ZooKeeper has
  732. strong durability guarantees, which means it uses storage
  733. media to log changes before the operation responsible for the
  734. change is allowed to complete. You should be aware of this
  735. dependency then, and take great care if you want to ensure
  736. that ZooKeeper operations aren&rsquo;t held up by your media. Here
  737. are some things you can do to minimize that sort of
  738. degradation:
  739. </p>
  740. <ul>
  741. <li>
  742. <p>ZooKeeper's transaction log must be on a dedicated
  743. device. (A dedicated partition is not enough.) ZooKeeper
  744. writes the log sequentially, without seeking Sharing your
  745. log device with other processes can cause seeks and
  746. contention, which in turn can cause multi-second
  747. delays.</p>
  748. </li>
  749. <li>
  750. <p>Do not put ZooKeeper in a situation that can cause a
  751. swap. In order for ZooKeeper to function with any sort of
  752. timeliness, it simply cannot be allowed to swap.
  753. Therefore, make certain that the maximum heap size given
  754. to ZooKeeper is not bigger than the amount of real memory
  755. available to ZooKeeper. For more on this, see
  756. <a href="#sc_commonProblems">Things to Avoid</a>
  757. below. </p>
  758. </li>
  759. </ul>
  760. <a name="sc_provisioning"></a>
  761. <h3 class="h4">Provisioning</h3>
  762. <p></p>
  763. <a name="sc_strengthsAndLimitations"></a>
  764. <h3 class="h4">Things to Consider: ZooKeeper Strengths and Limitations</h3>
  765. <p></p>
  766. <a name="sc_administering"></a>
  767. <h3 class="h4">Administering</h3>
  768. <p></p>
  769. <a name="sc_maintenance"></a>
  770. <h3 class="h4">Maintenance</h3>
  771. <p>Little long term maintenance is required for a ZooKeeper
  772. cluster however you must be aware of the following:</p>
  773. <a name="Ongoing+Data+Directory+Cleanup"></a>
  774. <h4>Ongoing Data Directory Cleanup</h4>
  775. <p>The ZooKeeper <a href="#var_datadir">Data
  776. Directory</a> contains files which are a persistent copy
  777. of the znodes stored by a particular serving ensemble. These
  778. are the snapshot and transactional log files. As changes are
  779. made to the znodes these changes are appended to a
  780. transaction log. Occasionally, when a log grows large, a
  781. snapshot of the current state of all znodes will be written
  782. to the filesystem and a new transaction log file is created
  783. for future transactions. During snapshotting, ZooKeeper may
  784. continue appending incoming transactions to the old log file.
  785. Therefore, some transactions which are newer than a snapshot
  786. may be found in the last transaction log preceding the
  787. snapshot.
  788. </p>
  789. <p>A ZooKeeper server <strong>will not remove
  790. old snapshots and log files</strong> when using the default
  791. configuration (see autopurge below), this is the
  792. responsibility of the operator. Every serving environment is
  793. different and therefore the requirements of managing these
  794. files may differ from install to install (backup for example).
  795. </p>
  796. <p>The PurgeTxnLog utility implements a simple retention
  797. policy that administrators can use. The <a href="api/index.html">API docs</a> contains details on
  798. calling conventions (arguments, etc...).
  799. </p>
  800. <p>In the following example the last count snapshots and
  801. their corresponding logs are retained and the others are
  802. deleted. The value of &lt;count&gt; should typically be
  803. greater than 3 (although not required, this provides 3 backups
  804. in the unlikely event a recent log has become corrupted). This
  805. can be run as a cron job on the ZooKeeper server machines to
  806. clean up the logs daily.</p>
  807. <pre class="code"> java -cp zookeeper.jar:lib/slf4j-api-1.7.5.jar:lib/slf4j-log4j12-1.7.5.jar:lib/log4j-1.2.17.jar:conf org.apache.zookeeper.server.PurgeTxnLog &lt;dataDir&gt; &lt;snapDir&gt; -n &lt;count&gt;</pre>
  808. <p>Automatic purging of the snapshots and corresponding
  809. transaction logs was introduced in version 3.4.0 and can be
  810. enabled via the following configuration parameters <strong>autopurge.snapRetainCount</strong> and <strong>autopurge.purgeInterval</strong>. For more on
  811. this, see <a href="#sc_advancedConfiguration">Advanced Configuration</a>
  812. below.</p>
  813. <a name="Debug+Log+Cleanup+%28log4j%29"></a>
  814. <h4>Debug Log Cleanup (log4j)</h4>
  815. <p>See the section on <a href="#sc_logging">logging</a> in this document. It is
  816. expected that you will setup a rolling file appender using the
  817. in-built log4j feature. The sample configuration file in the
  818. release tar's conf/log4j.properties provides an example of
  819. this.
  820. </p>
  821. <a name="sc_supervision"></a>
  822. <h3 class="h4">Supervision</h3>
  823. <p>You will want to have a supervisory process that manages
  824. each of your ZooKeeper server processes (JVM). The ZK server is
  825. designed to be "fail fast" meaning that it will shutdown
  826. (process exit) if an error occurs that it cannot recover
  827. from. As a ZooKeeper serving cluster is highly reliable, this
  828. means that while the server may go down the cluster as a whole
  829. is still active and serving requests. Additionally, as the
  830. cluster is "self healing" the failed server once restarted will
  831. automatically rejoin the ensemble w/o any manual
  832. interaction.</p>
  833. <p>Having a supervisory process such as <a href="http://cr.yp.to/daemontools.html">daemontools</a> or
  834. <a href="http://en.wikipedia.org/wiki/Service_Management_Facility">SMF</a>
  835. (other options for supervisory process are also available, it's
  836. up to you which one you would like to use, these are just two
  837. examples) managing your ZooKeeper server ensures that if the
  838. process does exit abnormally it will automatically be restarted
  839. and will quickly rejoin the cluster.</p>
  840. <p>It is also recommended to configure the ZooKeeper server process to
  841. terminate and dump its heap if an
  842. <span class="codefrag computeroutput">OutOfMemoryError</span> occurs. This is achieved
  843. by launching the JVM with the following arguments on Linux and Windows
  844. respectively. The <span class="codefrag filename">zkServer.sh</span> and
  845. <span class="codefrag filename">zkServer.cmd</span> scripts that ship with ZooKeeper set
  846. these options.
  847. </p>
  848. <pre class="code">-XX:+HeapDumpOnOutOfMemoryError -XX:OnOutOfMemoryError='kill -9 %p'</pre>
  849. <pre class="code">"-XX:+HeapDumpOnOutOfMemoryError" "-XX:OnOutOfMemoryError=cmd /c taskkill /pid %%%%p /t /f"</pre>
  850. <a name="sc_monitoring"></a>
  851. <h3 class="h4">Monitoring</h3>
  852. <p>The ZooKeeper service can be monitored in one of two
  853. primary ways; 1) the command port through the use of <a href="#sc_zkCommands">4 letter words</a> and 2) <a href="zookeeperJMX.html">JMX</a>. See the appropriate section for
  854. your environment/requirements.</p>
  855. <a name="sc_logging"></a>
  856. <h3 class="h4">Logging</h3>
  857. <p>
  858. ZooKeeper uses <strong><a href="http://www.slf4j.org">SLF4J</a></strong>
  859. version 1.7.5 as its logging infrastructure. For backward compatibility it is bound to
  860. <strong>LOG4J</strong> but you can use
  861. <strong><a href="http://logback.qos.ch/">LOGBack</a></strong>
  862. or any other supported logging framework of your choice.
  863. </p>
  864. <p>
  865. The ZooKeeper default <span class="codefrag filename">log4j.properties</span>
  866. file resides in the <span class="codefrag filename">conf</span> directory. Log4j requires that
  867. <span class="codefrag filename">log4j.properties</span> either be in the working directory
  868. (the directory from which ZooKeeper is run) or be accessible from the classpath.
  869. </p>
  870. <p>For more information about SLF4J, see
  871. <a href="http://www.slf4j.org/manual.html">its manual</a>.</p>
  872. <p>For more information about LOG4J, see
  873. <a href="http://logging.apache.org/log4j/1.2/manual.html#defaultInit">Log4j Default Initialization Procedure</a>
  874. of the log4j manual.</p>
  875. <a name="sc_troubleshooting"></a>
  876. <h3 class="h4">Troubleshooting</h3>
  877. <dl>
  878. <dt>
  879. <term> Server not coming up because of file corruption</term>
  880. </dt>
  881. <dd>
  882. <p>A server might not be able to read its database and fail to come up because of
  883. some file corruption in the transaction logs of the ZooKeeper server. You will
  884. see some IOException on loading ZooKeeper database. In such a case,
  885. make sure all the other servers in your ensemble are up and working. Use "stat"
  886. command on the command port to see if they are in good health. After you have verified that
  887. all the other servers of the ensemble are up, you can go ahead and clean the database
  888. of the corrupt server. Delete all the files in datadir/version-2 and datalogdir/version-2/.
  889. Restart the server.
  890. </p>
  891. </dd>
  892. </dl>
  893. <a name="sc_configuration"></a>
  894. <h3 class="h4">Configuration Parameters</h3>
  895. <p>ZooKeeper's behavior is governed by the ZooKeeper configuration
  896. file. This file is designed so that the exact same file can be used by
  897. all the servers that make up a ZooKeeper server assuming the disk
  898. layouts are the same. If servers use different configuration files, care
  899. must be taken to ensure that the list of servers in all of the different
  900. configuration files match.</p>
  901. <div class="note">
  902. <div class="label">Note</div>
  903. <div class="content">
  904. <p>In 3.5.0 and later, some of these parameters should be placed in
  905. a dynamic configuration file. If they are placed in the static
  906. configuration file, ZooKeeper will automatically move them over to the
  907. dynamic configuration file. See <a href="zookeeperReconfig.html">
  908. Dynamic Reconfiguration</a> for more information.</p>
  909. </div>
  910. </div>
  911. <a name="sc_minimumConfiguration"></a>
  912. <h4>Minimum Configuration</h4>
  913. <p>Here are the minimum configuration keywords that must be defined
  914. in the configuration file:</p>
  915. <dl>
  916. <dt>
  917. <term>clientPort</term>
  918. </dt>
  919. <dd>
  920. <p>the port to listen for client connections; that is, the
  921. port that clients attempt to connect to.</p>
  922. </dd>
  923. <dt>
  924. <term>secureClientPort</term>
  925. </dt>
  926. <dd>
  927. <p>the port to listen on for secure client connections using SSL.
  928. <strong>clientPort</strong> specifies
  929. the port for plaintext connections while <strong>
  930. secureClientPort</strong> specifies the port for SSL
  931. connections. Specifying both enables mixed-mode while omitting
  932. either will disable that mode.</p>
  933. <p>Note that SSL feature will be enabled when user plugs-in
  934. zookeeper.serverCnxnFactory, zookeeper.clientCnxnSocket as Netty.</p>
  935. </dd>
  936. <dt>
  937. <term>dataDir</term>
  938. </dt>
  939. <dd>
  940. <p>the location where ZooKeeper will store the in-memory
  941. database snapshots and, unless specified otherwise, the
  942. transaction log of updates to the database.</p>
  943. <div class="note">
  944. <div class="label">Note</div>
  945. <div class="content">
  946. <p>Be careful where you put the transaction log. A
  947. dedicated transaction log device is key to consistent good
  948. performance. Putting the log on a busy device will adversely
  949. effect performance.</p>
  950. </div>
  951. </div>
  952. </dd>
  953. <dt>
  954. <term>tickTime</term>
  955. </dt>
  956. <dd>
  957. <p>the length of a single tick, which is the basic time unit
  958. used by ZooKeeper, as measured in milliseconds. It is used to
  959. regulate heartbeats, and timeouts. For example, the minimum
  960. session timeout will be two ticks.</p>
  961. </dd>
  962. </dl>
  963. <a name="sc_advancedConfiguration"></a>
  964. <h4>Advanced Configuration</h4>
  965. <p>The configuration settings in the section are optional. You can
  966. use them to further fine tune the behaviour of your ZooKeeper servers.
  967. Some can also be set using Java system properties, generally of the
  968. form <em>zookeeper.keyword</em>. The exact system
  969. property, when available, is noted below.</p>
  970. <dl>
  971. <dt>
  972. <term>dataLogDir</term>
  973. </dt>
  974. <dd>
  975. <p>(No Java system property)</p>
  976. <p>This option will direct the machine to write the
  977. transaction log to the <strong>dataLogDir</strong> rather than the <strong>dataDir</strong>. This allows a dedicated log
  978. device to be used, and helps avoid competition between logging
  979. and snapshots.</p>
  980. <div class="note">
  981. <div class="label">Note</div>
  982. <div class="content">
  983. <p>Having a dedicated log device has a large impact on
  984. throughput and stable latencies. It is highly recommended to
  985. dedicate a log device and set <strong>dataLogDir</strong> to point to a directory on
  986. that device, and then make sure to point <strong>dataDir</strong> to a directory
  987. <em>not</em> residing on that device.</p>
  988. </div>
  989. </div>
  990. </dd>
  991. <dt>
  992. <term>globalOutstandingLimit</term>
  993. </dt>
  994. <dd>
  995. <p>(Java system property: <strong>zookeeper.globalOutstandingLimit.</strong>)</p>
  996. <p>Clients can submit requests faster than ZooKeeper can
  997. process them, especially if there are a lot of clients. To
  998. prevent ZooKeeper from running out of memory due to queued
  999. requests, ZooKeeper will throttle clients so that there is no
  1000. more than globalOutstandingLimit outstanding requests in the
  1001. system. The default limit is 1,000.</p>
  1002. </dd>
  1003. <dt>
  1004. <term>preAllocSize</term>
  1005. </dt>
  1006. <dd>
  1007. <p>(Java system property: <strong>zookeeper.preAllocSize</strong>)</p>
  1008. <p>To avoid seeks ZooKeeper allocates space in the
  1009. transaction log file in blocks of preAllocSize kilobytes. The
  1010. default block size is 64M. One reason for changing the size of
  1011. the blocks is to reduce the block size if snapshots are taken
  1012. more often. (Also, see <strong>snapCount</strong>).</p>
  1013. </dd>
  1014. <dt>
  1015. <term>snapCount</term>
  1016. </dt>
  1017. <dd>
  1018. <p>(Java system property: <strong>zookeeper.snapCount</strong>)</p>
  1019. <p>ZooKeeper records its transactions using snapshots and
  1020. a transaction log (think write-ahead log).The number of
  1021. transactions recorded in the transaction log before a snapshot
  1022. can be taken (and the transaction log rolled) is determined
  1023. by snapCount. In order to prevent all of the machines in the quorum
  1024. from taking a snapshot at the same time, each ZooKeeper server
  1025. will take a snapshot when the number of transactions in the transaction log
  1026. reaches a runtime generated random value in the [snapCount/2+1, snapCount]
  1027. range.The default snapCount is 100,000.</p>
  1028. </dd>
  1029. <dt>
  1030. <term>maxClientCnxns</term>
  1031. </dt>
  1032. <dd>
  1033. <p>(No Java system property)</p>
  1034. <p>Limits the number of concurrent connections (at the socket
  1035. level) that a single client, identified by IP address, may make
  1036. to a single member of the ZooKeeper ensemble. This is used to
  1037. prevent certain classes of DoS attacks, including file
  1038. descriptor exhaustion. The default is 60. Setting this to 0
  1039. entirely removes the limit on concurrent connections.</p>
  1040. </dd>
  1041. <dt>
  1042. <term>clientPortAddress</term>
  1043. </dt>
  1044. <dd>
  1045. <p>
  1046. <strong>New in 3.3.0:</strong> the
  1047. address (ipv4, ipv6 or hostname) to listen for client
  1048. connections; that is, the address that clients attempt
  1049. to connect to. This is optional, by default we bind in
  1050. such a way that any connection to the <strong>clientPort</strong> for any
  1051. address/interface/nic on the server will be
  1052. accepted.</p>
  1053. </dd>
  1054. <dt>
  1055. <term>minSessionTimeout</term>
  1056. </dt>
  1057. <dd>
  1058. <p>(No Java system property)</p>
  1059. <p>
  1060. <strong>New in 3.3.0:</strong> the
  1061. minimum session timeout in milliseconds that the server
  1062. will allow the client to negotiate. Defaults to 2 times
  1063. the <strong>tickTime</strong>.</p>
  1064. </dd>
  1065. <dt>
  1066. <term>maxSessionTimeout</term>
  1067. </dt>
  1068. <dd>
  1069. <p>(No Java system property)</p>
  1070. <p>
  1071. <strong>New in 3.3.0:</strong> the
  1072. maximum session timeout in milliseconds that the server
  1073. will allow the client to negotiate. Defaults to 20 times
  1074. the <strong>tickTime</strong>.</p>
  1075. </dd>
  1076. <dt>
  1077. <term>fsync.warningthresholdms</term>
  1078. </dt>
  1079. <dd>
  1080. <p>(Java system property: <strong>zookeeper.fsync.warningthresholdms</strong>)</p>
  1081. <p>
  1082. <strong>New in 3.3.4:</strong> A
  1083. warning message will be output to the log whenever an
  1084. fsync in the Transactional Log (WAL) takes longer than
  1085. this value. The values is specified in milliseconds and
  1086. defaults to 1000. This value can only be set as a
  1087. system property.</p>
  1088. </dd>
  1089. <dt>
  1090. <term>autopurge.snapRetainCount</term>
  1091. </dt>
  1092. <dd>
  1093. <p>(No Java system property)</p>
  1094. <p>
  1095. <strong>New in 3.4.0:</strong>
  1096. When enabled, ZooKeeper auto purge feature retains
  1097. the <strong>autopurge.snapRetainCount</strong> most
  1098. recent snapshots and the corresponding transaction logs in the
  1099. <strong>dataDir</strong> and <strong>dataLogDir</strong> respectively and deletes the rest.
  1100. Defaults to 3. Minimum value is 3.</p>
  1101. </dd>
  1102. <dt>
  1103. <term>autopurge.purgeInterval</term>
  1104. </dt>
  1105. <dd>
  1106. <p>(No Java system property)</p>
  1107. <p>
  1108. <strong>New in 3.4.0:</strong> The
  1109. time interval in hours for which the purge task has to
  1110. be triggered. Set to a positive integer (1 and above)
  1111. to enable the auto purging. Defaults to 0.</p>
  1112. </dd>
  1113. <dt>
  1114. <term>syncEnabled</term>
  1115. </dt>
  1116. <dd>
  1117. <p>(Java system property: <strong>zookeeper.observer.syncEnabled</strong>)</p>
  1118. <p>
  1119. <strong>New in 3.4.6, 3.5.0:</strong>
  1120. The observers now log transaction and write snapshot to disk
  1121. by default like the participants. This reduces the recovery time
  1122. of the observers on restart. Set to "false" to disable this
  1123. feature. Default is "true"</p>
  1124. </dd>
  1125. </dl>
  1126. <a name="sc_clusterOptions"></a>
  1127. <h4>Cluster Options</h4>
  1128. <p>The options in this section are designed for use with an ensemble
  1129. of servers -- that is, when deploying clusters of servers.</p>
  1130. <dl>
  1131. <dt>
  1132. <term>electionAlg</term>
  1133. </dt>
  1134. <dd>
  1135. <p>(No Java system property)</p>
  1136. <p>Election implementation to use. A value of "1" corresponds to the
  1137. non-authenticated UDP-based version of fast leader election, "2"
  1138. corresponds to the authenticated UDP-based version of fast
  1139. leader election, and "3" corresponds to TCP-based version of
  1140. fast leader election. Currently, algorithm 3 is the default.</p>
  1141. <div class="note">
  1142. <div class="label">Note</div>
  1143. <div class="content">
  1144. <p> The implementations of leader election 1, and 2 are now
  1145. <strong> deprecated </strong>. We have the intention
  1146. of removing them in the next release, at which point only the
  1147. FastLeaderElection will be available.
  1148. </p>
  1149. </div>
  1150. </div>
  1151. </dd>
  1152. <dt>
  1153. <term>initLimit</term>
  1154. </dt>
  1155. <dd>
  1156. <p>(No Java system property)</p>
  1157. <p>Amount of time, in ticks (see <a href="#id_tickTime">tickTime</a>), to allow followers to
  1158. connect and sync to a leader. Increased this value as needed, if
  1159. the amount of data managed by ZooKeeper is large.</p>
  1160. </dd>
  1161. <dt>
  1162. <term>leaderServes</term>
  1163. </dt>
  1164. <dd>
  1165. <p>(Java system property: zookeeper.<strong>leaderServes</strong>)</p>
  1166. <p>Leader accepts client connections. Default value is "yes".
  1167. The leader machine coordinates updates. For higher update
  1168. throughput at the slight expense of read throughput the leader
  1169. can be configured to not accept clients and focus on
  1170. coordination. The default to this option is yes, which means
  1171. that a leader will accept client connections.</p>
  1172. <div class="note">
  1173. <div class="label">Note</div>
  1174. <div class="content">
  1175. <p>Turning on leader selection is highly recommended when
  1176. you have more than three ZooKeeper servers in an ensemble.</p>
  1177. </div>
  1178. </div>
  1179. </dd>
  1180. <dt>
  1181. <term>server.x=[hostname]:nnnnn[:nnnnn], etc</term>
  1182. </dt>
  1183. <dd>
  1184. <p>(No Java system property)</p>
  1185. <p>servers making up the ZooKeeper ensemble. When the server
  1186. starts up, it determines which server it is by looking for the
  1187. file <span class="codefrag filename">myid</span> in the data directory. That file
  1188. contains the server number, in ASCII, and it should match
  1189. <strong>x</strong> in <strong>server.x</strong> in the left hand side of this
  1190. setting.</p>
  1191. <p>The list of servers that make up ZooKeeper servers that is
  1192. used by the clients must match the list of ZooKeeper servers
  1193. that each ZooKeeper server has.</p>
  1194. <p>There are two port numbers <strong>nnnnn</strong>.
  1195. The first followers use to connect to the leader, and the second is for
  1196. leader election. If you want to test multiple servers on a single machine, then
  1197. different ports can be used for each server.</p>
  1198. </dd>
  1199. <dt>
  1200. <term>syncLimit</term>
  1201. </dt>
  1202. <dd>
  1203. <p>(No Java system property)</p>
  1204. <p>Amount of time, in ticks (see <a href="#id_tickTime">tickTime</a>), to allow followers to sync
  1205. with ZooKeeper. If followers fall too far behind a leader, they
  1206. will be dropped.</p>
  1207. </dd>
  1208. <dt>
  1209. <term>group.x=nnnnn[:nnnnn]</term>
  1210. </dt>
  1211. <dd>
  1212. <p>(No Java system property)</p>
  1213. <p>Enables a hierarchical quorum construction."x" is a group identifier
  1214. and the numbers following the "=" sign correspond to server identifiers.
  1215. The left-hand side of the assignment is a colon-separated list of server
  1216. identifiers. Note that groups must be disjoint and the union of all groups
  1217. must be the ZooKeeper ensemble. </p>
  1218. <p> You will find an example <a href="zookeeperHierarchicalQuorums.html">here</a>
  1219. </p>
  1220. </dd>
  1221. <dt>
  1222. <term>weight.x=nnnnn</term>
  1223. </dt>
  1224. <dd>
  1225. <p>(No Java system property)</p>
  1226. <p>Used along with "group", it assigns a weight to a server when
  1227. forming quorums. Such a value corresponds to the weight of a server
  1228. when voting. There are a few parts of ZooKeeper that require voting
  1229. such as leader election and the atomic broadcast protocol. By default
  1230. the weight of server is 1. If the configuration defines groups, but not
  1231. weights, then a value of 1 will be assigned to all servers.
  1232. </p>
  1233. <p> You will find an example <a href="zookeeperHierarchicalQuorums.html">here</a>
  1234. </p>
  1235. </dd>
  1236. <dt>
  1237. <term>cnxTimeout</term>
  1238. </dt>
  1239. <dd>
  1240. <p>(Java system property: zookeeper.<strong>cnxTimeout</strong>)</p>
  1241. <p>Sets the timeout value for opening connections for leader election notifications.
  1242. Only applicable if you are using electionAlg 3.
  1243. </p>
  1244. <div class="note">
  1245. <div class="label">Note</div>
  1246. <div class="content">
  1247. <p>Default value is 5 seconds.</p>
  1248. </div>
  1249. </div>
  1250. </dd>
  1251. <dt>
  1252. <term>standaloneEnabled</term>
  1253. </dt>
  1254. <dd>
  1255. <p>(No Java system property)</p>
  1256. <p>
  1257. <strong>New in 3.5.0:</strong>
  1258. When set to false, a single server can be started in replicated
  1259. mode, a lone participant can run with observers, and a cluster
  1260. can reconfigure down to one node, and up from one node. The
  1261. default is true for backwards compatibility. It can be set
  1262. using QuorumPeerConfig's setStandaloneEnabled method or by
  1263. adding "standaloneEnabled=false" or "standaloneEnabled=true"
  1264. to a server's config file.
  1265. </p>
  1266. </dd>
  1267. <dt>
  1268. <term>reconfigEnabled</term>
  1269. </dt>
  1270. <dd>
  1271. <p>(No Java system property)</p>
  1272. <p>
  1273. <strong>New in 3.5.3:</strong>
  1274. This controls the enabling or disabling of
  1275. <a href="zookeeperReconfig.html">
  1276. Dynamic Reconfiguration</a> feature. When the feature
  1277. is enabled, users can perform reconfigure operations through
  1278. the ZooKeeper client API or through ZooKeeper command line tools
  1279. assuming users are authorized to perform such operations.
  1280. When the feature is disabled, no user, including the super user,
  1281. can perform a reconfiguration. Any attempt to reconfigure will return an error.
  1282. <strong>"reconfigEnabled"</strong> option can be set as
  1283. <strong>"reconfigEnabled=false"</strong> or
  1284. <strong>"reconfigEnabled=true"</strong>
  1285. to a server's config file, or using QuorumPeerConfig's
  1286. setReconfigEnabled method. The default value is false.
  1287. If present, the value should be consistent across every server in
  1288. the entire ensemble. Setting the value as true on some servers and false
  1289. on other servers will cause inconsistent behavior depending on which server
  1290. is elected as leader. If the leader has a setting of
  1291. <strong>"reconfigEnabled=true"</strong>, then the ensemble
  1292. will have reconfig feature enabled. If the leader has a setting of
  1293. <strong>"reconfigEnabled=false"</strong>, then the ensemble
  1294. will have reconfig feature disabled. It is thus recommended to have a consistent
  1295. value for <strong>"reconfigEnabled"</strong> across servers
  1296. in the ensemble.
  1297. </p>
  1298. </dd>
  1299. <dt>
  1300. <term>4lw.commands.whitelist</term>
  1301. </dt>
  1302. <dd>
  1303. <p>(Java system property: <strong>zookeeper.4lw.commands.whitelist</strong>)</p>
  1304. <p>
  1305. <strong>New in 3.5.3:</strong>
  1306. A list of comma separated <a href="#sc_4lw">Four Letter Words</a>
  1307. commands that user wants to use. A valid Four Letter Words
  1308. command must be put in this list else ZooKeeper server will
  1309. not enable the command.
  1310. By default the whitelist only contains "srvr" command
  1311. which zkServer.sh uses. The rest of four letter word commands are disabled
  1312. by default.
  1313. </p>
  1314. <p>Here's an example of the configuration that enables stat, ruok, conf, and isro
  1315. command while disabling the rest of Four Letter Words command:</p>
  1316. <pre class="code">
  1317. 4lw.commands.whitelist=stat, ruok, conf, isro
  1318. </pre>
  1319. <p>If you really need enable all four letter word commands by default, you can use
  1320. the asterisk option so you don't have to include every command one by one in the list.
  1321. As an example, this will enable all four letter word commands:
  1322. </p>
  1323. <pre class="code">
  1324. 4lw.commands.whitelist=*
  1325. </pre>
  1326. </dd>
  1327. <dt>
  1328. <term>tcpKeepAlive</term>
  1329. </dt>
  1330. <dd>
  1331. <p>(Java system property: <strong>zookeeper.tcpKeepAlive</strong>)</p>
  1332. <p>
  1333. <strong>New in 3.5.4:</strong>
  1334. Setting this to true sets the TCP keepAlive flag on the
  1335. sockets used by quorum members to perform elections.
  1336. This will allow for connections between quorum members to
  1337. remain up when there is network infrastructure that may
  1338. otherwise break them. Some NATs and firewalls may terminate
  1339. or lose state for long running or idle connections.</p>
  1340. <p> Enabling this option relies on OS level settings to work
  1341. properly, check your operating system's options regarding TCP
  1342. keepalive for more information. Defaults to
  1343. <strong>false</strong>.
  1344. </p>
  1345. </dd>
  1346. </dl>
  1347. <p></p>
  1348. <a name="sc_authOptions"></a>
  1349. <h4>Encryption, Authentication, Authorization Options</h4>
  1350. <p>The options in this section allow control over
  1351. encryption/authentication/authorization performed by the service.</p>
  1352. <dl>
  1353. <dt>
  1354. <term>DigestAuthenticationProvider.superDigest</term>
  1355. </dt>
  1356. <dd>
  1357. <p>(Java system property: <strong>zookeeper.DigestAuthenticationProvider.superDigest</strong>)</p>
  1358. <p>By default this feature is <strong>disabled</strong>
  1359. </p>
  1360. <p>
  1361. <strong>New in 3.2:</strong>
  1362. Enables a ZooKeeper ensemble administrator to access the
  1363. znode hierarchy as a "super" user. In particular no ACL
  1364. checking occurs for a user authenticated as
  1365. super.</p>
  1366. <p>org.apache.zookeeper.server.auth.DigestAuthenticationProvider
  1367. can be used to generate the superDigest, call it with
  1368. one parameter of "super:&lt;password&gt;". Provide the
  1369. generated "super:&lt;data&gt;" as the system property value
  1370. when starting each server of the ensemble.</p>
  1371. <p>When authenticating to a ZooKeeper server (from a
  1372. ZooKeeper client) pass a scheme of "digest" and authdata
  1373. of "super:&lt;password&gt;". Note that digest auth passes
  1374. the authdata in plaintext to the server, it would be
  1375. prudent to use this authentication method only on
  1376. localhost (not over the network) or over an encrypted
  1377. connection.</p>
  1378. </dd>
  1379. <dt>
  1380. <term>X509AuthenticationProvider.superUser</term>
  1381. </dt>
  1382. <dd>
  1383. <p>(Java system property: <strong>zookeeper.X509AuthenticationProvider.superUser</strong>)</p>
  1384. <p>The SSL-backed way to enable a ZooKeeper ensemble
  1385. administrator to access the znode hierarchy as a "super" user.
  1386. When this parameter is set to an X500 principal name, only an
  1387. authenticated client with that principal will be able to bypass
  1388. ACL checking and have full privileges to all znodes.</p>
  1389. </dd>
  1390. <dt>
  1391. <term>zookeeper.superUser</term>
  1392. </dt>
  1393. <dd>
  1394. <p>(Java system property: <strong>zookeeper.superUser</strong>)</p>
  1395. <p>Similar to <strong>zookeeper.X509AuthenticationProvider.superUser</strong>
  1396. but is generic for SASL based logins. It stores the name of
  1397. a user that can access the znode hierarchy as a "super" user.
  1398. </p>
  1399. </dd>
  1400. <dt>
  1401. <term>ssl.keyStore.location and ssl.keyStore.password</term>
  1402. </dt>
  1403. <dd>
  1404. <p>(Java system properties: <strong>
  1405. zookeeper.ssl.keyStore.location</strong> and <strong>zookeeper.ssl.keyStore.password</strong>)</p>
  1406. <p>Specifies the file path to a JKS containing the local
  1407. credentials to be used for SSL connections, and the
  1408. password to unlock the file.</p>
  1409. </dd>
  1410. <dt>
  1411. <term>ssl.trustStore.location and ssl.trustStore.password</term>
  1412. </dt>
  1413. <dd>
  1414. <p>(Java system properties: <strong>
  1415. zookeeper.ssl.trustStore.location</strong> and <strong>zookeeper.ssl.trustStore.password</strong>)</p>
  1416. <p>Specifies the file path to a JKS containing the remote
  1417. credentials to be used for SSL connections, and the
  1418. password to unlock the file.</p>
  1419. </dd>
  1420. <dt>
  1421. <term>ssl.authProvider</term>
  1422. </dt>
  1423. <dd>
  1424. <p>(Java system property: <strong>zookeeper.ssl.authProvider</strong>)</p>
  1425. <p>Specifies a subclass of <strong>
  1426. org.apache.zookeeper.auth.X509AuthenticationProvider</strong>
  1427. to use for secure client authentication. This is useful in
  1428. certificate key infrastructures that do not use JKS. It may be
  1429. necessary to extend <strong>javax.net.ssl.X509KeyManager
  1430. </strong> and <strong>javax.net.ssl.X509TrustManager</strong>
  1431. to get the desired behavior from the SSL stack. To configure the
  1432. ZooKeeper server to use the custom provider for authentication,
  1433. choose a scheme name for the custom AuthenticationProvider and
  1434. set the property <strong>zookeeper.authProvider.[scheme]
  1435. </strong> to the fully-qualified class name of the custom
  1436. implementation. This will load the provider into the ProviderRegistry.
  1437. Then set this property <strong>
  1438. zookeeper.ssl.authProvider=[scheme]</strong> and that provider
  1439. will be used for secure authentication.</p>
  1440. </dd>
  1441. </dl>
  1442. <a name="Experimental+Options%2FFeatures"></a>
  1443. <h4>Experimental Options/Features</h4>
  1444. <p>New features that are currently considered experimental.</p>
  1445. <dl>
  1446. <dt>
  1447. <term>Read Only Mode Server</term>
  1448. </dt>
  1449. <dd>
  1450. <p>(Java system property: <strong>readonlymode.enabled</strong>)</p>
  1451. <p>
  1452. <strong>New in 3.4.0:</strong>
  1453. Setting this value to true enables Read Only Mode server
  1454. support (disabled by default). ROM allows clients
  1455. sessions which requested ROM support to connect to the
  1456. server even when the server might be partitioned from
  1457. the quorum. In this mode ROM clients can still read
  1458. values from the ZK service, but will be unable to write
  1459. values and see changes from other clients. See
  1460. ZOOKEEPER-784 for more details.
  1461. </p>
  1462. </dd>
  1463. </dl>
  1464. <a name="Unsafe+Options"></a>
  1465. <h4>Unsafe Options</h4>
  1466. <p>The following options can be useful, but be careful when you use
  1467. them. The risk of each is explained along with the explanation of what
  1468. the variable does.</p>
  1469. <dl>
  1470. <dt>
  1471. <term>forceSync</term>
  1472. </dt>
  1473. <dd>
  1474. <p>(Java system property: <strong>zookeeper.forceSync</strong>)</p>
  1475. <p>Requires updates to be synced to media of the transaction
  1476. log before finishing processing the update. If this option is
  1477. set to no, ZooKeeper will not require updates to be synced to
  1478. the media.</p>
  1479. </dd>
  1480. <dt>
  1481. <term>jute.maxbuffer:</term>
  1482. </dt>
  1483. <dd>
  1484. <p>(Java system property:<strong>
  1485. jute.maxbuffer</strong>)</p>
  1486. <p>This option can only be set as a Java system property.
  1487. There is no zookeeper prefix on it. It specifies the maximum
  1488. size of the data that can be stored in a znode. The default is
  1489. 0xfffff, or just under 1M. If this option is changed, the system
  1490. property must be set on all servers and clients otherwise
  1491. problems will arise. This is really a sanity check. ZooKeeper is
  1492. designed to store data on the order of kilobytes in size.</p>
  1493. </dd>
  1494. <dt>
  1495. <term>skipACL</term>
  1496. </dt>
  1497. <dd>
  1498. <p>(Java system property: <strong>zookeeper.skipACL</strong>)</p>
  1499. <p>Skips ACL checks. This results in a boost in throughput,
  1500. but opens up full access to the data tree to everyone.</p>
  1501. </dd>
  1502. <dt>
  1503. <term>quorumListenOnAllIPs</term>
  1504. </dt>
  1505. <dd>
  1506. <p>When set to true the ZooKeeper server will listen
  1507. for connections from its peers on all available IP addresses,
  1508. and not only the address configured in the server list of the
  1509. configuration file. It affects the connections handling the
  1510. ZAB protocol and the Fast Leader Election protocol. Default
  1511. value is <strong>false</strong>.</p>
  1512. </dd>
  1513. </dl>
  1514. <a name="Disabling+data+directory+autocreation"></a>
  1515. <h4>Disabling data directory autocreation</h4>
  1516. <p>
  1517. <strong>New in 3.5:</strong> The default
  1518. behavior of a ZooKeeper server is to automatically create the
  1519. data directory (specified in the configuration file) when
  1520. started if that directory does not already exist. This can be
  1521. inconvenient and even dangerous in some cases. Take the case
  1522. where a configuration change is made to a running server,
  1523. wherein the <strong>dataDir</strong> parameter
  1524. is accidentally changed. When the ZooKeeper server is
  1525. restarted it will create this non-existent directory and begin
  1526. serving - with an empty znode namespace. This scenario can
  1527. result in an effective "split brain" situation (i.e. data in
  1528. both the new invalid directory and the original valid data
  1529. store). As such is would be good to have an option to turn off
  1530. this autocreate behavior. In general for production
  1531. environments this should be done, unfortunately however the
  1532. default legacy behavior cannot be changed at this point and
  1533. therefore this must be done on a case by case basis. This is
  1534. left to users and to packagers of ZooKeeper distributions.
  1535. </p>
  1536. <p>When running <strong>zkServer.sh</strong> autocreate can be disabled
  1537. by setting the environment variable <strong>ZOO_DATADIR_AUTOCREATE_DISABLE</strong> to 1.
  1538. When running ZooKeeper servers directly from class files this
  1539. can be accomplished by setting <strong>zookeeper.datadir.autocreate=false</strong> on
  1540. the java command line, i.e. <strong>-Dzookeeper.datadir.autocreate=false</strong>
  1541. </p>
  1542. <p>When this feature is disabled, and the ZooKeeper server
  1543. determines that the required directories do not exist it will
  1544. generate an error and refuse to start.
  1545. </p>
  1546. <p>A new script <strong>zkServer-initialize.sh</strong> is provided to
  1547. support this new feature. If autocreate is disabled it is
  1548. necessary for the user to first install ZooKeeper, then create
  1549. the data directory (and potentially txnlog directory), and
  1550. then start the server. Otherwise as mentioned in the previous
  1551. paragraph the server will not start. Running <strong>zkServer-initialize.sh</strong> will create the
  1552. required directories, and optionally setup the myid file
  1553. (optional command line parameter). This script can be used
  1554. even if the autocreate feature itself is not used, and will
  1555. likely be of use to users as this (setup, including creation
  1556. of the myid file) has been an issue for users in the past.
  1557. Note that this script ensures the data directories exist only,
  1558. it does not create a config file, but rather requires a config
  1559. file to be available in order to execute.
  1560. </p>
  1561. <a name="sc_db_existence_validation"></a>
  1562. <h4>Enabling db existence validation</h4>
  1563. <p>
  1564. <strong>New in 3.6.0:</strong> The default
  1565. behavior of a ZooKeeper server on startup when no data tree
  1566. is found is to set zxid to zero and join the quorum as a
  1567. voting member. This can be dangerous if some event (e.g. a
  1568. rogue 'rm -rf') has removed the data directory while the
  1569. server was down since this server may help elect a leader
  1570. that is missing transactions. Enabling db existence validation
  1571. will change the behavior on startup when no data tree is
  1572. found: the server joins the ensemble as a non-voting participant
  1573. until it is able to sync with the leader and acquire an up-to-date
  1574. version of the ensemble data. To indicate an empty data tree is
  1575. expected (ensemble creation), the user should place a file
  1576. 'initialize' in the same directory as 'myid'. This file will
  1577. be detected and deleted by the server on startup.
  1578. </p>
  1579. <p> Initialization validation can be enabled when running
  1580. ZooKeeper servers directly from class files by setting
  1581. <strong>zookeeper.db.autocreate=false</strong>
  1582. on the java command line, i.e.
  1583. <strong>-Dzookeeper.db.autocreate=false</strong>.
  1584. Running <strong>zkServer-initialize.sh</strong>
  1585. will create the required initialization file.
  1586. </p>
  1587. <a name="sc_performance_options"></a>
  1588. <h4>Performance Tuning Options</h4>
  1589. <p>
  1590. <strong>New in 3.5.0:</strong> Several subsystems have been reworked
  1591. to improve read throughput. This includes multi-threading of the NIO communication subsystem and
  1592. request processing pipeline (Commit Processor). NIO is the default client/server communication
  1593. subsystem. Its threading model comprises 1 acceptor thread, 1-N selector threads and 0-M
  1594. socket I/O worker threads. In the request processing pipeline the system can be configured
  1595. to process multiple read request at once while maintaining the same consistency guarantee
  1596. (same-session read-after-write). The Commit Processor threading model comprises 1 main
  1597. thread and 0-N worker threads.
  1598. </p>
  1599. <p>
  1600. The default values are aimed at maximizing read throughput on a dedicated ZooKeeper machine.
  1601. Both subsystems need to have sufficient amount of threads to achieve peak read throughput.
  1602. </p>
  1603. <dl>
  1604. <dt>
  1605. <term>zookeeper.nio.numSelectorThreads</term>
  1606. </dt>
  1607. <dd>
  1608. <p>(Java system property only: <strong>zookeeper.nio.numSelectorThreads</strong>)
  1609. </p>
  1610. <p>
  1611. <strong>New in 3.5.0:</strong>
  1612. Number of NIO selector threads. At least 1 selector thread required.
  1613. It is recommended to use more than one selector for large numbers
  1614. of client connections. The default value is sqrt( number of cpu cores / 2 ).
  1615. </p>
  1616. </dd>
  1617. <dt>
  1618. <term>zookeeper.nio.numWorkerThreads</term>
  1619. </dt>
  1620. <dd>
  1621. <p>(Java system property only: <strong>zookeeper.nio.numWorkerThreads</strong>)
  1622. </p>
  1623. <p>
  1624. <strong>New in 3.5.0:</strong>
  1625. Number of NIO worker threads. If configured with 0 worker threads, the selector threads
  1626. do the socket I/O directly. The default value is 2 times the number of cpu cores.
  1627. </p>
  1628. </dd>
  1629. <dt>
  1630. <term>zookeeper.commitProcessor.numWorkerThreads</term>
  1631. </dt>
  1632. <dd>
  1633. <p>(Java system property only: <strong>zookeeper.commitProcessor.numWorkerThreads</strong>)
  1634. </p>
  1635. <p>
  1636. <strong>New in 3.5.0:</strong>
  1637. Number of Commit Processor worker threads. If configured with 0 worker threads, the main thread
  1638. will process the request directly. The default value is the number of cpu cores.
  1639. </p>
  1640. </dd>
  1641. <dt>
  1642. <term>znode.container.checkIntervalMs</term>
  1643. </dt>
  1644. <dd>
  1645. <p>(Java system property only)</p>
  1646. <p>
  1647. <strong>New in 3.6.0:</strong> The
  1648. time interval in milliseconds for each check of candidate container
  1649. and ttl nodes. Default is "60000".</p>
  1650. </dd>
  1651. <dt>
  1652. <term>znode.container.maxPerMinute</term>
  1653. </dt>
  1654. <dd>
  1655. <p>(Java system property only)</p>
  1656. <p>
  1657. <strong>New in 3.6.0:</strong> The
  1658. maximum number of container and ttl nodes that can be deleted per
  1659. minute. This prevents herding during container deletion.
  1660. Default is "10000".</p>
  1661. </dd>
  1662. </dl>
  1663. <a name="Communication+using+the+Netty+framework"></a>
  1664. <h4>Communication using the Netty framework</h4>
  1665. <p>
  1666. <a href="http://netty.io">Netty</a>
  1667. is an NIO based client/server communication framework, it
  1668. simplifies (over NIO being used directly) many of the
  1669. complexities of network level communication for java
  1670. applications. Additionally the Netty framework has built
  1671. in support for encryption (SSL) and authentication
  1672. (certificates). These are optional features and can be
  1673. turned on or off individually.
  1674. </p>
  1675. <p>In versions 3.5+, a ZooKeeper server can use Netty
  1676. instead of NIO (default option) by setting the environment
  1677. variable <strong>zookeeper.serverCnxnFactory</strong>
  1678. to <strong>org.apache.zookeeper.server.NettyServerCnxnFactory</strong>;
  1679. for the client, set <strong>zookeeper.clientCnxnSocket</strong>
  1680. to <strong>org.apache.zookeeper.ClientCnxnSocketNetty</strong>.
  1681. </p>
  1682. <p>
  1683. TBD - tuning options for netty - currently there are none that are netty specific but we should add some. Esp around max bound on the number of reader worker threads netty creates.
  1684. </p>
  1685. <p>
  1686. TBD - how to manage encryption
  1687. </p>
  1688. <p>
  1689. TBD - how to manage certificates
  1690. </p>
  1691. <a name="sc_adminserver_config"></a>
  1692. <h4>AdminServer configuration</h4>
  1693. <p>
  1694. <strong>New in 3.5.0:</strong> The following
  1695. options are used to configure the <a href="#sc_adminserver">AdminServer</a>.</p>
  1696. <dl>
  1697. <dt>
  1698. <term>admin.enableServer</term>
  1699. </dt>
  1700. <dd>
  1701. <p>(Java system property: <strong>zookeeper.admin.enableServer</strong>)</p>
  1702. <p>Set to "false" to disable the AdminServer. By default the
  1703. AdminServer is enabled.</p>
  1704. </dd>
  1705. <dt>
  1706. <term>admin.serverAddress</term>
  1707. </dt>
  1708. <dd>
  1709. <p>(Java system property: <strong>zookeeper.admin.serverAddress</strong>)</p>
  1710. <p>The address the embedded Jetty server listens on. Defaults to 0.0.0.0.</p>
  1711. </dd>
  1712. <dt>
  1713. <term>admin.serverPort</term>
  1714. </dt>
  1715. <dd>
  1716. <p>(Java system property: <strong>zookeeper.admin.serverPort</strong>)</p>
  1717. <p>The port the embedded Jetty server listens on. Defaults to 8080.</p>
  1718. </dd>
  1719. <dt>
  1720. <term>admin.idleTimeout</term>
  1721. </dt>
  1722. <dd>
  1723. <p>(Java system property: <strong>zookeeper.admin.idleTimeout</strong>)</p>
  1724. <p>Set the maximum idle time in milliseconds that a connection can wait
  1725. before sending or receiving data. Defaults to 30000 ms.</p>
  1726. </dd>
  1727. <dt>
  1728. <term>admin.commandURL</term>
  1729. </dt>
  1730. <dd>
  1731. <p>(Java system property: <strong>zookeeper.admin.commandURL</strong>)</p>
  1732. <p>The URL for listing and issuing commands relative to the
  1733. root URL. Defaults to "/commands".</p>
  1734. </dd>
  1735. </dl>
  1736. <a name="sc_zkCommands"></a>
  1737. <h3 class="h4">ZooKeeper Commands</h3>
  1738. <a name="sc_4lw"></a>
  1739. <h4>The Four Letter Words</h4>
  1740. <p>ZooKeeper responds to a small set of commands. Each command is
  1741. composed of four letters. You issue the commands to ZooKeeper via telnet
  1742. or nc, at the client port.</p>
  1743. <p>Three of the more interesting commands: "stat" gives some
  1744. general information about the server and connected clients,
  1745. while "srvr" and "cons" give extended details on server and
  1746. connections respectively.</p>
  1747. <p>
  1748. <strong>New in 3.5.3:</strong>
  1749. Four Letter Words need to be explicitly white listed before using.
  1750. Please refer <strong>4lw.commands.whitelist</strong>
  1751. described in <a href="#sc_clusterOptions">
  1752. cluster configuration section</a> for details.
  1753. Moving forward, Four Letter Words will be deprecated, please use
  1754. <a href="#sc_adminserver">AdminServer</a> instead.
  1755. </p>
  1756. <dl>
  1757. <dt>
  1758. <term>conf</term>
  1759. </dt>
  1760. <dd>
  1761. <p>
  1762. <strong>New in 3.3.0:</strong> Print
  1763. details about serving configuration.</p>
  1764. </dd>
  1765. <dt>
  1766. <term>cons</term>
  1767. </dt>
  1768. <dd>
  1769. <p>
  1770. <strong>New in 3.3.0:</strong> List
  1771. full connection/session details for all clients connected
  1772. to this server. Includes information on numbers of packets
  1773. received/sent, session id, operation latencies, last
  1774. operation performed, etc...</p>
  1775. </dd>
  1776. <dt>
  1777. <term>crst</term>
  1778. </dt>
  1779. <dd>
  1780. <p>
  1781. <strong>New in 3.3.0:</strong> Reset
  1782. connection/session statistics for all connections.</p>
  1783. </dd>
  1784. <dt>
  1785. <term>dump</term>
  1786. </dt>
  1787. <dd>
  1788. <p>Lists the outstanding sessions and ephemeral nodes. This
  1789. only works on the leader.</p>
  1790. </dd>
  1791. <dt>
  1792. <term>envi</term>
  1793. </dt>
  1794. <dd>
  1795. <p>Print details about serving environment</p>
  1796. </dd>
  1797. <dt>
  1798. <term>ruok</term>
  1799. </dt>
  1800. <dd>
  1801. <p>Tests if server is running in a non-error state. The server
  1802. will respond with imok if it is running. Otherwise it will not
  1803. respond at all.</p>
  1804. <p>A response of "imok" does not necessarily indicate that the
  1805. server has joined the quorum, just that the server process is active
  1806. and bound to the specified client port. Use "stat" for details on
  1807. state wrt quorum and client connection information.</p>
  1808. </dd>
  1809. <dt>
  1810. <term>srst</term>
  1811. </dt>
  1812. <dd>
  1813. <p>Reset server statistics.</p>
  1814. </dd>
  1815. <dt>
  1816. <term>srvr</term>
  1817. </dt>
  1818. <dd>
  1819. <p>
  1820. <strong>New in 3.3.0:</strong> Lists
  1821. full details for the server.</p>
  1822. </dd>
  1823. <dt>
  1824. <term>stat</term>
  1825. </dt>
  1826. <dd>
  1827. <p>Lists brief details for the server and connected
  1828. clients.</p>
  1829. </dd>
  1830. <dt>
  1831. <term>wchs</term>
  1832. </dt>
  1833. <dd>
  1834. <p>
  1835. <strong>New in 3.3.0:</strong> Lists
  1836. brief information on watches for the server.</p>
  1837. </dd>
  1838. <dt>
  1839. <term>wchc</term>
  1840. </dt>
  1841. <dd>
  1842. <p>
  1843. <strong>New in 3.3.0:</strong> Lists
  1844. detailed information on watches for the server, by
  1845. session. This outputs a list of sessions(connections)
  1846. with associated watches (paths). Note, depending on the
  1847. number of watches this operation may be expensive (ie
  1848. impact server performance), use it carefully.</p>
  1849. </dd>
  1850. <dt>
  1851. <term>dirs</term>
  1852. </dt>
  1853. <dd>
  1854. <p>
  1855. <strong>New in 3.5.1:</strong>
  1856. Shows the total size of snapshot and log files in bytes
  1857. </p>
  1858. </dd>
  1859. <dt>
  1860. <term>wchp</term>
  1861. </dt>
  1862. <dd>
  1863. <p>
  1864. <strong>New in 3.3.0:</strong> Lists
  1865. detailed information on watches for the server, by path.
  1866. This outputs a list of paths (znodes) with associated
  1867. sessions. Note, depending on the number of watches this
  1868. operation may be expensive (ie impact server performance),
  1869. use it carefully.</p>
  1870. </dd>
  1871. <dt>
  1872. <term>mntr</term>
  1873. </dt>
  1874. <dd>
  1875. <p>
  1876. <strong>New in 3.4.0:</strong> Outputs a list
  1877. of variables that could be used for monitoring the health of the cluster.</p>
  1878. <pre class="code">$ echo mntr | nc localhost 2185
  1879. zk_version 3.4.0
  1880. zk_avg_latency 0
  1881. zk_max_latency 0
  1882. zk_min_latency 0
  1883. zk_packets_received 70
  1884. zk_packets_sent 69
  1885. zk_outstanding_requests 0
  1886. zk_server_state leader
  1887. zk_znode_count 4
  1888. zk_watch_count 0
  1889. zk_ephemerals_count 0
  1890. zk_approximate_data_size 27
  1891. zk_followers 4 - only exposed by the Leader
  1892. zk_synced_followers 4 - only exposed by the Leader
  1893. zk_pending_syncs 0 - only exposed by the Leader
  1894. zk_open_file_descriptor_count 23 - only available on Unix platforms
  1895. zk_max_file_descriptor_count 1024 - only available on Unix platforms
  1896. </pre>
  1897. <p>The output is compatible with java properties format and the content
  1898. may change over time (new keys added). Your scripts should expect changes.</p>
  1899. <p>ATTENTION: Some of the keys are platform specific and some of the keys are only exported by the Leader. </p>
  1900. <p>The output contains multiple lines with the following format:</p>
  1901. <pre class="code">key \t value</pre>
  1902. </dd>
  1903. <dt>
  1904. <term>isro</term>
  1905. </dt>
  1906. <dd>
  1907. <p>
  1908. <strong>New in 3.4.0:</strong> Tests if
  1909. server is running in read-only mode. The server will respond with
  1910. "ro" if in read-only mode or "rw" if not in read-only mode.</p>
  1911. </dd>
  1912. <dt>
  1913. <term>gtmk</term>
  1914. </dt>
  1915. <dd>
  1916. <p>Gets the current trace mask as a 64-bit signed long value in
  1917. decimal format. See <span class="codefrag command">stmk</span> for an explanation of
  1918. the possible values.</p>
  1919. </dd>
  1920. <dt>
  1921. <term>stmk</term>
  1922. </dt>
  1923. <dd>
  1924. <p>Sets the current trace mask. The trace mask is 64 bits,
  1925. where each bit enables or disables a specific category of trace
  1926. logging on the server. Log4J must be configured to enable
  1927. <span class="codefrag command">TRACE</span> level first in order to see trace logging
  1928. messages. The bits of the trace mask correspond to the following
  1929. trace logging categories.</p>
  1930. <table class="ForrestTable" cellspacing="1" cellpadding="4">
  1931. <caption>Trace Mask Bit Values</caption>
  1932. <title>Trace Mask Bit Values</title>
  1933. <tr>
  1934. <td>0b0000000000</td>
  1935. <td>Unused, reserved for future use.</td>
  1936. </tr>
  1937. <tr>
  1938. <td>0b0000000010</td>
  1939. <td>Logs client requests, excluding ping
  1940. requests.</td>
  1941. </tr>
  1942. <tr>
  1943. <td>0b0000000100</td>
  1944. <td>Unused, reserved for future use.</td>
  1945. </tr>
  1946. <tr>
  1947. <td>0b0000001000</td>
  1948. <td>Logs client ping requests.</td>
  1949. </tr>
  1950. <tr>
  1951. <td>0b0000010000</td>
  1952. <td>Logs packets received from the quorum peer that is
  1953. the current leader, excluding ping requests.</td>
  1954. </tr>
  1955. <tr>
  1956. <td>0b0000100000</td>
  1957. <td>Logs addition, removal and validation of client
  1958. sessions.</td>
  1959. </tr>
  1960. <tr>
  1961. <td>0b0001000000</td>
  1962. <td>Logs delivery of watch events to client
  1963. sessions.</td>
  1964. </tr>
  1965. <tr>
  1966. <td>0b0010000000</td>
  1967. <td>Logs ping packets received from the quorum peer
  1968. that is the current leader.</td>
  1969. </tr>
  1970. <tr>
  1971. <td>0b0100000000</td>
  1972. <td>Unused, reserved for future use.</td>
  1973. </tr>
  1974. <tr>
  1975. <td>0b1000000000</td>
  1976. <td>Unused, reserved for future use.</td>
  1977. </tr>
  1978. </table>
  1979. <p>All remaining bits in the 64-bit value are unused and
  1980. reserved for future use. Multiple trace logging categories are
  1981. specified by calculating the bitwise OR of the documented values.
  1982. The default trace mask is 0b0100110010. Thus, by default, trace
  1983. logging includes client requests, packets received from the
  1984. leader and sessions.</p>
  1985. <p>To set a different trace mask, send a request containing the
  1986. <span class="codefrag command">stmk</span> four-letter word followed by the trace
  1987. mask represented as a 64-bit signed long value. This example uses
  1988. the Perl <span class="codefrag command">pack</span> function to construct a trace
  1989. mask that enables all trace logging categories described above and
  1990. convert it to a 64-bit signed long value with big-endian byte
  1991. order. The result is appended to <span class="codefrag command">stmk</span> and sent
  1992. to the server using netcat. The server responds with the new
  1993. trace mask in decimal format.</p>
  1994. <pre class="code">$ perl -e "print 'stmk', pack('q&gt;', 0b0011111010)" | nc localhost 2181
  1995. 250
  1996. </pre>
  1997. </dd>
  1998. </dl>
  1999. <p>Here's an example of the <strong>ruok</strong>
  2000. command:</p>
  2001. <pre class="code">$ echo ruok | nc 127.0.0.1 5111
  2002. imok
  2003. </pre>
  2004. <a name="sc_adminserver"></a>
  2005. <h4>The AdminServer</h4>
  2006. <p>
  2007. <strong>New in 3.5.0: </strong>The AdminServer is
  2008. an embedded Jetty server that provides an HTTP interface to the four
  2009. letter word commands. By default, the server is started on port 8080,
  2010. and commands are issued by going to the URL "/commands/[command name]",
  2011. e.g., http://localhost:8080/commands/stat. The command response is
  2012. returned as JSON. Unlike the original protocol, commands are not
  2013. restricted to four-letter names, and commands can have multiple names;
  2014. for instance, "stmk" can also be referred to as "set_trace_mask". To
  2015. view a list of all available commands, point a browser to the URL
  2016. /commands (e.g., http://localhost:8080/commands). See the <a href="#sc_adminserver_config">AdminServer configuration options</a>
  2017. for how to change the port and URLs.</p>
  2018. <p>The AdminServer is enabled by default, but can be disabled by either:</p>
  2019. <ul>
  2020. <li>
  2021. <p>Setting the zookeeper.admin.enableServer system
  2022. property to false.</p>
  2023. </li>
  2024. <li>
  2025. <p>Removing Jetty from the classpath. (This option is
  2026. useful if you would like to override ZooKeeper's jetty
  2027. dependency.)</p>
  2028. </li>
  2029. </ul>
  2030. <p>Note that the TCP four letter word interface is still available if
  2031. the AdminServer is disabled.</p>
  2032. <a name="sc_dataFileManagement"></a>
  2033. <h3 class="h4">Data File Management</h3>
  2034. <p>ZooKeeper stores its data in a data directory and its transaction
  2035. log in a transaction log directory. By default these two directories are
  2036. the same. The server can (and should) be configured to store the
  2037. transaction log files in a separate directory than the data files.
  2038. Throughput increases and latency decreases when transaction logs reside
  2039. on a dedicated log devices.</p>
  2040. <a name="The+Data+Directory"></a>
  2041. <h4>The Data Directory</h4>
  2042. <p>This directory has two or three files in it:</p>
  2043. <ul>
  2044. <li>
  2045. <p>
  2046. <span class="codefrag filename">myid</span> - contains a single integer in
  2047. human readable ASCII text that represents the server id.</p>
  2048. </li>
  2049. <li>
  2050. <p>
  2051. <span class="codefrag filename">initialize</span> - presence indicates lack of
  2052. data tree is expected. Cleaned up once data tree is created.</p>
  2053. </li>
  2054. <li>
  2055. <p>
  2056. <span class="codefrag filename">snapshot.&lt;zxid&gt;</span> - holds the fuzzy
  2057. snapshot of a data tree.</p>
  2058. </li>
  2059. </ul>
  2060. <p>Each ZooKeeper server has a unique id. This id is used in two
  2061. places: the <span class="codefrag filename">myid</span> file and the configuration file.
  2062. The <span class="codefrag filename">myid</span> file identifies the server that
  2063. corresponds to the given data directory. The configuration file lists
  2064. the contact information for each server identified by its server id.
  2065. When a ZooKeeper server instance starts, it reads its id from the
  2066. <span class="codefrag filename">myid</span> file and then, using that id, reads from the
  2067. configuration file, looking up the port on which it should
  2068. listen.</p>
  2069. <p>The <span class="codefrag filename">snapshot</span> files stored in the data
  2070. directory are fuzzy snapshots in the sense that during the time the
  2071. ZooKeeper server is taking the snapshot, updates are occurring to the
  2072. data tree. The suffix of the <span class="codefrag filename">snapshot</span> file names
  2073. is the <em>zxid</em>, the ZooKeeper transaction id, of the
  2074. last committed transaction at the start of the snapshot. Thus, the
  2075. snapshot includes a subset of the updates to the data tree that
  2076. occurred while the snapshot was in process. The snapshot, then, may
  2077. not correspond to any data tree that actually existed, and for this
  2078. reason we refer to it as a fuzzy snapshot. Still, ZooKeeper can
  2079. recover using this snapshot because it takes advantage of the
  2080. idempotent nature of its updates. By replaying the transaction log
  2081. against fuzzy snapshots ZooKeeper gets the state of the system at the
  2082. end of the log.</p>
  2083. <a name="The+Log+Directory"></a>
  2084. <h4>The Log Directory</h4>
  2085. <p>The Log Directory contains the ZooKeeper transaction logs.
  2086. Before any update takes place, ZooKeeper ensures that the transaction
  2087. that represents the update is written to non-volatile storage. A new
  2088. log file is started when the number of transactions written to the
  2089. current log file reaches a (variable) threshold. The threshold is
  2090. computed using the same parameter which influences the frequency of
  2091. snapshotting (see snapCount above). The log file's suffix is the first
  2092. zxid written to that log.</p>
  2093. <a name="sc_filemanagement"></a>
  2094. <h4>File Management</h4>
  2095. <p>The format of snapshot and log files does not change between
  2096. standalone ZooKeeper servers and different configurations of
  2097. replicated ZooKeeper servers. Therefore, you can pull these files from
  2098. a running replicated ZooKeeper server to a development machine with a
  2099. stand-alone ZooKeeper server for trouble shooting.</p>
  2100. <p>Using older log and snapshot files, you can look at the previous
  2101. state of ZooKeeper servers and even restore that state. The
  2102. LogFormatter class allows an administrator to look at the transactions
  2103. in a log.</p>
  2104. <p>The ZooKeeper server creates snapshot and log files, but
  2105. never deletes them. The retention policy of the data and log
  2106. files is implemented outside of the ZooKeeper server. The
  2107. server itself only needs the latest complete fuzzy snapshot, all log
  2108. files following it, and the last log file preceding it. The latter
  2109. requirement is necessary to include updates which happened after this
  2110. snapshot was started but went into the existing log file at that time.
  2111. This is possible because snapshotting and rolling over of logs
  2112. proceed somewhat independently in ZooKeeper. See the
  2113. <a href="#sc_maintenance">maintenance</a> section in
  2114. this document for more details on setting a retention policy
  2115. and maintenance of ZooKeeper storage.
  2116. </p>
  2117. <div class="note">
  2118. <div class="label">Note</div>
  2119. <div class="content">
  2120. <p>The data stored in these files is not encrypted. In the case of
  2121. storing sensitive data in ZooKeeper, necessary measures need to be
  2122. taken to prevent unauthorized access. Such measures are external to
  2123. ZooKeeper (e.g., control access to the files) and depend on the
  2124. individual settings in which it is being deployed. </p>
  2125. </div>
  2126. </div>
  2127. <a name="Recovery+-+TxnLogToolkit"></a>
  2128. <h4>Recovery - TxnLogToolkit</h4>
  2129. <p>TxnLogToolkit is a command line tool shipped with ZooKeeper which
  2130. is capable of recovering transaction log entries with broken CRC.</p>
  2131. <p>Running it without any command line parameters or with the "-h,--help"
  2132. argument, it outputs the following help page:</p>
  2133. <pre class="code">
  2134. $ bin/zkTxnLogToolkit.sh
  2135. usage: TxnLogToolkit [-dhrv] txn_log_file_name
  2136. -d,--dump Dump mode. Dump all entries of the log file. (this is the default)
  2137. -h,--help Print help message
  2138. -r,--recover Recovery mode. Re-calculate CRC for broken entries.
  2139. -v,--verbose Be verbose in recovery mode: print all entries, not just fixed ones.
  2140. -y,--yes Non-interactive mode: repair all CRC errors without asking
  2141. </pre>
  2142. <p>The default behaviour is safe: it dumps the entries of the given
  2143. transaction log file to the screen: (same as using '-d,--dump' parameter)</p>
  2144. <pre class="code">
  2145. $ bin/zkTxnLogToolkit.sh log.100000001
  2146. ZooKeeper Transactional Log File with dbid 0 txnlog format version 2
  2147. 4/5/18 2:15:58 PM CEST session 0x16295bafcc40000 cxid 0x0 zxid 0x100000001 createSession 30000
  2148. <strong>CRC ERROR - 4/5/18 2:16:05 PM CEST session 0x16295bafcc40000 cxid 0x1 zxid 0x100000002 closeSession null</strong>
  2149. 4/5/18 2:16:05 PM CEST session 0x16295bafcc40000 cxid 0x1 zxid 0x100000002 closeSession null
  2150. 4/5/18 2:16:12 PM CEST session 0x26295bafcc90000 cxid 0x0 zxid 0x100000003 createSession 30000
  2151. 4/5/18 2:17:34 PM CEST session 0x26295bafcc90000 cxid 0x0 zxid 0x200000001 closeSession null
  2152. 4/5/18 2:17:34 PM CEST session 0x16295bd23720000 cxid 0x0 zxid 0x200000002 createSession 30000
  2153. 4/5/18 2:18:02 PM CEST session 0x16295bd23720000 cxid 0x2 zxid 0x200000003 create '/andor,#626262,v{s{31,s{'world,'anyone}}},F,1
  2154. EOF reached after 6 txns.
  2155. </pre>
  2156. <p>There's a CRC error in the 2nd entry of the above transaction log file. In <strong>dump</strong>
  2157. mode, the toolkit only prints this information to the screen without touching the original file. In
  2158. <strong>recovery</strong> mode (-r,--recover flag) the original file still remains
  2159. untouched and all transactions will be copied over to a new txn log file with ".fixed" suffix. It recalculates
  2160. CRC values and copies the calculated value, if it doesn't match the original txn entry.
  2161. By default, the tool works interactively: it asks for confirmation whenever CRC error encountered.</p>
  2162. <pre class="code">
  2163. $ bin/zkTxnLogToolkit.sh -r log.100000001
  2164. ZooKeeper Transactional Log File with dbid 0 txnlog format version 2
  2165. CRC ERROR - 4/5/18 2:16:05 PM CEST session 0x16295bafcc40000 cxid 0x1 zxid 0x100000002 closeSession null
  2166. Would you like to fix it (Yes/No/Abort) ?
  2167. </pre>
  2168. <p>Answering <strong>Yes</strong> means the newly calculated CRC value will be outputted
  2169. to the new file. <strong>No</strong> means that the original CRC value will be copied over.
  2170. <strong>Abort</strong> will abort the entire operation and exits.
  2171. (In this case the ".fixed" will not be deleted and left in a half-complete state: contains only entries which
  2172. have already been processed or only the header if the operation was aborted at the first entry.)</p>
  2173. <pre class="code">
  2174. $ bin/zkTxnLogToolkit.sh -r log.100000001
  2175. ZooKeeper Transactional Log File with dbid 0 txnlog format version 2
  2176. CRC ERROR - 4/5/18 2:16:05 PM CEST session 0x16295bafcc40000 cxid 0x1 zxid 0x100000002 closeSession null
  2177. Would you like to fix it (Yes/No/Abort) ? y
  2178. EOF reached after 6 txns.
  2179. Recovery file log.100000001.fixed has been written with 1 fixed CRC error(s)
  2180. </pre>
  2181. <p>The default behaviour of recovery is to be silent: only entries with CRC error get printed to the screen.
  2182. One can turn on verbose mode with the -v,--verbose parameter to see all records.
  2183. Interactive mode can be turned off with the -y,--yes parameter. In this case all CRC errors will be fixed
  2184. in the new transaction file.</p>
  2185. <a name="sc_commonProblems"></a>
  2186. <h3 class="h4">Things to Avoid</h3>
  2187. <p>Here are some common problems you can avoid by configuring
  2188. ZooKeeper correctly:</p>
  2189. <dl>
  2190. <dt>
  2191. <term>inconsistent lists of servers</term>
  2192. </dt>
  2193. <dd>
  2194. <p>The list of ZooKeeper servers used by the clients must match
  2195. the list of ZooKeeper servers that each ZooKeeper server has.
  2196. Things work okay if the client list is a subset of the real list,
  2197. but things will really act strange if clients have a list of
  2198. ZooKeeper servers that are in different ZooKeeper clusters. Also,
  2199. the server lists in each Zookeeper server configuration file
  2200. should be consistent with one another.</p>
  2201. </dd>
  2202. <dt>
  2203. <term>incorrect placement of transaction log</term>
  2204. </dt>
  2205. <dd>
  2206. <p>The most performance critical part of ZooKeeper is the
  2207. transaction log. ZooKeeper syncs transactions to media before it
  2208. returns a response. A dedicated transaction log device is key to
  2209. consistent good performance. Putting the log on a busy device will
  2210. adversely effect performance. If you only have one storage device,
  2211. put trace files on NFS and increase the snapshotCount; it doesn't
  2212. eliminate the problem, but it should mitigate it.</p>
  2213. </dd>
  2214. <dt>
  2215. <term>incorrect Java heap size</term>
  2216. </dt>
  2217. <dd>
  2218. <p>You should take special care to set your Java max heap size
  2219. correctly. In particular, you should not create a situation in
  2220. which ZooKeeper swaps to disk. The disk is death to ZooKeeper.
  2221. Everything is ordered, so if processing one request swaps the
  2222. disk, all other queued requests will probably do the same. the
  2223. disk. DON'T SWAP.</p>
  2224. <p>Be conservative in your estimates: if you have 4G of RAM, do
  2225. not set the Java max heap size to 6G or even 4G. For example, it
  2226. is more likely you would use a 3G heap for a 4G machine, as the
  2227. operating system and the cache also need memory. The best and only
  2228. recommend practice for estimating the heap size your system needs
  2229. is to run load tests, and then make sure you are well below the
  2230. usage limit that would cause the system to swap.</p>
  2231. </dd>
  2232. <dt>
  2233. <term>Publicly accessible deployment</term>
  2234. </dt>
  2235. <dd>
  2236. <p>
  2237. A ZooKeeper ensemble is expected to operate in a trusted computing environment.
  2238. It is thus recommended to deploy ZooKeeper behind a firewall.
  2239. </p>
  2240. </dd>
  2241. </dl>
  2242. <a name="sc_bestPractices"></a>
  2243. <h3 class="h4">Best Practices</h3>
  2244. <p>For best results, take note of the following list of good
  2245. Zookeeper practices:</p>
  2246. <p>For multi-tenant installations see the <a href="zookeeperProgrammers.html#ch_zkSessions">section</a>
  2247. detailing ZooKeeper "chroot" support, this can be very useful
  2248. when deploying many applications/services interfacing to a
  2249. single ZooKeeper cluster.</p>
  2250. </div>
  2251. <p align="right">
  2252. <font size="-2"></font>
  2253. </p>
  2254. </div>
  2255. <!--+
  2256. |end content
  2257. +-->
  2258. <div class="clearboth">&nbsp;</div>
  2259. </div>
  2260. <div id="footer">
  2261. <!--+
  2262. |start bottomstrip
  2263. +-->
  2264. <div class="lastmodified">
  2265. <script type="text/javascript"><!--
  2266. document.write("Last Published: " + document.lastModified);
  2267. // --></script>
  2268. </div>
  2269. <div class="copyright">
  2270. Copyright &copy;
  2271. <a href="http://www.apache.org/licenses/">The Apache Software Foundation.</a>
  2272. </div>
  2273. <!--+
  2274. |end bottomstrip
  2275. +-->
  2276. </div>
  2277. </body>
  2278. </html>