liuzhangxing
/
hadoop
forked from apache/hadoop


			
				
					
						
						
							123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475
							<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<META http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta content="Apache Forrest" name="Generator">
<meta name="Forrest-version" content="0.8">
<meta name="Forrest-skin-name" content="pelt">
<title>Hadoop Vaidya: A Performance Diagnostic Tool for Map-Reduce Jobs</title>
<link type="text/css" href="skin/basic.css" rel="stylesheet">
<link media="screen" type="text/css" href="skin/screen.css" rel="stylesheet">
<link media="print" type="text/css" href="skin/print.css" rel="stylesheet">
<link type="text/css" href="skin/profile.css" rel="stylesheet">
<script src="skin/getBlank.js" language="javascript" type="text/javascript"></script><script src="skin/getMenu.js" language="javascript" type="text/javascript"></script><script src="skin/fontsize.js" language="javascript" type="text/javascript"></script>
<link rel="shortcut icon" href="images/favicon.ico">
</head>
<body onload="init()">
<script type="text/javascript">ndeSetTextSize();</script>
<div id="top">
<!--+
    |breadtrail
    +-->
<div class="breadtrail">
<a href="http://www.apache.org/">Apache</a> &gt; <a href="http://hadoop.apache.org/">Hadoop</a> &gt; <a href="http://hadoop.apache.org/core/">Core</a><script src="skin/breadcrumbs.js" language="JavaScript" type="text/javascript"></script>
</div>
<!--+
    |header
    +-->
<div class="header">
<!--+
    |start group logo
    +-->
<div class="grouplogo">
<a href="http://hadoop.apache.org/"><img class="logoImage" alt="Hadoop" src="images/hadoop-logo.jpg" title="Apache Hadoop"></a>
</div>
<!--+
    |end group logo
    +-->
<!--+
    |start Project Logo
    +-->
<div class="projectlogo">
<a href="http://hadoop.apache.org/core/"><img class="logoImage" alt="Hadoop" src="images/core-logo.gif" title="Scalable Computing Platform"></a>
</div>
<!--+
    |end Project Logo
    +-->
<!--+
    |start Search
    +-->
<div class="searchbox">
<form action="http://www.google.com/search" method="get" class="roundtopsmall">
<input value="hadoop.apache.org" name="sitesearch" type="hidden"><input onFocus="getBlank (this, 'Search the site with google');" size="25" name="q" id="query" type="text" value="Search the site with google">&nbsp; 
                    <input name="Search" value="Search" type="submit">
</form>
</div>
<!--+
    |end search
    +-->
<!--+
    |start Tabs
    +-->
<ul id="tabs">
<li>
<a class="unselected" href="http://hadoop.apache.org/core/">Project</a>
</li>
<li>
<a class="unselected" href="http://wiki.apache.org/hadoop">Wiki</a>
</li>
<li class="current">
<a class="selected" href="index.html">Hadoop 0.20 Documentation</a>
</li>
</ul>
<!--+
    |end Tabs
    +-->
</div>
</div>
<div id="main">
<div id="publishedStrip">
<!--+
    |start Subtabs
    +-->
<div id="level2tabs"></div>
<!--+
    |end Endtabs
    +-->
<script type="text/javascript"><!--
document.write("Last Published: " + document.lastModified);
//  --></script>
</div>
<!--+
    |breadtrail
    +-->
<div class="breadtrail">

             &nbsp;
           </div>
<!--+
    |start Menu, mainarea
    +-->
<!--+
    |start Menu
    +-->
<div id="menu">
<div onclick="SwitchMenu('menu_selected_1.1', 'skin/')" id="menu_selected_1.1Title" class="menutitle" style="background-image: url('skin/images/chapter_open.gif');">Documentation</div>
<div id="menu_selected_1.1" class="selectedmenuitemgroup" style="display: block;">
<div class="menuitem">
<a href="index.html">Overview</a>
</div>
<div class="menuitem">
<a href="quickstart.html">Hadoop Quick Start</a>
</div>
<div class="menuitem">
<a href="cluster_setup.html">Hadoop Cluster Setup</a>
</div>
<div class="menuitem">
<a href="mapred_tutorial.html">Hadoop Map/Reduce Tutorial</a>
</div>
<div class="menuitem">
<a href="commands_manual.html">Hadoop Command Guide</a>
</div>
<div class="menuitem">
<a href="hdfs_shell.html">Hadoop FS Shell Guide</a>
</div>
<div class="menuitem">
<a href="distcp.html">Hadoop DistCp Guide</a>
</div>
<div class="menuitem">
<a href="native_libraries.html">Hadoop Native Libraries</a>
</div>
<div class="menuitem">
<a href="streaming.html">Hadoop Streaming</a>
</div>
<div class="menuitem">
<a href="hadoop_archives.html">Hadoop Archives</a>
</div>
<div class="menuitem">
<a href="hdfs_user_guide.html">HDFS User Guide</a>
</div>
<div class="menuitem">
<a href="hdfs_design.html">HDFS Architecture</a>
</div>
<div class="menuitem">
<a href="hdfs_permissions_guide.html">HDFS Admin Guide: Permissions</a>
</div>
<div class="menuitem">
<a href="hdfs_quota_admin_guide.html">HDFS Admin Guide: Quotas</a>
</div>
<div class="menuitem">
<a href="SLG_user_guide.html">HDFS Utilities</a>
</div>
<div class="menuitem">
<a href="libhdfs.html">HDFS C API</a>
</div>
<div class="menuitem">
<a href="hod_user_guide.html">HOD User Guide</a>
</div>
<div class="menuitem">
<a href="hod_admin_guide.html">HOD Admin Guide</a>
</div>
<div class="menuitem">
<a href="hod_config_guide.html">HOD Config Guide</a>
</div>
<div class="menuitem">
<a href="capacity_scheduler.html">Capacity Scheduler</a>
</div>
<div class="menupage">
<div class="menupagetitle">Hadoop Vaidya</div>
</div>
<div class="menuitem">
<a href="api/index.html">API Docs</a>
</div>
<div class="menuitem">
<a href="jdiff/changes.html">API Changes</a>
</div>
<div class="menuitem">
<a href="http://wiki.apache.org/hadoop/">Wiki</a>
</div>
<div class="menuitem">
<a href="http://wiki.apache.org/hadoop/FAQ">FAQ</a>
</div>
<div class="menuitem">
<a href="releasenotes.html">Release Notes</a>
</div>
<div class="menuitem">
<a href="changes.html">Change Log</a>
</div>
</div>
<div id="credit"></div>
<div id="roundbottom">
<img style="display: none" class="corner" height="15" width="15" alt="" src="skin/images/rc-b-l-15-1body-2menu-3menu.png"></div>
<!--+
  |alternative credits
  +-->
<div id="credit2"></div>
</div>
<!--+
    |end Menu
    +-->
<!--+
    |start content
    +-->
<div id="content">
<div title="Portable Document Format" class="pdflink">
<a class="dida" href="vaidya.pdf"><img alt="PDF -icon" src="skin/images/pdfdoc.gif" class="skin"><br>
        PDF</a>
</div>
<h1>Hadoop Vaidya: A Performance Diagnostic Tool for Map-Reduce Jobs</h1>
<div id="minitoc-area">
<ul class="minitoc">
<li>
<a href="#Purpose">Purpose</a>
</li>
<li>
<a href="#Pre-requisites">Pre-requisites</a>
</li>
<li>
<a href="#Overview">Overview</a>
</li>
<li>
<a href="#Terminology">Terminology</a>
</li>
<li>
<a href="#How+to+Execute+the+Hadoop+Vaidya+Tool">How to Execute the Hadoop Vaidya Tool</a>
</li>
<li>
<a href="#How+to+Write+and+Execute+your+own+Tests">How to Write and Execute your own Tests</a>
</li>
</ul>
</div>
  
    
<a name="N1000D"></a><a name="Purpose"></a>
<h2 class="h3">Purpose</h2>
<div class="section">
<p>This document describes various user-facing facets of the Hadoop Vaidya tool. It
         describes both, how to execute a default set of rules against your Map-Reduce job counters,
         as well as how to write and execute new rules to detect specific performance problems. 
      </p>
<p>At present few sample test rules are provided with the tool with the objective of growing the rules database over the time. 
         You are welcome to contribute the new rules for everyone's benefit and for that follow the similar
         <a href="http://wiki.apache.org/hadoop/HowToContribute">procedure</a> 
         specified on Apache Hadoop website.
      </p>
</div>
    
    
<a name="N1001E"></a><a name="Pre-requisites"></a>
<h2 class="h3">Pre-requisites</h2>
<div class="section">
<p>Ensure that Hadoop is installed and configured. More details:</p>
<ul>
        
<li>
          Make sure HADOOP_HOME environment variable is set.
        </li>
        
<li>
          Make sure Java is installed and configured as a part of the Hadoop installation.
        </li>
      
</ul>
</div>
    
    
<a name="N10031"></a><a name="Overview"></a>
<h2 class="h3">Overview</h2>
<div class="section">
<p>Hadoop Vaidya (Vaidya in Sanskrit language means "one who knows", or "a physician") 
	    is a rule based performance diagnostic tool for 
        Map/Reduce jobs. It performs a post execution analysis of map/reduce 
        job by parsing and collecting execution statistics through job history 
        and job configuration files. It runs a set of predefined tests/rules 
        against job execution statistics to diagnose various performance problems. 
        Each test rule detects a specific performance problem with the Map/Reduce job and provides 
        a targeted advice to the user. This tool generates an XML report based on 
        the evaluation results of individual test rules.
      </p>
</div>
  
    
<a name="N1003B"></a><a name="Terminology"></a>
<h2 class="h3">Terminology</h2>
<div class="section">
<p> This section describes main concepts and terminology involved with Hadoop Vaidya,</p>
<ul>
			
<li> 
<em>PostExPerformanceDiagnoser</em>: This class extends the base Diagnoser class and acts as a driver for post execution performance analysis of Map/Reduce Jobs. 
                       It detects performance inefficiencies by executing a set of performance diagnosis rules against the job execution statistics.</li>
			
<li> 
<em>Job Statistics</em>: This includes the job configuration information (job.xml) and various counters logged by Map/Reduce job as a part of the job history log
		           file. The counters are parsed and collected into the Job Statistics data structures, which contains global job level aggregate counters and 
			     a set of counters for each Map and Reduce task.</li>
			
<li> 
<em>Diagnostic Test/Rule</em>: This is a program logic that detects the inefficiency of M/R job based on the job statistics. The
				 description of the Test is specified as an XML element (DiagnosticTest) in a test description file e.g. 
				 default tests description file, <em>$HADOOP_HOME/contrib/vaidya/conf/postex_diagnosis_tests.xml</em>. The actual logic is coded as
				 a java class and referenced in the DiagnosticTest XML element. </li>
		
</ul>
<p></p>
<p>Following section describes the <em>DiagnosticTest</em> XML element in a diagnostic test description file </p>
<ul>
			
<li> 
<em>DiagnosticTest{Title}</em>: Specifies a short name/description of the test.</li>
			
<li> 
<em>DiagnosticTest{ClassName}</em>: Specifies fully qualified class name that implements the test logic.</li>
			
<li> 
<em>DiagnosticTest{Description}</em>: Specifies a full description of the test rule.</li>
			
<li> 
<em>DiagnosticTest{Importance}</em>: Specifies a declarative value for overall importance of the test rule. (Values: High, Medium, Low)</li>
			
<li> 
<em>DiagnosticTest{SuccessThreshod}</em>: This is a threshold value specified by test case writer such that if impact level of the test case
				 is lesser, then test is declared as PASSED (or NEGATIVE). The impact level is calculated and returned
				 by individual test's evaluate function, specifying the degree of problem job has with respect to the condition being evaluated.</li>
			
<li> 
<em>DiagnosticTest{Prescription}</em>: This is a targeted advice written by the test case adviser for the user to follow when test is not PASSED. </li>
			
<li> 
<em>DiagonsticTest{InputElement}</em>: This is a test specific input that test writer has to optionally provide. This will be supplied to individual test case
                       class so that test writer can use it within test case. This is typically a test configuration information such that test writer need not change the
                       Java code for test case but rather can configure the test case using these input values. </li>
		
</ul>
<p></p>
<p>Following section describes the performance analysis report generated by the tool in XML format</p>
<ul>
			
<li> 
<em>PostExPerformanceDiagnosticReport</em>: This is a document (root) element from the XML report generated by the tool. </li>
			
<li> 
<em>TestReportElement</em>: This is a XML report element from the test report document, one for each individual test specified in test description
				 file </li>  
			
<li> 
<em>TestReportElement{TestTitle}</em>: Will be included from DiagnosticTest{Title} </li>
			
<li> 
<em>TestReportElement{TestDescription}</em>: Will be included from DiagnosticTest{Description} </li>
			
<li> 
<em>TestReportElement{TestImportance}</em>: Will be included from DiagnosticTest{Importance} </li>
			
<li> 
<em>TestReportElement{TestSeverity}</em>: This is a product of Test Impact level and Test Importance. It indicates overall severity of the test.</li>
			
<li> 
<em>TestReportElement{ReferenceDetails}</em>: This is a test specific runtime information provided by test case to support the test result and severity. Typically
				 Test writer should print the test impact level in this section. </li>
			
<li> 
<em>TestReportElement{TestResults}</em>: This is boolean outcome of the test based on the SuccessThreshold specified by test writer in the DiagnosticTest description. The 
				 test PASSED(NEGATIVE) indicates no problem vs. FAILED (POSITIVE) indicates a potential problem with the job for given test case. </li>
			
<li> 
<em>TestReportElement{TestPrescription}</em>: This will be included from DiagnosticTest{Prescription}, unless test case writer overrides it in the test case class through getPrescription()
				 method </li>
		
</ul>
</div>
	
	
<a name="N100D0"></a><a name="How+to+Execute+the+Hadoop+Vaidya+Tool"></a>
<h2 class="h3">How to Execute the Hadoop Vaidya Tool</h2>
<div class="section">
<p>Script to execute Hadoop Vaidya is in <span class="codefrag">$HADOOP_HOME/contrib/vaidya/bin/</span> directory.
		   It comes with a default set of rules defined in file: 
           <span class="codefrag">$HADOOP_HOME/contrib/vaidya/conf/postex_diagnosis_tests.xml</span> 
</p>
<ul>
			
<li>Make sure HADOOP_HOME environment variable is set and Java is installed and configured.</li>
			
<li>Execute the Hadoop Vaidya script with -help (or without any arguments) to get the command line help. e.g. 
                       <span class="codefrag">=&gt;sh $HADOOP_HOME/contrib/vaidya/bin/vaidya.sh -help</span>
</li>
			
<li>User needs to 
				 supply job's configuration file (<span class="codefrag">-jobconf job_conf.xml</span>), job history log file (<span class="codefrag">-joblog job_history_log_file</span>), and optionally the test description
				 file (<span class="codefrag">-testconf postex_diagonostic_tests.xml</span>). If test description file is not specified then the default one is picked up from the Hadoop Vaidya Jar (<span class="codefrag">$HADOOP_HOME/contrib/vaidya/hadoop-{version}-vaidya.jar</span>).
				 This default test description file is also available at following location for users to make a local copy, modify and add new test rules: 
			     <span class="codefrag">$HADOOP_HOME/contrib/vaidya/conf/postex_diagnostic_tests.xml</span>
</li>
			
<li> Use <span class="codefrag">-report report_file</span> option to store the xml report into specified report_file. </li>  
		 
</ul>
</div>
	
    
<a name="N10102"></a><a name="How+to+Write+and+Execute+your+own+Tests"></a>
<h2 class="h3">How to Write and Execute your own Tests</h2>
<div class="section">
<p>Writing and executing your own test rules is not very hard. You can take a look at Hadoop Vaidya source code for existing set of tests. 
		   The source code is at this <a href="http://svn.apache.org/viewvc/hadoop/core/trunk/src/contrib/vaidya/src/java/org/apache/hadoop/vaidya/">hadoop svn repository location</a>
		   . The default set of tests are under <span class="codefrag">"postexdiagnosis/tests/"</span> folder.</p>
<ul>
		  
<li>Writing a test class for your new test case should extend the <span class="codefrag">org.apache.hadoop.vaidya.DiagnosticTest</span> class and 
		       it should override following three methods from the base class, 
              <ul> 
				
<li> evaluate() </li>
				
<li> getPrescription() </li> 
 				
<li> getReferenceDetails() </li> 
              
</ul>
          
</li>
		  
<li>Make a local copy of the <span class="codefrag">$HADOOP_HOME/contrib/vaidya/conf/postex_diagnostic_tests.xml</span> file or create a new test description XML file.</li>
		  
<li>Add the test description element for your new test case to this test description file.</li>
		  
<li>Compile your new test class (or multiple classes), archive them into a Jar file and add it to the CLASSPATH e.g. (<span class="codefrag">export CLASSPATH=$CLASSPATH:newtests.jar</span>)</li>
		  
<li>Execute the Hadoop Vaidya script with the job configuration, job history log and reference to newly created test description file using <em>--testconf</em> option. 
		  <span class="codefrag">=&gt;sh $HADOOP_HOME/contrib/vaidya/bin/vaidya.sh -joblog job_history_log_file -jobconf job.xml -testconf new_test_description_file -report report.xml</span>
</li>
		
</ul>
</div>
	
    
<p> 
</p>
    
<p> 
</p>
    
<p>
      
<em>Java and JNI are trademarks or registered trademarks of 
      Sun Microsystems, Inc. in the United States and other countries.</em>
    
</p>
    
  
</div>
<!--+
    |end content
    +-->
<div class="clearboth">&nbsp;</div>
</div>
<div id="footer">
<!--+
    |start bottomstrip
    +-->
<div class="lastmodified">
<script type="text/javascript"><!--
document.write("Last Published: " + document.lastModified);
//  --></script>
</div>
<div class="copyright">
        Copyright &copy;
         2008 <a href="http://www.apache.org/licenses/">The Apache Software Foundation.</a>
</div>
<!--+
    |end bottomstrip
    +-->
</div>
</body>
</html>