hadoop/BUILDING.txt

Build instructions for Hadoop

----------------------------------------------------------------------------------
Requirements:

* Unix System
* JDK 1.6
* Maven 3.0
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.4.1+ (for MapReduce and HDFS)
* CMake 2.6 or newer (if compiling native code)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)

----------------------------------------------------------------------------------
Maven main modules:

  hadoop                            (Main Hadoop project)
         - hadoop-project           (Parent POM for all Hadoop Maven modules.             )
                                    (All plugins & dependencies versions are defined here.)
         - hadoop-project-dist      (Parent POM for modules that generate distributions.)
         - hadoop-annotations       (Generates the Hadoop doclet used to generated the Javadocs)
         - hadoop-assemblies        (Maven assemblies used by the different modules)
         - hadoop-common-project    (Hadoop Common)
         - hadoop-hdfs-project      (Hadoop HDFS)
         - hadoop-mapreduce-project (Hadoop MapReduce)
         - hadoop-tools             (Hadoop tools like Streaming, Distcp, etc.)
         - hadoop-dist              (Hadoop distribution assembler)

----------------------------------------------------------------------------------
Where to run Maven from?

  It can be run from any module. The only catch is that if not run from utrunk
  all modules that are not part of the build run must be installed in the local
  Maven cache or available in a Maven repository.

----------------------------------------------------------------------------------
Maven build goals:

 * Clean                     : mvn clean
 * Compile                   : mvn compile [-Pnative]
 * Run tests                 : mvn test [-Pnative]
 * Create JAR                : mvn package
 * Run findbugs              : mvn compile findbugs:findbugs
 * Run checkstyle            : mvn compile checkstyle:checkstyle
 * Install JAR in M2 cache   : mvn install
 * Deploy JAR to Maven repo  : mvn deploy
 * Run clover                : mvn test -Pclover [-DcloverLicenseLocation=${user.name}/.clover.license]
 * Run Rat                   : mvn apache-rat:check
 * Build javadocs            : mvn javadoc:javadoc
 * Build distribution        : mvn package [-Pdist][-Pdocs][-Psrc][-Pnative][-Dtar]
 * Change Hadoop version     : mvn versions:set -DnewVersion=NEWVERSION

 Build options:

  * Use -Pnative to compile/bundle native code
  * Use -Pdocs to generate & bundle the documentation in the distribution (using -Pdist)
  * Use -Psrc to create a project source TAR.GZ
  * Use -Dtar to create a TAR with the distribution (using -Pdist)

 Snappy build options:

   Snappy is a compression library that can be utilized by the native code.
   It is currently an optional component, meaning that Hadoop can be built with
   or without this dependency.

  * Use -Drequire.snappy to fail the build if libsnappy.so is not found.
    If this option is not specified and the snappy library is missing,
    we silently build a version of libhadoop.so that cannot make use of snappy.
    This option is recommended if you plan on making use of snappy and want
    to get more repeatable builds.

  * Use -Dsnappy.prefix to specify a nonstandard location for the libsnappy
    header files and library files. You do not need this option if you have
    installed snappy using a package manager.
  * Use -Dsnappy.lib to specify a nonstandard location for the libsnappy library
    files.  Similarly to snappy.prefix, you do not need this option if you have
    installed snappy using a package manager.
  * Use -Dbundle.snappy to copy the contents of the snappy.lib directory into
    the final tar file. This option requires that -Dsnappy.lib is also given,
    and it ignores the -Dsnappy.prefix option.

   Tests options:

  * Use -DskipTests to skip tests when running the following Maven goals:
    'package',  'install', 'deploy' or 'verify'
  * -Dtest=<TESTCLASSNAME>,<TESTCLASSNAME#METHODNAME>,....
  * -Dtest.exclude=<TESTCLASSNAME>
  * -Dtest.exclude.pattern=**/<TESTCLASSNAME1>.java,**/<TESTCLASSNAME2>.java

----------------------------------------------------------------------------------
Building distributions:

Create binary distribution without native code and without documentation:

  $ mvn package -Pdist -DskipTests -Dtar

Create binary distribution with native code and with documentation:

  $ mvn package -Pdist,native,docs -DskipTests -Dtar

Create source distribution:

  $ mvn package -Psrc -DskipTests

Create source and binary distributions with native code and documentation:

  $ mvn package -Pdist,native,docs,src -DskipTests -Dtar

Create a local staging version of the website (in /tmp/hadoop-site)

  $ mvn clean site; mvn site:stage -DstagingDirectory=/tmp/hadoop-site

----------------------------------------------------------------------------------