hadoop/CHANGES.branch-1-win.txt

Branch-hadoop-1-win (branched from branch-1 at r1293509 on 2012-02-24 17:46:42)

    HADOOP-8223 Initial patch for branch-1-win. (David Lao via Sanjay)

    HADOOP-8234 Enable user group mappings on Windows. (Bikas Saha via Sanjay)

    MAPREDUCE-4201 Getting PID not working on Windows. Termination of 
    Task/TaskJVM's not working. (Bikas Saha via Sanjay)

    MAPREDUCE-4204 Refactor ProcfsBasedProcessTree to make the resource 
    collection object pluggable. (Bikas Saha via Sanjay)

    HADOOP-8412 TestModTime, TestDelegationToken and TestAuthenticationToken fail
    intermittently on Windows. (Ivan Mitic via Sanjay Radia)

    HADOOP-8235 Support file permissions and ownership on Windows for 
    RawLocalFileSystem. (Chuan Liu via sanjay)

    HADOOP-8411 TestStorageDirecotyFailure, TestTaskLogsTruncater, 
    TestWebHdfsUrl and TestSecurityUtil fail on Windows.
    (Ivan Mitic via Sanjay Radia)

    HADOOP-8440 HarFileSystem.decodeHarURI fails for URIs whose host contains 
    numbers. (Ivan Mitic via Sanjay Radia)

    HADOOP-8374 Improve support for hard link manipulation on Windows.
    (Bikas Saha via Sanjay)

    HDFS-3424 TestDatanodeBlockScanner and TestReplication fail intermittently 
    on Windows (Bikas Saha via Sanjay)

    HADOOP-8424. Fix classpath so that web-ui works on Windows. (Bikas Saha
    via acmurthy) 

    MAPREDUCE-4321. Fix both DCE & LCE to use File.getAbsolutePath of the
    taskjvm.(sh,cmd) to ensure it works on Windows. (Ivan Mitic via acmurthy)

    MAPREDUCE-4260 Use JobObject to spawn tasks on Windows (Bikas Saha via Sanjay)

    HADOOP-8440 HarFileSystem.decodeHarURI 2nd patch.
    (Ivan Mitic via Sanjay Radia)

    HADOOP-8409 Fix TestCommandLineJobSubmission and TestGenericOptionsParser
    to work for windows (Ivan Mitic via Sanjay Radia) 

    HADOOP-8454 Fix the ‘chmod =[perm]’ bug in winutils. (Chuan Liu via sanjay)

    MAPREDUCE-4203. Added an implementation of the process tree for Windows.
    (Bikas Saha via acmurthy) 

    HADOOP-8486 Resource leak ... SequenceFile. (Kanna Karanam via Sanjay)

    HADOOP-8534 Some tests leave a config file open causing failure on windows.
    (Ivan Mitic via Sanjay Radia)

    HADOOP-8487 Many HDFS tests use a test path intended for local file system
    tests. (Ivan Mitic via Sanjay Radia)

    MAPREDUCE-4332 Fix command-length abort issues on Windows. (Ivan Mitic via
    acmurthy)

    MAPREDUCE-4369. Fix streaming test failures caused by
    WindowsResourceCalculatorPlugin. (Bikas Saha via acmurthy) 

    MAPREDUCE-4368. Fix TaskRunner to deal with java.library.path with a
    quoted path on Windows. (John Gordon via acmurthy) 

    HADOOP-8414 Address problems related to localhost resolving to 127.0.0.1 
    on Windows. (Ivan Mitic via Sanjay Radia)

    HADOOP-8544 Move an assertion location in 'winutils chmod'.
    (Chuan Liu via sanjay)

    MAPREDUCE-4598. Add support for NodeHealthCheck script for Windows.
    (Bikas Saha via acmurthy)

    MAPREDUCE-4510. Fix needless check/logging of getconf in Windows.
    (Bikas Saha via acmurthy)

    HADOOP-8453. Added unit tests for winutils. (Chuan Liu via acmurthy)

    MAPREDUCE-4374. Completed support for configurable environment for child
    map/reduce tasks. (Chuan Liu via acmurthy)

    HADOOP-8734. Modify LocalJobRunner to support private distributed cache.
    (Ivan Mitic via vinodkv)

    HDFS-3564. Improve BlockPlacementPolicy interface and change
    BlockPlacementPolicyDefault to make it more extendible.  (Sumadhur Reddy
    Bolli via szetszwo)

    HDFS-3566. Add AzureBlockPlacementPolicy to handle fault and upgrade
    domains in Azure.  (Sumadhur Reddy Bolli via szetszwo)

    HADOOP-8911. CRLF characters in source and text files.

    HDFS-6527. Backport HADOOP-7389: Use of TestingGroups by tests causes
    subsequent tests to fail.  (Ivan Mitic via szetszwo)

    HADOOP-8618. Fix build failures due to release 1.0.3 merge.
    (Bikas Saha via suresh)

    HDFS-3763. TestNameNodeMXBean fails on Windows. (Brandon Li via suresh)

    HADOOP-8739. Fix command line parsing on Windows. (Bikas Saha via
    acmurthy)

    HDFS-3766. Fix TestStorageRestore on Windows. (Brandon Li via suresh)

    HADOOP-8657. Fix TestCLI to not hardcode file length since it's
    encoding/platform dependent. (Bikas Saha via acmurthy) 

    MAPREDUCE-4597. Fix intermittent failures in TestKillSubProcesses.
    (Bikas Saha via acmurthy)

    HADOOP-8732. Fix intermittent failures in tests by serializing process
    creation to get around a bug in MS Windows. (Ivan Mitic via acmurthy)

    HDFS-3833. TestDFSShell fails on windows due to concurrent file 
    read/write. (Brandon Li via suresh)

    HADOOP-8664. hadoop streaming job need the full path to commands even 
    when they are in the path. (Bikas Saha via suresh)

    MAPREDUCE-4564. Fix issues with timing out shell processes. (Bikas Saha
    via acmurthy)

    HADOOP-8457. Address file ownership issue for users in Admin group on
    windows. (Ivan Mitic via suresh)

    MAPREDUCE-4400. Avoid task finish sleep to improve small job/workflow
    latency. (llu)

    HADOOP-8733. Fix the tests TestStreamingTaskLog, TestJvmManager
    and TestLinuxTaskControllerLaunchArgs to not fail on Windows. (Ivan Mitic via
    vinodkv)

    HADOOP-8694. Add support for windows native symbolic links.
    (Chuan Liu via suresh)
    
    HADOOP-8836. UGI should throw exception in case winutils.exe cannot be
    loaded. (Bikas Saha via suresh)

    MAPREDUCE-4657. WindowsResourceCalculatorPlugin has NPE.
    (Bikas Saha via suresh)

    HADOOP-8879. TestUserGroupInformation fails on Windows when run as
    Administrator. (Bikas Saha via suresh)

    HADOOP-8880. Missing jersey jars as dependency in the pom causes 
    hive tests to fail. (gkesavan via suresh)

    HADOOP-8903. support HADOOP_USER_CLASSPATH_FIRST in windows hadoop cmd.
    (Thejas Nair via suresh)

    HADOOP-8908. Refactor winutil.exe related code. (Chuan Liu via suresh)

    HADOOP-8911. CRLF characters in source and text files.
    (Raja Aluri via suresh)

    HADOOP-8912. Add .gitattributes file to prevent CRLF and LF mismatches 
    for source and text files. (Raja Aluri via suresh)

    HADOOP-8868. FileUtil#chmod should normalize the path before calling 
    into shell APIs. (Ivan Mitic via suresh)

    HADOOP-8936. Local users should be able to query a domain user's 
    groups on Windows. (Chuan Liu via suresh)

    HADOOP-8869. Links at the bottom of the jobdetails page do not render 
    correctly in IE9. (Ivan Mitic via suresh)

    HADOOP-8935. Make 'winutils ls' show the SID if the owner does not 
    exist on the system. (Chuan Liu via suresh)

    HDFS-4065. TestDFSShell.testGet sporadically fails attempting to corrupt 
    block files due to race condition. (Chris Nauroth via suresh)

    HADOOP-8899. Classpath exceeds maximum Windows command limit.
    (Ahmed El Baz via suresh)

    HADOOP-8763. Implement set-group-owner for Windows correctly. (Chuan Liu via
    vinodkv)

    HADOOP-8902. Enable Gridmix v1 & v2 benchmarks on Windows platform.
    (Mike Liddell via suresh)

    HADOOP-8564. Port and extend Hadoop native libraries for Windows to address
    datanode concurrent reading and writing issue. (Chuan Liu via suresh)

    HADOOP-4093. Fix a bug that AzureBlockPlacementPolicy#chooseTarget only
    returns one DN when replication factor is greater than 3.  (Jing Zhao via
    szetszwo)

    HADOOP-8907. Provide means to look for zlib1.dll next to hadoop.dll 
    on Windows. (Ivan Mitic via suresh)

    HADOOP-8872. FileSystem#length returns zero for symlinks on 
    windows+java6. (Ivan Mitic via suresh)

    HADOOP-8949. Remove FileUtil.CygPathCommand dead code. (Chris Nauroth via 
    suresh)

    HADOOP-8420. Hadoop Common creating package-info.java must not depend on sh.
    (Bikas Saha via suresh)

    HADOOP-8972. Move winutils tests from bat to Java. (Ivan Mitic via suresh)

    HADOOP-8983. Minor fixes to windows batch scripts to accept alternate 
    config directory location. (Ramya Bharathi Nimmagadda via suresh)

    HADOOP-9007. TestJobStatusPersistency.testJobStoreDisablingWithInvalidPath
    failure. (Chuan Liu via suresh)

    HADOOP-9006. Winutils should keep Administrators privileges intact.
    (Chuan Liu via suresh)

    HADOOP-9026. Hadoop.cmd fails to initialize if user's %path% variable has
    parenthesis. (Ivan Mitic via suresh)

    HADOOP-9027. Build fails on Windows without sh/sed/echo in the path.
    (Ivan Mitic via suresh)

    HADOOP-8847. Change untar to use Java API on Windows instead of spawning
    tar process. (Bikas Saha via suresh)

    HADOOP-8456. Support spaces in user names and group names in results 
    returned via winutils. (Ivan Mitic via suresh)

    HADOOP-9050. Remove java5 dependencies from build. (Raja Aluri via suresh)

    HADOOP-9062. hadoop-env.cmd overwrites the value of *_OPTS set before
    install. (Ganeshan Iyer via suresh)

    HADOOP-9074. Hadoop install scripts for Windows. (Steve Maine,
    Ramya Nimmagadda, Chuan Liu, Brad Sarsfield and Ivan Mitic via suresh)

    HADOOP-9110. HADOOP-9110. winutils ls off-by-one error indexing MONTHS array
    can cause access violation. (Chris Nauroth via suresh)

    HADOOP-9102. winutils task isAlive does not return a non-zero exit code if
    the requested task is not alive. (Chris Nauroth via suresh)

    HADOOP-9061. Java6+Windows does not work well with symlinks.
    (Ivan Mitic via suresh)

    HADOOP-8645. HADOOP_HOME and -Dhadoop.home (from hadoop wrapper script) are
    not uniformly handled. (John Gordon via suresh)

    HADOOP-9185. TestFileCreation.testFsClose should clean up on exit.
    (Arpit Agarwal via suresh)

    HADOOP-9250. Windows installer bugfixes. (Ivan Mitic via suresh)

    HADOOP-9463. branch-1-win fails to build with OpenJDK7.
    (Ivan Mitic via suresh)

    HADOOP-8731. Fixed TrackerDistributedCacheManager to address failure in the
    test TestTrackerDistributedCacheManager on Windows. (Ivan Mitic via vinodkv)

    HADOOP-9480. Windows installer should pick the config from 
    src\packages\win\template\conf. (Ivan Mitic via suresh)

    HADOOP-9452. Windows install scripts bugfixes. (Ivan Mitic via suresh)

    HADOOP-9472. Cleanup hadoop-config.cmd. (Ivan Mitic via suresh)

    HADOOP-9043. disallow in winutils creating symlinks with forwards slashes.
    (Chris Nauroth via suresh)

    HADOOP-9579. Contrib ant test target not setting the java.library.path.
    (ivanmi)

    HADOOP-9552. Windows log4j template should suppress info messages from
    mortbay.log. (ivanmi)

    HDFS-4677. Editlog should support synchronous writes. (ivanmi)

    MAPREDUCE-5224. JobTracker should allow the system directory to be in
    non-default FS. (Xi Fang via ivanmi)

    HADOOP-9681. FileUtil.unTarUsingJava() should close the InputStream upon
    finishing. (Chuan Liu via cnauroth)

    HADOOP-9677. TestSetupAndCleanupFailure#testWithDFS fails on Windows.
    (Xi Fang via cnauroth)

    MAPREDUCE-5330. JVM manager should not forcefully kill the process on
    Signal.TERM on Windows. (Xi Fang via cnauroth)

    HADOOP-9687. TestJobTrackerQuiescence and TestFileLengthOnClusterRestart
    failed caused by incorrect DFS path construction on Windows. (Xi Fang via
    cnauroth)

    HADOOP-9685. hadoop-config.cmd: builds a classpath that is too long on
    windows. (Raja Aluri via cnauroth)

    MAPREDUCE-5371. TestProxyUserFromEnv#testProxyUserFromEnvironment failed
    caused by domains of windows users. (Xi Fang via cnauroth)

    MAPREDUCE-5278. Distributed cache is broken when JT staging dir is not on the
    default FS. (Xi Fang via cnauroth)

    HADOOP-9718. Branch-1-win TestGroupFallback#testGroupWithFallback() failed
    caused by java.lang.UnsatisfiedLinkError. (Xi Fang via cnauroth)

    HADOOP-9719. Branch-1-win TestFsShellReturnCode#testChgrp() failed caused by
    incorrect exit codes. (Xi Fang via cnauroth)

    HADOOP-9722. Branch-1-win TestNativeIO failed caused by Window incompatible
    test case. (Xi Fang via cnauroth)

    HDFS-4975. Branch-1-win TestReplicationPolicy failed caused by stale data
    node handling. (Xi Fang via cnauroth)

    MAPREDUCE-5391. TestNonLocalJobJarSubmission fails on Windows due to missing
    classpath entries. (cnauroth)

    HDFS-5001. TestAzureBlockPlacementPolicy and
    TestReplicationPolicyWithNodeGroup failed caused by 1) old APIs and
    2) incorrect value of depthOfAllLeaves. (Xi Fang via cnauroth)

    HADOOP-9609. Remove sh dependency of bin-package target. (Chuan Liu via
    cnauroth)

    HADOOP-9790. Job token path is not unquoted properly. (Xi Fang via cnauroth)

    HADOOP-9551. Backport common utils introduced with HADOOP-9413
    to branch-1-win. (ivanmi)

    HADOOP-9802. Support Snappy codec on Windows. (cnauroth)

    HADOOP-9791. Add a test case covering long paths for new FileUtil access
    check methods (ivanmi)

    HADOOP-10094. NPE in GenericOptionsParser#preProcessForWindows().
    (Enis Soztutar via cnauroth)

  Merged from branch-1

    HDFS-385. Backport: Add support for an experimental API that allows a
    module external to HDFS to specify how HDFS blocks should be placed.
    (Sumadhur Reddy Bolli via szetszwo)

    HADOOP-8820. Backport HADOOP-8469 and HADOOP-8470: Make NetworkTopology
    class pluggable and add NetworkTopologyWithNodeGroup, a 4-layer
    implementation of NetworkTopology.  (Junping Du and Jing Zhao via szetszwo)

    HDFS-3941. Backport HDFS-3498 and HDFS-3601: Support replica removal in
    BlockPlacementPolicy and make BlockPlacementPolicyDefault extensible for
    reusing code in subclasses, and add BlockPlacementPolicyWithNodeGroup to
    support block placement with 4-layer network topology.  (Junping Du and
    Jing Zhao via szetszwo)

    HADOOP-7868. Hadoop native fails to compile when default linker
    option is -Wl,--as-needed. (Trevor Robinson via eli)

    HADOOP-6496. HttpServer sends wrong content-type for CSS files.
    (Todd Lipcon, Backport by Ivan Mitic via suresh).

    HADOOP-7827. jsp pages missing DOCTYPE tags. (Dave Vronay, Backport by
    Ivan Mitic via suresh)

    HADOOP-8900. BuiltInGzipDecompressor throws IOException - stored gzip size
    doesn't match decompressed size. (Andy Isaacson via suresh)

    HADOOP-9036. Fix racy test case TestSinkQueue (Backport HADOOP-7292).
    (Luke Lu backport by suresh)

    HADOOP-9095. Backport HADOOP-8372: NetUtils.normalizeHostName() incorrectly
    handles hostname starting with a numeric character.  (Jing Zhao via
    szetszwo)

    HADOOP-9099. TestNetUtils fails if "UnknownHost" is resolved as a valid
    hostname.  (Ivan Mitic via szetszwo)

    HADOOP-9111. Change some JUnit 3 tests to JUnit 4 so that @Ignore tests can
    be run with ant 1.8.x.  (Jing Zhao via szetszwo)
    HADOOP-9090. Support on-demand publish of metrics. (Mostafa Elhemali via
    suresh)

    HADOOP-8617. Backport HADOOP-6148, HADOOP-6166 and HADOOP-7333 for a pure
    Java CRC32 calculator implementation.  (Brandon Li via szetszwo)

    HADOOP-7096. Allow setting of end-of-record delimiter for TextInputFormat
    (Ahmed Radwan, backported by suresh)

    HDFS-3942. Backport HDFS-3495 and HDFS-4234: Update Balancer to support new
    NetworkTopology with NodeGroup and use generic code for choosing datanode
    in Balancer.  (Junping Du via szetszwo)

    HDFS-4320. Add a separate configuration for namenode rpc address instead
    of using fs.default.name. (Mostafa Elhemali via suresh)

    HDFS-4341. Set default data dir permission in MiniDFSClusterWithNodeGroup.
    (Ivan Mitic via szetszwo)

    HDFS-4337. Backport HDFS-4240: For nodegroup-aware block placement, when a
    node is excluded, the nodes in the same nodegroup should also be excluded.
    (Meng Gong via szetszwo)

    HADOOP-9174. TestSecurityUtil fails with Open JDK 7. (Arpit Agarwal via
    suresh)

    HADOOP-9175. TestWritableName fails with Open JDK 7. (Arpit Agarwal via
    suresh)

    MAPREDUCE-4909. TestKeyValueTextInputFormat fails with Open JDK 7 on
    Windows. (Arpit Agarwal via suresh)

    HDFS-4355. TestNameNodeMetrics.testCorruptBlock fails with open JDK7.
    (Brandon Li via suresh)

    HDFS-4358. TestCheckpoint failure with JDK7. (Arpit Agarwal via suresh)

    HADOOP-9179. TestFileSystem fails with open JDK7. (Brandon Li via suresh)

    MAPREDUCE-4914. TestMiniMRDFSSort fails with openJDK7. (Brandon Li via
    suresh)

    MAPREDUCE-4915. TestShuffleExceptionCount fails with open JDK7.
    (Brandon Li via suresh)

    HADOOP-9191. TestAccessControlList and TestJobHistoryConfig fail with
    JDK7. (Arpit Agarwal via suresh)

    HADOOP-9177. Address issues that reported by static code analysis on
    winutils. (Ivan Mitic via suresh)

    MAPREDUCE-4969. TestKeyValueTextInputFormat test fails with Open JDK 7.
    (Arpit Agarwal via suresh)

    HADOOP-8973. DiskChecker cannot reliably detect an inaccessible disk on
    Windows with NTFS ACLs. (Chris Nauroth via suresh)

    MAPREDUCE-782. Use PureJavaCrc32 in mapreduce spills. (Todd Lipcon, backport by Brandon Li via sseth)

    HADOOP-9071. Configure ivy log levels for resolve/retrieve.
    (Giridharan Kesavan via suresh)

    HDFS-4413. Secondary namenode won't start if HDFS isn't the default
    file system. (Mostafa Elhemali via suresh)

    HADOOP-9467. Metrics2 record filter should check name as well as tags.
    (Ganeshan Iyler via llu)

    HADOOP-9450. HADOOP_USER_CLASSPATH_FIRST is not honored; CLASSPATH
    is PREpended instead of APpended. (Chris Nauroth and harsh via harsh)

    MAPREDUCE-5161. Merge MAPREDUCE-1806 from branch-1 to branch-1-win.
    CombineFileInputFormat fix for paths not on default FS (Chris Nauroth via
    bikas)

    MAPREDUCE-5066. Added a timeout for the job.end.notification.url.
    (Ivan Mitic via acmurthy)

    MAPREDUCE-4737. Ensure that mapreduce APIs are semantically consistent
    with mapred API w.r.t Mapper.cleanup and Reducer.cleanup; in the sense
    that cleanup is now called even if there is an error. The old mapred API
    already ensures that Mapper.close and Reducer.close are invoked during
    error handling. Note that it is an incompatible change, however end-users
    can override Mapper.run and Reducer.run to get the old (inconsistent)
    behaviour. (acmurthy)

    HDFS-4782. Backport edit log corruption toleration. (Chris Nauroth via
    szetszwo)

    HADOOP-8745. Incorrect version numbers in hadoop-core POM.
    (Matthias Friedrich via eli)

    HADOOP-8981. TestMetricsSystemImpl fails on Windows. (Xuan Gong, backported
    by Chris Nauroth via suresh)

    HADOOP-9624. TestFSMainOperationsLocalFileSystem failed when the Hadoop test
    root path has "X" in its name. (Xi Fang via cnauroth)

    HDFS-4261. Fix bugs in Balaner causing infinite loop and
    TestBalancerWithNodeGroup timeing out.  (Junping Du via szetszwo)

    HDFS-3794. WebHDFS Open used with Offset returns the original (and incorrect)
    Content Length in the HTTP Header. (Tsz Wo (Nicholas), SZE via cnauroth)

    MAPREDUCE-5405. Job recovery can fail if task log directory symlink from
    prior run still exists. (cnauroth)

    HADOOP-9507. LocalFileSystem rename() is broken in some cases when
    destination exists. (cnauroth)

    HADOOP-9665. Fixed BlockDecompressorStream#decompress to return -1 rather
    than throw EOF at end of file. (Zhijie Shen via acmurthy)

    HADOOP-9768. chown and chgrp reject users and groups with spaces on platforms
    where spaces are otherwise acceptable. (cnauroth)

    HADOOP-9801. Configuration#writeXml uses platform defaulting encoding, which
    may mishandle multi-byte characters. (cnauroth)

    HDFS-4963. Improve multihoming support in namenode. (Arpit Agarwal via
    cnauroth)

    HDFS-5211. Race condition between DistributedFileSystem#close and
    FileSystem#close can cause return of a closed DistributedFileSystem instance
    from the FileSystem cache. (cnauroth)

    MAPREDUCE-5351. Fixed a memory leak in JobTracker due to stable FS objects in
    FSCache. (Sandy Ryza via acmurthy)

    MAPREDUCE-5508. JobTracker memory leak caused by unreleased FileSystem
    objects in JobInProgress#cleanupJob. (Xi Fang via cnauroth)

    HADOOP-10142. Avoid groups lookup for unprivileged users such as "dr.who"
    (vinay, backported by Xi Fang via cnauroth)

    HDFS-5516. WebHDFS does not require user name when anonymous http requests
    are disallowed. (Miodrag Radulovic via cnauroth)

    MAPREDUCE-5808. Port output replication factor configurable for terasort to
    Hadoop 1.x. (Chuan Liu via cnauroth)