HADOOP-4687 split the contrib dirs

git-svn-id: https://svn.apache.org/repos/asf/hadoop/core/branches/HADOOP-4687/hdfs@776180 13f79535-47bb-0310-9956-ffa450edef68

src/contrib/fuse-dfs/Makefile.am

@@ -0,0 +1,27 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+@GLOBAL_HEADER_MK@
+
+@PRODUCT_MK@
+
+SUBDIRS = . src
+
+clean:
+	rm -rf autom4te.cache config.guess config.log config.status config.sub configure depcomp src/.deps install-sh Makefile.in src/Makefile.in src/Makefile missing Makefile src/fuse_dfs.o src/fuse_dfs aclocal.m4
+
+
+@GLOBAL_FOOTER_MK@

src/contrib/fuse-dfs/README

@@ -0,0 +1,131 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+Fuse-DFS
+
+Supports reads, writes, and directory operations (e.g., cp, ls, more, cat, find, less, rm, mkdir, mv, rmdir).  Things like touch, chmod, chown, and permissions are in the works. Fuse-dfs currently shows all files as owned by nobody.
+
+Contributing
+
+It's pretty straightforward to add functionality to fuse-dfs as fuse makes things relatively simple. Some other tasks require also augmenting libhdfs to expose more hdfs functionality to C. See [http://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&mode=hide&pid=12310240&sorter/order=DESC&sorter/field=priority&resolution=-1&component=12312376  contrib/fuse-dfs JIRAs]
+
+Requirements
+
+ * Hadoop with compiled libhdfs.so
+ * Linux kernel > 2.6.9 with fuse, which is the default or Fuse 2.7.x, 2.8.x installed. See: [http://fuse.sourceforge.net/]
+ * modprobe fuse to load it
+ * fuse-dfs executable (see below)
+ * fuse_dfs_wrapper.sh installed in /bin or other appropriate location (see below)
+
+
+BUILDING
+
+   1. in HADOOP_HOME: `ant compile-libhdfs -Dlibhdfs=1
+   2. in HADOOP_HOME: `ant package` to deploy libhdfs
+   3. in HADOOP_HOME: `ant compile-contrib -Dlibhdfs=1 -Dfusedfs=1`
+
+NOTE: for amd64 architecture, libhdfs will not compile unless you edit
+the Makefile in src/c++/libhdfs/Makefile and set OS_ARCH=amd64
+(probably the same for others too). See [https://issues.apache.org/jira/browse/HADOOP-3344 HADOOP-3344]
+
+Common build problems include not finding the libjvm.so in JAVA_HOME/jre/lib/OS_ARCH/server or not finding fuse in FUSE_HOME or /usr/local.
+
+
+CONFIGURING
+
+Look at all the paths in fuse_dfs_wrapper.sh and either correct them or set them in your environment before running. (note for automount and mount as root, you probably cannot control the environment, so best to set them in the wrapper)
+
+INSTALLING
+
+1. `mkdir /export/hdfs` (or wherever you want to mount it)
+
+2. `fuse_dfs_wrapper.sh dfs://hadoop_server1.foo.com:9000 /export/hdfs -d` and from another terminal, try `ls /export/hdfs`
+
+If 2 works, try again dropping the debug mode, i.e., -d
+
+(note - common problems are that you don't have libhdfs.so or libjvm.so or libfuse.so on your LD_LIBRARY_PATH, and your CLASSPATH does not contain hadoop and other required jars.)
+
+Also note, fuse-dfs will write error/warn messages to the syslog - typically in /var/log/messages
+
+You can use fuse-dfs to mount multiple hdfs instances by just changing the server/port name and directory mount point above.
+
+DEPLOYING
+
+in a root shell do the following:
+
+1. add the following to /etc/fstab
+
+fuse_dfs#dfs://hadoop_server.foo.com:9000 /export/hdfs fuse -oallow_other,rw,-ousetrash,-oinitchecks 0 0
+
+
+2. Mount using: `mount /export/hdfs`. Expect problems with not finding fuse_dfs. You will need to probably add this to /sbin and then problems finding the above 3 libraries. Add these using ldconfig.
+
+
+Fuse DFS takes the following mount options (i.e., on the command line or the comma separated list of options in /etc/fstab:
+
+-oserver=%s  (optional place to specify the server but in fstab use the format above)
+-oport=%d (optional port see comment on server option)
+-oentry_timeout=%d (how long directory entries are cached by fuse in seconds - see fuse docs)
+-oattribute_timeout=%d (how long attributes are cached by fuse in seconds - see fuse docs)
+-oprotected=%s (a colon separated list of directories that fuse-dfs should not allow to be deleted or moved - e.g., /user:/tmp)
+-oprivate (not often used but means only the person who does the mount can use the filesystem - aka ! allow_others in fuse speak)
+-ordbuffer=%d (in KBs how large a buffer should fuse-dfs use when doing hdfs reads)
+ro 
+rw
+-ousetrash (should fuse dfs throw things in /Trash when deleting them)
+-onotrash (opposite of usetrash)
+-odebug (do not daemonize - aka -d in fuse speak)
+-obig_writes (use fuse big_writes option so as to allow better performance of writes on kernels >= 2.6.26)
+-initchecks - have fuse-dfs try to connect to hdfs to ensure all is ok upon startup. recommended to have this  on
+The defaults are:
+
+entry,attribute_timeouts = 60 seconds
+rdbuffer = 10 MB
+protected = null
+debug = 0
+notrash
+private = 0
+
+EXPORTING
+
+Add the following to /etc/exports:
+
+/export/hdfs *.foo.com(no_root_squash,rw,fsid=1,sync)
+
+NOTE - you cannot export this with a FUSE module built into the kernel
+- e.g., kernel 2.6.17. For info on this, refer to the FUSE wiki.
+
+
+RECOMMENDATIONS
+
+1. From /bin, `ln -s $HADOOP_HOME/contrib/fuse-dfs/fuse_dfs* .`
+
+2. Always start with debug on so you can see if you are missing a classpath or something like that.
+
+3. use -obig_writes
+
+4. use -initchecks
+
+KNOWN ISSUES 
+
+1. if you alias `ls` to `ls --color=auto` and try listing a directory with lots (over thousands) of files, expect it to be slow and at 10s of thousands, expect it to be very very slow.  This is because `--color=auto` causes ls to stat every file in the directory. Since fuse-dfs does not cache attribute entries when doing a readdir, 
+this is very slow. see [https://issues.apache.org/jira/browse/HADOOP-3797 HADOOP-3797]
+
+2. Writes are approximately 33% slower than the DFSClient. TBD how to optimize this. see: [https://issues.apache.org/jira/browse/HADOOP-3805 HADOOP-3805] - try using -obig_writes if on a >2.6.26 kernel, should perform much better since bigger writes implies less context switching.
+
+3. Reads are ~20-30% slower even with the read buffering. 
+
+4. fuse-dfs and underlying libhdfs have no support for permissions. See [https://issues.apache.org/jira/browse/HADOOP-3536 HADOOP-3536] 

src/contrib/fuse-dfs/acinclude.m4

@@ -0,0 +1,270 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+AC_DEFUN([FUSE_DFS_INITIALIZE],
+[
+AM_INIT_AUTOMAKE([ foreign 1.9.5 no-define ])
+if test "x$1" = "xlocalinstall"; then
+wdir=`pwd`
+# To use $wdir undef quote.
+#
+##########
+AC_PREFIX_DEFAULT([`pwd`/install])
+echo
+fi
+AC_PROG_CC
+AC_PROG_CXX
+AC_PROG_RANLIB(RANLIB, ranlib)
+AC_PATH_PROGS(BASH, bash)
+AC_PATH_PROGS(PERL, perl)
+AC_PATH_PROGS(PYTHON, python)
+AC_PATH_PROGS(AR, ar)
+AC_PATH_PROGS(ANT, ant)
+PRODUCT_MK=""
+])
+
+AC_DEFUN([FUSE_DFS_WITH_EXTERNAL_PATH],
+[
+cdir=`pwd`
+AC_MSG_CHECKING([Checking EXTERNAL_PATH set to])
+AC_ARG_WITH([externalpath],
+  [ --with-externalpath=DIR User specified path to external fuse dfs components.],
+  [
+    if test "x${EXTERNAL_PATH}" != "x"; then
+       echo ""
+       echo "ERROR: You have already set EXTERNAL_PATH in your environment"
+       echo "Cannot override it using --with-externalpath. Unset EXTERNAL_PATH to use this option"
+       exit 1
+    fi
+    EXTERNAL_PATH=$withval
+  ],
+  [
+    if test "x${EXTERNAL_PATH}" = "x"; then
+       EXTERNAL_PATH=$1
+    fi
+  ]
+)
+if test "x${EXTERNAL_PATH}" = "x"; then
+   export EXTERNAL_PATH="$cdir/external"
+   GLOBAL_HEADER_MK="include ${EXTERNAL_PATH}/global_header.mk"
+   GLOBAL_FOOTER_MK="include ${EXTERNAL_PATH}/global_footer.mk"
+else
+   export EXTERNAL_PATH
+   GLOBAL_HEADER_MK="include ${EXTERNAL_PATH}/global_header.mk"
+   GLOBAL_FOOTER_MK="include ${EXTERNAL_PATH}/global_footer.mk"
+fi
+AC_MSG_RESULT($EXTERNAL_PATH)
+if test ! -d ${EXTERNAL_PATH}; then
+       echo ""
+       echo "ERROR: EXTERNAL_PATH set to an nonexistent directory ${EXTERNAL_PATH}"
+       exit 1
+fi
+AC_SUBST(EXTERNAL_PATH)
+AC_SUBST(GLOBAL_HEADER_MK)
+AC_SUBST(GLOBAL_FOOTER_MK)
+])
+
+# Set option to enable shared mode. Set DEBUG and OPT for use in Makefile.am.
+AC_DEFUN([FUSE_DFS_ENABLE_DEFAULT_OPT_BUILD],
+[
+AC_MSG_CHECKING([whether to enable optimized build])
+AC_ARG_ENABLE([opt],
+  [  --disable-opt     Set up debug mode.],
+  [
+     ENABLED_OPT=$enableval
+  ],
+  [
+     ENABLED_OPT="yes"
+  ]
+)
+if test "$ENABLED_OPT" = "yes"
+then
+     CFLAGS="-Wall -O3"
+     CXXFLAGS="-Wall -O3"
+else
+     CFLAGS="-Wall -g"
+     CXXFLAGS="-Wall -g"
+fi
+AC_MSG_RESULT($ENABLED_OPT)
+AM_CONDITIONAL([OPT], [test "$ENABLED_OPT" = yes])
+AM_CONDITIONAL([DEBUG], [test "$ENABLED_OPT" = no])
+])
+
+# Set option to enable debug mode. Set DEBUG and OPT for use in Makefile.am.
+AC_DEFUN([FUSE_DFS_ENABLE_DEFAULT_DEBUG_BUILD],
+[
+AC_MSG_CHECKING([whether to enable debug build])
+AC_ARG_ENABLE([debug],
+  [  --disable-debug     Set up opt mode.],
+  [
+     ENABLED_DEBUG=$enableval
+  ],
+  [
+     ENABLED_DEBUG="yes"
+  ]
+)
+if test "$ENABLED_DEBUG" = "yes"
+then
+     CFLAGS="-Wall -g"
+     CXXFLAGS="-Wall -g"
+else
+     CFLAGS="-Wall -O3"
+     CXXFLAGS="-Wall -O3"
+fi
+AC_MSG_RESULT($ENABLED_DEBUG)
+AM_CONDITIONAL([DEBUG], [test "$ENABLED_DEBUG" = yes])
+AM_CONDITIONAL([OPT], [test "$ENABLED_DEBUG" = no])
+])
+
+# Set option to enable static libs.
+AC_DEFUN([FUSE_DFS_ENABLE_DEFAULT_STATIC],
+[
+SHARED=""
+STATIC=""
+AC_MSG_CHECKING([whether to enable static mode])
+AC_ARG_ENABLE([static],
+  [  --disable-static     Set up shared mode.],
+  [
+     ENABLED_STATIC=$enableval
+  ],
+  [
+     ENABLED_STATIC="yes"
+  ]
+)
+if test "$ENABLED_STATIC" = "yes"
+then
+     LTYPE=".a"
+else
+     LTYPE=".so"
+     SHARED_CXXFLAGS="-fPIC"
+     SHARED_CFLAGS="-fPIC"
+     SHARED_LDFLAGS="-shared -fPIC"
+     AC_SUBST(SHARED_CXXFLAGS)
+     AC_SUBST(SHARED_CFLAGS)
+     AC_SUBST(SHARED_LDFLAGS)
+fi
+AC_MSG_RESULT($ENABLED_STATIC)
+AC_SUBST(LTYPE)
+AM_CONDITIONAL([STATIC], [test "$ENABLED_STATIC" = yes])
+AM_CONDITIONAL([SHARED], [test "$ENABLED_STATIC" = no])
+])
+
+# Set option to enable shared libs.
+AC_DEFUN([FUSE_DFS_ENABLE_DEFAULT_SHARED],
+[
+SHARED=""
+STATIC=""
+AC_MSG_CHECKING([whether to enable shared mode])
+AC_ARG_ENABLE([shared],
+  [  --disable-shared     Set up static mode.],
+  [
+    ENABLED_SHARED=$enableval
+  ],
+  [
+     ENABLED_SHARED="yes"
+  ]
+)
+if test "$ENABLED_SHARED" = "yes"
+then
+     LTYPE=".so"
+     SHARED_CXXFLAGS="-fPIC"
+     SHARED_CFLAGS="-fPIC"
+     SHARED_LDFLAGS="-shared -fPIC"
+     AC_SUBST(SHARED_CXXFLAGS)
+     AC_SUBST(SHARED_CFLAGS)
+     AC_SUBST(SHARED_LDFLAGS)
+else
+     LTYPE=".a"
+fi
+AC_MSG_RESULT($ENABLED_SHARED)
+AC_SUBST(LTYPE)
+AM_CONDITIONAL([SHARED], [test "$ENABLED_SHARED" = yes])
+AM_CONDITIONAL([STATIC], [test "$ENABLED_SHARED" = no])
+])
+
+# Generates define flags and conditionals as specified by user.
+# This gets enabled *only* if user selects --enable-<FEATURE> otion.
+AC_DEFUN([FUSE_DFS_ENABLE_FEATURE],
+[
+ENABLE=""
+flag="$1"
+value="$3"
+AC_MSG_CHECKING([whether to enable $1])
+AC_ARG_ENABLE([$2],
+  [  --enable-$2     Enable $2.],
+  [
+     ENABLE=$enableval
+  ],
+  [
+     ENABLE="no"
+  ]
+)
+AM_CONDITIONAL([$1], [test "$ENABLE" = yes])
+if test "$ENABLE" = "yes"
+then
+   if test "x${value}" = "x"
+   then
+       AC_DEFINE([$1])
+   else
+       AC_DEFINE_UNQUOTED([$1], [$value])
+   fi
+fi
+AC_MSG_RESULT($ENABLE)
+])
+
+
+# can also use eval $2=$withval;AC_SUBST($2)
+AC_DEFUN([FUSE_DFS_WITH_PATH],
+[
+USRFLAG=""
+USRFLAG=$1
+AC_MSG_CHECKING([Checking $1 set to])
+AC_ARG_WITH([$2],
+  [ --with-$2=DIR User specified path.],
+  [
+    LOC=$withval
+    eval $USRFLAG=$withval
+  ],
+  [
+    LOC=$3
+    eval $USRFLAG=$3
+  ]
+)
+AC_SUBST([$1])
+AC_MSG_RESULT($LOC)
+])
+
+AC_DEFUN([FUSE_DFS_SET_FLAG_VALUE],
+[
+SETFLAG=""
+AC_MSG_CHECKING([Checking $1 set to])
+SETFLAG=$1
+eval $SETFLAG=\"$2\"
+AC_SUBST([$SETFLAG])
+AC_MSG_RESULT($2)
+])
+
+# NOTES
+# if using if else bourne stmt you must have more than a macro in it.
+# EX1 is not correct. EX2 is correct
+# EX1: if test "$XX" = "yes"; then
+#        AC_SUBST(xx)
+#      fi
+# EX2: if test "$XX" = "yes"; then
+#        xx="foo"
+#        AC_SUBST(xx)
+#      fi

src/contrib/fuse-dfs/bootstrap.sh

@@ -0,0 +1,22 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#!/bin/sh
+
+aclocal
+automake -a
+autoconf
+./configure

src/contrib/fuse-dfs/build.xml

@@ -0,0 +1,109 @@
+<?xml version="1.0"?>
+
+<!--
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+-->
+
+<project name="fuse-dfs" default="compile">
+
+  <import file="../build-contrib.xml"/>
+
+  <target name="check-libhdfs-fuse">
+    <condition property="libhdfs-fuse">
+      <and>
+        <isset property="fusedfs"/>
+        <isset property="libhdfs"/>
+      </and>
+    </condition>
+  </target>
+
+
+  <target name="check-libhdfs-exists" if="fusedfs">
+  <property name="libhdfs.lib" value="${hadoop.root}/build/libhdfs/libhdfs.so"/>
+        <available file="${libhdfs.lib}" property="libhdfs-exists"/>
+    <fail message="libhdfs.so does not exist: ${libhdfs.lib}. Please check flags -Dlibhdfs=1 -Dfusedfs=1 are set or first try ant compile-libhdfs -Dlibhdfs=1">
+         <condition>
+            <not><isset property="libhdfs-exists"/></not>
+          </condition>
+   </fail>
+   </target>
+
+  <!-- override compile target !-->
+  <target name="compile" depends="check-libhdfs-fuse,check-libhdfs-exists" if="libhdfs-fuse">
+    <echo message="contrib: ${name}"/>
+
+    <condition property="perms" value="1" else="0">
+    <not>
+      <isset property="libhdfs.noperms"/>
+    </not>
+    </condition>
+
+    <exec executable="/bin/sh" failonerror="true">
+      <arg value="${root}/bootstrap.sh"/>
+    </exec>
+    <exec executable="make" failonerror="true">
+      <env key="OS_NAME" value="${os.name}"/>
+      <env key="OS_ARCH" value="${os.arch}"/>
+      <env key="HADOOP_HOME" value="${hadoop.root}"/>
+      <env key="PACKAGE_VERSION" value="0.1.0"/>
+
+      <env key="PERMS" value="${perms}"/>
+    </exec>
+    <mkdir dir="${build.dir}"/>
+    <mkdir dir="${build.dir}/test"/>
+    <exec executable="cp" failonerror="true">
+    <arg line="${root}/src/fuse_dfs ${build.dir}"/>
+    </exec>
+    <mkdir dir="${build.dir}/test"/>
+    <exec executable="cp" failonerror="true">
+    <arg line="${root}/src/fuse_dfs_wrapper.sh ${build.dir}"/>
+    </exec>
+
+  </target>
+
+  <!-- override jar target !-->
+  <target name="jar"/>
+
+  <!-- override package target !-->
+  <target name="package" depends="check-libhdfs-fuse" if="libhdfs-fuse">
+    <echo message="contrib: ${name}"/>
+
+    <mkdir dir="${dist.dir}/contrib/${name}"/>
+    <exec executable="cp">
+      <arg value="-p"/>
+      <arg value="README"/>
+      <arg value="src/fuse_dfs"/>
+      <arg value="src/fuse_dfs_wrapper.sh"/>
+      <arg value="${dist.dir}/contrib/${name}"/>
+    </exec>
+  </target>
+
+  <target name="test" if="fusedfs">
+    <echo message="testing FuseDFS ..."/>
+   <antcall target="hadoopbuildcontrib.test"> 
+   </antcall>
+  </target>  
+
+  <!-- override clean target !-->
+  <target name="clean" depends="check-libhdfs-fuse" if="libhdfs-fuse">
+    <echo message="contrib: ${name}"/>
+
+    <exec executable="make">
+      <arg value="clean"/>
+    </exec>
+  </target>
+
+</project>

src/contrib/fuse-dfs/configure.ac

@@ -0,0 +1,76 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Autoconf input file
+# $Id$
+# AC - autoconf
+#########################################################################
+# Section 1:
+# DO NOT TOUCH EXCEPT TO CHANGE Product-Name and Rev# IN AC_INIT
+
+AC_PREREQ(2.52)
+AC_INIT([fuse_dfs], [0.1.0])
+#AC_CONFIG_AUX_DIR([/usr/share/automake-1.9])
+# To install locally
+AC_CANONICAL_TARGET()
+FUSE_DFS_INITIALIZE([localinstall])
+AC_PREFIX_DEFAULT([`pwd`])
+
+
+
+#case $target in
+#*64*intel)
+#     OS_ARCH=intel64 ;;
+#*64*amd* | *64*unknown*)
+#     OS_ARCH=amd64 ;;
+#$esac
+#AC_SUBST(OS_ARCH)
+ DEFS=""
+AC_SUBST([DEFS])
+
+AC_FUNC_GETGROUPS
+AC_TYPE_GETGROUPS
+
+############################################################################
+# Section 2:
+# User Configurable system defaults. Change With CAUTION!
+
+# User can include custom makefile rules. Uncomment and update only <name> in PRODUCT_MK.
+# Include where appropriate in any Makefile.am as @PRODUCT_MK@
+
+# Default path to external components and shared build tools
+# To point to other locations set  environment variable EXTERNAL_PATH.
+# DO NOT change default. Changing default value requires changing bootstrap.sh.
+FUSE_DFS_WITH_EXTERNAL_PATH([`pwd`])
+
+# Pre-defined macro to set optimized build mode. Configure with --disable-opt option to turn off optimization. Default CXXFLAGS set to '-Wall -O3'. In debug mode  CXXFLAGS is  '-Wall -g'
+# FUSE_DFSENABLE_DEFAULT_DEBUG_BUILD
+FUSE_DFS_ENABLE_DEFAULT_OPT_BUILD
+
+# Predefined macro to set static library mode. Configure with --disable-static option to turn off static lib mode.
+# FUSE_DFS_ENABLE_DEFAULT_SHARED
+FUSE_DFS_ENABLE_DEFAULT_STATIC
+
+AC_CONFIG_FILES(Makefile src/Makefile)
+
+
+############################################################################
+# Section 4:
+# DO NOT TOUCH.
+
+AC_SUBST(PRODUCT_MK)
+AC_OUTPUT

src/contrib/fuse-dfs/global_footer.mk

@@ -0,0 +1,18 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+thriftstyle : $(XBUILT_SOURCES)
+

src/contrib/fuse-dfs/global_header.mk

@@ -0,0 +1,51 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+ifneq ($$(XBUILT_SOURCES),)
+    XBUILT_SOURCES := $$(XBUILT_SOURCES) $$(XTARGET)
+else
+    XBUILT_SOURCES := $$(XTARGET)
+endif
+
+showvars:
+	@echo BUILD_SOURCES = $(BUILT_SOURCES)
+	@echo XBUILTSOURCES = $(XBUILT_SOURCES)
+	@echo DEFS = $(DEFS)
+	@echo CXXFLAGS = $(CXXFLAGS)
+	@echo AM_CXXFLAGS = $(AM_CXXFLAGS)
+	@echo CPPFLAGS = $(CPPFLAGS)
+	@echo AM_CPPFLAGS = $(AM_CPPFLAGS)
+	@echo LDFLAGS = $(LDFLAGS)
+	@echo AM_LDFLAGS = $(AM_LDFLAGS)
+	@echo LDADD = $(LDADD)
+	@echo LIBS = $(LIBS)
+	@echo EXTERNAL_LIBS = $(EXTERNAL_LIBS)
+	@echo EXTERNAL_PATH = $(EXTERNAL_PATH)
+	@echo MAKE = $(MAKE)
+	@echo MAKE_FLAGS = $(MAKE_FLAGS)
+	@echo AM_MAKEFLAGS = $(AM_MAKEFLAGS)
+	@echo top_builddir = $(top_builddir)
+	@echo top_srcdir = $(top_srcdir)
+	@echo srcdir = $(srcdir)
+	@echo PHPVAL = $(PHPVAL)
+	@echo PHPCONFIGDIR  = $(PHPCONFIGDIR)
+	@echo PHPCONFIGINCLUDEDIR = $(PHPCONFIGINCLUDEDIR)
+	@echo PHPCONFIGINCLUDES  = $(PHPCONFIGINCLUDES)
+	@echo PHPCONFIGLDFLAGS  = $(PHPCONFIGLDFLAGS)
+	@echo PHPCONFIGLIBS  = $(PHPCONFIGLIBS)
+
+clean-common:
+	rm -rf gen-*

src/contrib/fuse-dfs/ivy.xml

@@ -0,0 +1,37 @@
+<?xml version="1.0" ?>
+<ivy-module version="1.0">
+  <info organisation="org.apache.hadoop" module="${ant.project.name}">
+    <license name="Apache 2.0"/>
+    <ivyauthor name="Apache Hadoop Team" url="http://hadoop.apache.org"/>
+    <description>
+        FUSE plugin for HDFS
+    </description>
+  </info>
+  <configurations defaultconfmapping="default">
+    <!--these match the Maven configurations-->
+    <conf name="default" extends="master,runtime"/>
+    <conf name="master" description="contains the artifact but no dependencies"/>
+    <conf name="runtime" description="runtime but not the artifact" />
+
+    <conf name="common" visibility="private" 
+      extends="runtime"
+      description="artifacts needed to compile/test the application"/>
+    <conf name="test" visibility="private" extends="runtime"/>
+  </configurations>
+
+  <publications>
+    <!--get the artifact from our module name-->
+    <artifact conf="master"/>
+  </publications>
+  <dependencies>
+    <dependency org="commons-logging"
+      name="commons-logging"
+      rev="${commons-logging.version}"
+      conf="common->default"/>
+    <dependency org="log4j"
+      name="log4j"
+      rev="${log4j.version}"
+      conf="common->master"/>
+    </dependencies>
+  
+</ivy-module>

src/contrib/fuse-dfs/ivy/libraries.properties

@@ -0,0 +1,5 @@
+#This properties file lists the versions of the various artifacts used by streaming.
+#It drives ivy and the generation of a maven POM
+
+#Please list the dependencies name with version if they are different from the ones 
+#listed in the global libraries.properties file (in alphabetical order)

src/contrib/fuse-dfs/src/Makefile.am

@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+bin_PROGRAMS = fuse_dfs
+fuse_dfs_SOURCES = fuse_dfs.c fuse_options.c fuse_trash.c fuse_stat_struct.c fuse_users.c fuse_init.c fuse_connect.c fuse_impls_access.c fuse_impls_chmod.c  fuse_impls_chown.c  fuse_impls_create.c  fuse_impls_flush.c fuse_impls_getattr.c  fuse_impls_mkdir.c  fuse_impls_mknod.c  fuse_impls_open.c fuse_impls_read.c fuse_impls_release.c fuse_impls_readdir.c fuse_impls_rename.c fuse_impls_rmdir.c fuse_impls_statfs.c fuse_impls_symlink.c fuse_impls_truncate.c fuse_impls_utimens.c  fuse_impls_unlink.c fuse_impls_write.c
+AM_CPPFLAGS= -DPERMS=$(PERMS) -D_FILE_OFFSET_BITS=64 -I$(JAVA_HOME)/include -I$(HADOOP_HOME)/src/c++/libhdfs/ -I$(JAVA_HOME)/include/linux/ -D_FUSE_DFS_VERSION=\"$(PACKAGE_VERSION)\" -DPROTECTED_PATHS=\"$(PROTECTED_PATHS)\" -I$(FUSE_HOME)/include
+AM_LDFLAGS= -L$(HADOOP_HOME)/build/libhdfs -lhdfs -L$(FUSE_HOME)/lib -lfuse -L$(JAVA_HOME)/jre/lib/$(OS_ARCH)/server -ljvm
+

src/contrib/fuse-dfs/src/fuse_connect.c

@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "hdfs.h"
+
+#include "fuse_dfs.h"
+#include "fuse_connect.h"
+#include "fuse_users.h" 
+
+
+#if PERMS
+
+/**
+ * Connects to the NN as the current user/group according to FUSE
+ *
+ */
+hdfsFS doConnectAsUser(const char *hostname, int port) {
+  uid_t uid = fuse_get_context()->uid;
+
+  char *user = getUsername(uid);
+  if (NULL == user)
+    return NULL;
+  int numgroups = 0;
+  char **groups = getGroups(uid, &numgroups);
+  hdfsFS fs = hdfsConnectAsUser(hostname, port, user, (const char **)groups, numgroups);
+  freeGroups(groups, numgroups);
+  if (user) 
+    free(user);
+  return fs;
+}
+
+#else
+
+hdfsFS doConnectAsUser(const char *hostname, int port) {
+  return hdfsConnect(hostname, port);
+}
+
+#endif

src/contrib/fuse-dfs/src/fuse_connect.h

@@ -0,0 +1,32 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef __FUSE_CONNECT_H__
+#define __FUSE_CONNECT_H__
+
+#include "fuse_dfs.h"
+
+/**
+ * Connects to the NN as the current user/group according to FUSE.
+ * Uses the fuse context to get the user name and groups.
+ * (if hadoop pre 0.19, will ignore user and group).
+ */
+
+hdfsFS doConnectAsUser(const char *hostname, int port) ;
+
+#endif

src/contrib/fuse-dfs/src/fuse_context_handle.h

@@ -0,0 +1,48 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef __FUSE_CONTEXT_HANDLE_H__
+#define __FUSE_CONTEXT_HANDLE_H__
+
+#include <hdfs.h>
+#include <stddef.h>
+#include <sys/types.h>
+
+//
+// Structure to store fuse_dfs specific data
+// this will be created and passed to fuse at startup
+// and fuse will pass it back to us via the context function
+// on every operation.
+//
+typedef struct dfs_context_struct {
+  int debug;
+  char *nn_hostname;
+  int nn_port;
+  hdfsFS fs;
+  int read_only;
+  int usetrash;
+  int direct_io;
+  char **protectedpaths;
+  size_t rdbuffer_size;
+  // todo:
+  // total hack city - use this to strip off the dfs url from the filenames. (in fuse_impls_readdir.c)
+  char dfs_uri[1024];
+  int dfs_uri_len;
+} dfs_context;
+
+#endif

src/contrib/fuse-dfs/src/fuse_dfs.c

@@ -0,0 +1,140 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "fuse_dfs.h"
+#include "fuse_options.h"
+#include "fuse_impls.h"
+#include "fuse_init.h"
+
+
+int is_protected(const char *path) {
+
+  dfs_context *dfs = (dfs_context*)fuse_get_context()->private_data;
+  assert(dfs != NULL);
+  assert(dfs->protectedpaths);
+
+  int i ;
+  for (i = 0; dfs->protectedpaths[i]; i++) {
+    if (strcmp(path, dfs->protectedpaths[i]) == 0) {
+      return 1;
+    }
+  }
+  return 0;
+}
+
+static struct fuse_operations dfs_oper = {
+  .getattr	= dfs_getattr,
+  .access	= dfs_access,
+  .readdir	= dfs_readdir,
+  .destroy       = dfs_destroy,
+  .init         = dfs_init,
+  .open	        = dfs_open,
+  .read	        = dfs_read,
+  .symlink	= dfs_symlink,
+  .statfs	= dfs_statfs,
+  .mkdir	= dfs_mkdir,
+  .rmdir	= dfs_rmdir,
+  .rename	= dfs_rename,
+  .unlink       = dfs_unlink,
+  .release      = dfs_release,
+  .create       = dfs_create,
+  .write	= dfs_write,
+  .flush        = dfs_flush,
+  .mknod        = dfs_mknod,
+	.utimens	= dfs_utimens,
+  .chmod	= dfs_chmod,
+  .chown	= dfs_chown,
+  .truncate	= dfs_truncate,
+};
+
+
+int main(int argc, char *argv[])
+{
+
+  umask(0);
+
+  extern const char *program;  
+  program = argv[0];
+  struct fuse_args args = FUSE_ARGS_INIT(argc, argv);
+
+  /* clear structure that holds our options */
+  memset(&options, 0, sizeof(struct options));
+
+  // some defaults
+  options.rdbuffer_size = 10*1024*1024; 
+  options.attribute_timeout = 60; 
+  options.entry_timeout = 60;
+
+  if (fuse_opt_parse(&args, &options, dfs_opts, dfs_options) == -1)
+    /** error parsing options */
+    return -1;
+
+
+  // Some fuse options we set
+  if (! options.private) {
+    fuse_opt_add_arg(&args, "-oallow_other");
+  }
+
+  if (!options.no_permissions) {
+    fuse_opt_add_arg(&args, "-odefault_permissions");
+  }
+
+  {
+    char buf[1024];
+
+    snprintf(buf, sizeof buf, "-oattr_timeout=%d",options.attribute_timeout);
+    fuse_opt_add_arg(&args, buf);
+
+    snprintf(buf, sizeof buf, "-oentry_timeout=%d",options.entry_timeout);
+    fuse_opt_add_arg(&args, buf);
+  }
+
+  if (options.server == NULL || options.port == 0) {
+    print_usage(argv[0]);
+    exit(0);
+  }
+
+
+  // 
+  // Check we can connect to hdfs
+  // 
+  if (options.initchecks == 1) {
+    hdfsFS temp;
+    if ((temp = hdfsConnect(options.server, options.port)) == NULL) {
+      const char *cp = getenv("CLASSPATH");
+      const char *ld = getenv("LD_LIBRARY_PATH");
+      fprintf(stderr, "FATAL: misconfiguration problem, cannot connect to hdfs - here's your environment\n");
+      fprintf(stderr, "LD_LIBRARY_PATH=%s\n",ld == NULL ? "NULL" : ld);
+      fprintf(stderr, "CLASSPATH=%s\n",cp == NULL ? "NULL" : cp);
+      syslog(LOG_ERR, "FATAL: misconfiguration problem, cannot connect to hdfs - here's your environment\n");
+      syslog(LOG_ERR, "LD_LIBRARY_PATH=%s\n",ld == NULL ? "NULL" : ld);
+      syslog(LOG_ERR, "CLASSPATH=%s\n",cp == NULL ? "NULL" : cp);
+      exit(0);
+    }  
+    temp = NULL;
+  }
+
+  int ret = fuse_main(args.argc, args.argv, &dfs_oper, NULL);
+
+  if (ret) printf("\n");
+
+  /** free arguments */
+  fuse_opt_free_args(&args);
+
+  return ret;
+}

src/contrib/fuse-dfs/src/fuse_dfs.h

@@ -0,0 +1,67 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef __FUSE_DFS_H__
+#define __FUSE_DFS_H__
+
+#define FUSE_USE_VERSION 26
+
+#include <stdio.h>
+#include <string.h>
+#include <errno.h>
+#include <assert.h>
+#include <strings.h>
+#include <syslog.h>
+
+#include <fuse.h>
+#include <fuse/fuse_opt.h>
+
+#ifdef HAVE_CONFIG_H
+#include <config.h>
+#endif
+
+#ifdef linux
+/* For pread()/pwrite() */
+#define _XOPEN_SOURCE 500
+#endif
+
+#ifdef HAVE_SETXATTR
+#include <sys/xattr.h>
+#endif
+
+//
+// Check if a path is in the mount option supplied protected paths.
+//
+int is_protected(const char *path);
+
+
+//#define DOTRACE
+#ifdef DOTRACE
+#define TRACE(x) \
+  syslog(LOG_ERR, "fuse_dfs TRACE - %s\n", x);  \
+  fprintf(stderr, "fuse_dfs TRACE - %s\n", x);
+
+#define TRACE1(x,y)                              \
+  syslog(LOG_ERR, "fuse_dfs TRACE - %s %s\n", x,y);  \
+  fprintf(stderr, "fuse_dfs TRACE - %s %s\n", x,y);
+#else
+#define TRACE(x) ; 
+#define TRACE1(x,y) ; 
+#endif
+
+#endif // __FUSE_DFS_H__

src/contrib/fuse-dfs/src/fuse_dfs_wrapper.sh

@@ -0,0 +1,40 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+if [ "$HADOOP_HOME" = "" ]; then
+export HADOOP_HOME=/usr/local/share/hadoop
+fi
+
+export PATH=$HADOOP_HOME/contrib/fuse_dfs:$PATH
+
+for f in ls $HADOOP_HOME/lib/*.jar $HADOOP_HOME/*.jar ; do
+export  CLASSPATH=$CLASSPATH:$f
+done
+
+if [ "$OS_ARCH" = "" ]; then
+export OS_ARCH=amd64
+fi
+
+if [ "$JAVA_HOME" = "" ]; then
+export  JAVA_HOME=/usr/local/java
+fi
+
+if [ "$LD_LIBRARY_PATH" = "" ]; then
+export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/$OS_ARCH/server:/usr/local/share/hdfs/libhdfs/:/usr/local/lib
+fi
+
+./fuse_dfs $@

src/contrib/fuse-dfs/src/fuse_file_handle.h

@@ -0,0 +1,44 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef __FUSE_FILE_HANDLE_H__
+#define __FUSE_FILE_HANDLE_H__
+
+#include <hdfs.h>
+#include <pthread.h>
+
+/**
+ *
+ * dfs_fh_struct is passed around for open files. Fuse provides a hook (the context) 
+ * for storing file specific data.
+ *
+ * 2 Types of information:
+ * a) a read buffer for performance reasons since fuse is typically called on 4K chunks only
+ * b) the hdfs fs handle 
+ *
+ */
+typedef struct dfs_fh_struct {
+  hdfsFile hdfsFH;
+  char *buf;
+  tSize bufferSize;  //what is the size of the buffer we have
+  off_t buffersStartOffset; //where the buffer starts in the file
+  hdfsFS fs; // for reads/writes need to access as the real user
+  pthread_mutex_t mutex;
+} dfs_fh;
+
+#endif

src/contrib/fuse-dfs/src/fuse_impls.h

@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+#ifndef __FUSE_IMPLS_H__
+#define __FUSE_IMPLS_H__
+
+#include <fuse.h>
+#include <syslog.h>
+
+#include "fuse_context_handle.h"
+
+/**
+ * Implementations of the various fuse hooks.
+ * All of these (should be) thread safe.
+ *
+ */
+
+int dfs_mkdir(const char *path, mode_t mode);
+int dfs_rename(const char *from, const char *to);
+int dfs_getattr(const char *path, struct stat *st);
+int dfs_readdir(const char *path, void *buf, fuse_fill_dir_t filler,
+                off_t offset, struct fuse_file_info *fi);
+int dfs_read(const char *path, char *buf, size_t size, off_t offset,
+                    struct fuse_file_info *fi);
+int dfs_statfs(const char *path, struct statvfs *st);
+int dfs_mkdir(const char *path, mode_t mode);
+int dfs_rename(const char *from, const char *to);
+int dfs_rmdir(const char *path);
+int dfs_unlink(const char *path);
+int dfs_utimens(const char *path, const struct timespec ts[2]);
+int dfs_chmod(const char *path, mode_t mode);
+int dfs_chown(const char *path, uid_t uid, gid_t gid);
+int dfs_open(const char *path, struct fuse_file_info *fi);
+int dfs_write(const char *path, const char *buf, size_t size,
+              off_t offset, struct fuse_file_info *fi);
+int dfs_release (const char *path, struct fuse_file_info *fi);
+int dfs_mknod(const char *path, mode_t mode, dev_t rdev) ;
+int dfs_create(const char *path, mode_t mode, struct fuse_file_info *fi);
+int dfs_flush(const char *path, struct fuse_file_info *fi);
+int dfs_access(const char *path, int mask);
+int dfs_truncate(const char *path, off_t size);
+int dfs_symlink(const char *from, const char *to);
+
+#endif
+
+
+

src/contrib/fuse-dfs/src/fuse_impls_access.c

@@ -0,0 +1,43 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "fuse_dfs.h"
+#include "fuse_impls.h"
+#include "fuse_connect.h"
+
+int dfs_access(const char *path, int mask)
+{
+  TRACE1("access", path)
+  // bugbug - I think we need the FileSystemAPI/libhdfs to expose this!
+  // retrieve dfs specific data
+  dfs_context *dfs = (dfs_context*)fuse_get_context()->private_data;
+
+  // check params and the context var
+  assert(dfs);
+  assert(path);
+
+  hdfsFS userFS;
+  if ((userFS = doConnectAsUser(dfs->nn_hostname,dfs->nn_port)) == NULL) {
+    syslog(LOG_ERR, "ERROR: could not connect to dfs %s:%d\n", __FILE__, __LINE__);
+    return -EIO;
+  }
+  //  return hdfsAccess(userFS, path, mask);
+  return 0;
+}
+
+

src/contrib/fuse-dfs/src/fuse_impls_chmod.c

@@ -0,0 +1,50 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "fuse_dfs.h"
+#include "fuse_impls.h"
+#include "fuse_users.h"
+#include "fuse_connect.h"
+
+int dfs_chmod(const char *path, mode_t mode)
+{
+  TRACE1("chmod", path)
+
+#if PERMS
+  // retrieve dfs specific data
+  dfs_context *dfs = (dfs_context*)fuse_get_context()->private_data;
+
+  // check params and the context var
+  assert(path);
+  assert(dfs);
+  assert('/' == *path);
+
+  hdfsFS userFS;
+  // if not connected, try to connect and fail out if we can't.
+  if ((userFS = doConnectAsUser(dfs->nn_hostname,dfs->nn_port))== NULL) {
+    syslog(LOG_ERR, "ERROR: could not connect to dfs %s:%d\n", __FILE__, __LINE__);
+    return -EIO;
+  }
+
+  if (hdfsChmod(userFS, path, (short)mode)) {
+    syslog(LOG_ERR,"ERROR: hdfs trying to chmod %s to %d",path, (int)mode);
+    return -EIO;
+  }
+#endif
+  return 0;
+}

src/contrib/fuse-dfs/src/fuse_impls_chown.c

@@ -0,0 +1,81 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "fuse_dfs.h"
+#include "fuse_users.h"
+#include "fuse_impls.h"
+#include "fuse_connect.h"
+
+ int dfs_chown(const char *path, uid_t uid, gid_t gid)
+{
+  TRACE1("chown", path)
+
+  int ret = 0;
+
+#if PERMS
+  char *user = NULL;
+  char *group = NULL;
+
+  // retrieve dfs specific data
+  dfs_context *dfs = (dfs_context*)fuse_get_context()->private_data;
+
+  // check params and the context var
+  assert(path);
+  assert(dfs);
+  assert('/' == *path);
+
+  user = getUsername(uid);
+  if (NULL == user) {
+    syslog(LOG_ERR,"Could not lookup the user id string %d\n",(int)uid); 
+    fprintf(stderr, "could not lookup userid %d\n", (int)uid); 
+    ret = -EIO;
+  }
+
+  if (0 == ret) {
+    group = getGroup(gid);
+    if (group == NULL) {
+      syslog(LOG_ERR,"Could not lookup the group id string %d\n",(int)gid); 
+      fprintf(stderr, "could not lookup group %d\n", (int)gid); 
+      ret = -EIO;
+    } 
+  }
+
+  hdfsFS userFS = NULL;
+  if (0 == ret) {
+    // if not connected, try to connect and fail out if we can't.
+    if ((userFS = doConnectAsUser(dfs->nn_hostname,dfs->nn_port))== NULL) {
+      syslog(LOG_ERR, "ERROR: could not connect to dfs %s:%d\n", __FILE__, __LINE__);
+      ret = -EIO;
+    }
+  }
+
+  if (0 == ret) {
+    //  fprintf(stderr, "DEBUG: chown %s %d->%s %d->%s\n", path, (int)uid, user, (int)gid, group);
+    if (hdfsChown(userFS, path, user, group)) {
+      syslog(LOG_ERR,"ERROR: hdfs trying to chown %s to %d/%d",path, (int)uid, gid);
+      ret = -EIO;
+    }
+  }
+  if (user) 
+    free(user);
+  if (group)
+    free(group);
+#endif
+  return ret;
+
+}

src/contrib/fuse-dfs/src/fuse_impls_create.c

@@ -0,0 +1,27 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "fuse_dfs.h"
+#include "fuse_impls.h"
+
+int dfs_create(const char *path, mode_t mode, struct fuse_file_info *fi)
+{
+  TRACE1("create", path)
+  fi->flags |= mode;
+  return dfs_open(path, fi);
+}

src/contrib/fuse-dfs/src/fuse_impls_flush.c

@@ -0,0 +1,55 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "fuse_dfs.h"
+#include "fuse_impls.h"
+#include "fuse_file_handle.h"
+
+int dfs_flush(const char *path, struct fuse_file_info *fi) {
+  TRACE1("flush", path)
+
+  // retrieve dfs specific data
+  dfs_context *dfs = (dfs_context*)fuse_get_context()->private_data;
+
+  // check params and the context var
+  assert(path);
+  assert(dfs);
+  assert('/' == *path);
+  assert(fi);
+
+  if (NULL == (void*)fi->fh) {
+    return  0;
+  }
+
+  // note that fuse calls flush on RO files too and hdfs does not like that and will return an error
+  if (fi->flags & O_WRONLY) {
+
+    dfs_fh *fh = (dfs_fh*)fi->fh;
+    assert(fh);
+    hdfsFile file_handle = (hdfsFile)fh->hdfsFH;
+    assert(file_handle);
+
+    assert(fh->fs);
+    if (hdfsFlush(fh->fs, file_handle) != 0) {
+      syslog(LOG_ERR, "ERROR: dfs problem - could not flush file_handle(%lx) for %s %s:%d\n",(long)file_handle,path, __FILE__, __LINE__);
+      return -EIO;
+    }
+  }
+
+  return 0;
+}

src/contrib/fuse-dfs/src/fuse_impls_getattr.c

@@ -0,0 +1,68 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "fuse_dfs.h"
+#include "fuse_impls.h"
+#include "fuse_stat_struct.h"
+
+int dfs_getattr(const char *path, struct stat *st)
+{
+  TRACE1("getattr", path)
+
+  // retrieve dfs specific data
+  dfs_context *dfs = (dfs_context*)fuse_get_context()->private_data;
+
+  // check params and the context var
+  assert(dfs);
+  assert(path);
+  assert(st);
+
+  // if not connected, try to connect and fail out if we can't.
+  if (NULL == dfs->fs && NULL == (dfs->fs = hdfsConnect(dfs->nn_hostname,dfs->nn_port))) {
+    syslog(LOG_ERR, "ERROR: could not connect to %s:%d %s:%d\n", dfs->nn_hostname, dfs->nn_port,__FILE__, __LINE__);
+    return -EIO;
+  }
+
+  // call the dfs API to get the actual information
+  hdfsFileInfo *info = hdfsGetPathInfo(dfs->fs,path);
+
+  if (NULL == info) {
+    return -ENOENT;
+  }
+
+  fill_stat_structure(&info[0], st);
+
+  // setup hard link info - for a file it is 1 else num entries in a dir + 2 (for . and ..)
+  if (info[0].mKind == kObjectKindDirectory) {
+    int numEntries = 0;
+    hdfsFileInfo *info = hdfsListDirectory(dfs->fs,path,&numEntries);
+
+    if (info) {
+      hdfsFreeFileInfo(info,numEntries);
+    }
+    st->st_nlink = numEntries + 2;
+  } else {
+    // not a directory
+    st->st_nlink = 1;
+  }
+
+  // free the info pointer
+  hdfsFreeFileInfo(info,1);
+
+  return 0;
+}

src/contrib/fuse-dfs/src/fuse_impls_mkdir.c

@@ -0,0 +1,68 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "fuse_dfs.h"
+#include "fuse_impls.h"
+#include "fuse_trash.h"
+#include "fuse_connect.h"
+
+int dfs_mkdir(const char *path, mode_t mode)
+{
+  TRACE1("mkdir", path)
+
+  // retrieve dfs specific data
+  dfs_context *dfs = (dfs_context*)fuse_get_context()->private_data;
+
+  // check params and the context var
+  assert(path);
+  assert(dfs);
+  assert('/' == *path);
+
+  if (is_protected(path)) {
+    syslog(LOG_ERR,"ERROR: hdfs trying to create the directory: %s", path);
+    return -EACCES;
+  }
+
+  if (dfs->read_only) {
+    syslog(LOG_ERR,"ERROR: hdfs is configured as read-only, cannot create the directory %s\n",path);
+    return -EACCES;
+  }
+  
+  hdfsFS userFS;
+  // if not connected, try to connect and fail out if we can't.
+  if ((userFS = doConnectAsUser(dfs->nn_hostname,dfs->nn_port))== NULL) {
+    syslog(LOG_ERR, "ERROR: could not connect to dfs %s:%d\n", __FILE__, __LINE__);
+    return -EIO;
+  }
+
+  // In theory the create and chmod should be atomic.
+
+  if (hdfsCreateDirectory(userFS, path)) {
+    syslog(LOG_ERR,"ERROR: hdfs trying to create directory %s",path);
+    return -EIO;
+  }
+
+#if PERMS
+  if (hdfsChmod(userFS, path, (short)mode)) {
+    syslog(LOG_ERR,"ERROR: hdfs trying to chmod %s to %d",path, (int)mode);
+    return -EIO;
+  }
+#endif
+  return 0;
+
+}

src/contrib/fuse-dfs/src/fuse_impls_mknod.c

@@ -0,0 +1,26 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "fuse_dfs.h"
+#include "fuse_impls.h"
+
+ int dfs_mknod(const char *path, mode_t mode, dev_t rdev) {
+  TRACE1("mknod", path)
+  syslog(LOG_DEBUG,"in dfs_mknod");
+  return 0;
+}

src/contrib/fuse-dfs/src/fuse_impls_open.c

@@ -0,0 +1,83 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "fuse_dfs.h"
+#include "fuse_impls.h"
+#include "fuse_connect.h"
+#include "fuse_file_handle.h"
+
+int dfs_open(const char *path, struct fuse_file_info *fi)
+{
+  TRACE1("open", path)
+
+  dfs_context *dfs = (dfs_context*)fuse_get_context()->private_data;
+
+  // check params and the context var
+  assert(path);
+  assert('/' == *path);
+  assert(dfs);
+
+  int ret = 0;
+
+  // 0x8000 is always passed in and hadoop doesn't like it, so killing it here
+  // bugbug figure out what this flag is and report problem to Hadoop JIRA
+  int flags = (fi->flags & 0x7FFF);
+
+  // retrieve dfs specific data
+  dfs_fh *fh = (dfs_fh*)malloc(sizeof (dfs_fh));
+  if (fh == NULL) {
+    syslog(LOG_ERR, "ERROR: malloc of new file handle failed %s:%d\n", __FILE__, __LINE__);
+    return -EIO;
+  }
+
+  if ((fh->fs = doConnectAsUser(dfs->nn_hostname,dfs->nn_port)) == NULL) {
+    syslog(LOG_ERR, "ERROR: could not connect to dfs %s:%d\n", __FILE__, __LINE__);
+    return -EIO;
+  }
+
+  if ((fh->hdfsFH = hdfsOpenFile(fh->fs, path, flags,  0, 3, 0)) == NULL) {
+    syslog(LOG_ERR, "ERROR: could not connect open file %s:%d\n", __FILE__, __LINE__);
+    return -EIO;
+  }
+
+  // 
+  // mutex needed for reads/writes
+  //
+  pthread_mutex_init(&fh->mutex, NULL);
+
+  if (fi->flags & O_WRONLY || fi->flags & O_CREAT) {
+    // write specific initialization
+    fh->buf = NULL;
+  } else  {
+    // read specific initialization
+
+    assert(dfs->rdbuffer_size > 0);
+
+    if (NULL == (fh->buf = (char*)malloc(dfs->rdbuffer_size*sizeof (char)))) {
+      syslog(LOG_ERR, "ERROR: could not allocate memory for file buffer for a read for file %s dfs %s:%d\n", path,__FILE__, __LINE__);
+      ret = -EIO;
+    }
+
+    fh->buffersStartOffset = 0;
+    fh->bufferSize = 0;
+  }
+
+  fi->fh = (uint64_t)fh;
+
+  return ret;
+}

src/contrib/fuse-dfs/src/fuse_impls_read.c

@@ -0,0 +1,162 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "fuse_dfs.h"
+#include "fuse_impls.h"
+#include "fuse_file_handle.h"
+
+static size_t min(const size_t x, const size_t y) {
+  return x < y ? x : y;
+}
+
+/**
+ * dfs_read
+ *
+ * Reads from dfs or the open file's buffer.  Note that fuse requires that
+ * either the entire read be satisfied or the EOF is hit or direct_io is enabled
+ *
+ */
+int dfs_read(const char *path, char *buf, size_t size, off_t offset,
+                   struct fuse_file_info *fi)
+{
+  TRACE1("read",path)
+  
+  // retrieve dfs specific data
+  dfs_context *dfs = (dfs_context*)fuse_get_context()->private_data;
+
+  // check params and the context var
+  assert(dfs);
+  assert(path);
+  assert(buf);
+  assert(offset >= 0);
+  assert(size >= 0);
+  assert(fi);
+
+  dfs_fh *fh = (dfs_fh*)fi->fh;
+
+  assert(fh != NULL);
+  assert(fh->fs != NULL);
+  assert(fh->hdfsFH != NULL);
+
+  // special case this as simplifies the rest of the logic to know the caller wanted > 0 bytes
+  if (size == 0)
+    return 0;
+
+  // If size is bigger than the read buffer, then just read right into the user supplied buffer
+  if ( size >= dfs->rdbuffer_size) {
+    int num_read;
+    size_t total_read = 0;
+    while (size - total_read > 0 && (num_read = hdfsPread(fh->fs, fh->hdfsFH, offset + total_read, buf + total_read, size - total_read)) > 0) {
+      total_read += num_read;
+    }
+    // if there was an error before satisfying the current read, this logic declares it an error
+    // and does not try to return any of the bytes read. Don't think it matters, so the code
+    // is just being conservative.
+    if (total_read < size && num_read < 0) {
+      total_read = -EIO;
+    }
+    return total_read;
+  }
+
+  //
+  // Critical section - protect from multiple reads in different threads accessing the read buffer
+  // (no returns until end)
+  //
+
+  pthread_mutex_lock(&fh->mutex);
+
+  // used only to check the postcondition of this function - namely that we satisfy
+  // the entire read or EOF is hit.
+  int isEOF = 0;
+  int ret = 0;
+
+  // check if the buffer is empty or
+  // the read starts before the buffer starts or
+  // the read ends after the buffer ends
+
+  if (fh->bufferSize == 0  || 
+      offset < fh->buffersStartOffset || 
+      offset + size > fh->buffersStartOffset + fh->bufferSize) 
+    {
+      // Read into the buffer from DFS
+      int num_read = 0;
+      size_t total_read = 0;
+
+      while (dfs->rdbuffer_size  - total_read > 0 &&
+             (num_read = hdfsPread(fh->fs, fh->hdfsFH, offset + total_read, fh->buf + total_read, dfs->rdbuffer_size - total_read)) > 0) {
+        total_read += num_read;
+      }
+
+      // if there was an error before satisfying the current read, this logic declares it an error
+      // and does not try to return any of the bytes read. Don't think it matters, so the code
+      // is just being conservative.
+      if (total_read < size && num_read < 0) {
+        // invalidate the buffer 
+        fh->bufferSize = 0; 
+        syslog(LOG_ERR, "Read error - pread failed for %s with return code %d %s:%d", path, (int)num_read, __FILE__, __LINE__);
+        ret = -EIO;
+      } else {
+        // Either EOF, all read or read beyond size, but then there was an error
+        fh->bufferSize = total_read;
+        fh->buffersStartOffset = offset;
+
+        if (dfs->rdbuffer_size - total_read > 0) {
+          // assert(num_read == 0); this should be true since if num_read < 0 handled above.
+          isEOF = 1;
+        }
+      }
+    }
+
+  //
+  // NOTE on EOF, fh->bufferSize == 0 and ret = 0 ,so the logic for copying data into the caller's buffer is bypassed, and
+  //  the code returns 0 as required
+  //
+  if (ret == 0 && fh->bufferSize > 0) {
+
+    assert(offset >= fh->buffersStartOffset);
+    assert(fh->buf);
+
+    const size_t bufferReadIndex = offset - fh->buffersStartOffset;
+    assert(bufferReadIndex >= 0 && bufferReadIndex < fh->bufferSize);
+
+    const size_t amount = min(fh->buffersStartOffset + fh->bufferSize - offset, size);
+    assert(amount >= 0 && amount <= fh->bufferSize);
+
+    const char *offsetPtr = fh->buf + bufferReadIndex;
+    assert(offsetPtr >= fh->buf);
+    assert(offsetPtr + amount <= fh->buf + fh->bufferSize);
+    
+    memcpy(buf, offsetPtr, amount);
+
+    ret = amount;
+  }
+
+  //
+  // Critical section end 
+  //
+  pthread_mutex_unlock(&fh->mutex);
+ 
+  // fuse requires the below and the code should guarantee this assertion
+  // 3 cases on return:
+  //   1. entire read satisfied
+  //   2. partial read and isEOF - including 0 size read
+  //   3. error 
+  assert(ret == size || isEOF || ret < 0);
+
+ return ret;
+}

src/contrib/fuse-dfs/src/fuse_impls_readdir.c

@@ -0,0 +1,117 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "fuse_dfs.h"
+#include "fuse_impls.h"
+#include "fuse_stat_struct.h"
+#include "fuse_connect.h"
+
+int dfs_readdir(const char *path, void *buf, fuse_fill_dir_t filler,
+                       off_t offset, struct fuse_file_info *fi)
+{
+  TRACE1("readdir",path)
+
+  (void) offset;
+  (void) fi;
+
+  // retrieve dfs specific data
+  dfs_context *dfs = (dfs_context*)fuse_get_context()->private_data;
+
+  // check params and the context var
+  assert(dfs);
+  assert(path);
+  assert(buf);
+
+  int path_len = strlen(path);
+
+  hdfsFS userFS;
+  // if not connected, try to connect and fail out if we can't.
+  if ((userFS = doConnectAsUser(dfs->nn_hostname,dfs->nn_port))== NULL) {
+    syslog(LOG_ERR, "ERROR: could not connect to dfs %s:%d\n", __FILE__, __LINE__);
+    return -EIO;
+  }
+
+  // call dfs to read the dir
+  int numEntries = 0;
+  hdfsFileInfo *info = hdfsListDirectory(userFS,path,&numEntries);
+  userFS = NULL;
+
+  // NULL means either the directory doesn't exist or maybe IO error.
+  if (NULL == info) {
+    return -ENOENT;
+  }
+
+  int i ;
+  for (i = 0; i < numEntries; i++) {
+
+    // check the info[i] struct
+    if (NULL == info[i].mName) {
+      syslog(LOG_ERR,"ERROR: for <%s> info[%d].mName==NULL %s:%d", path, i, __FILE__,__LINE__);
+      continue;
+    }
+
+    struct stat st;
+    fill_stat_structure(&info[i], &st);
+
+    // hack city: todo fix the below to something nicer and more maintainable but
+    // with good performance
+    // strip off the path but be careful if the path is solely '/'
+    // NOTE - this API started returning filenames as full dfs uris
+    const char *const str = info[i].mName + dfs->dfs_uri_len + path_len + ((path_len == 1 && *path == '/') ? 0 : 1);
+
+    // pack this entry into the fuse buffer
+    int res = 0;
+    if ((res = filler(buf,str,&st,0)) != 0) {
+      syslog(LOG_ERR, "ERROR: readdir filling the buffer %d %s:%d\n",res, __FILE__, __LINE__);
+    }
+  }
+
+  // insert '.' and '..'
+  const char *const dots [] = { ".",".."};
+  for (i = 0 ; i < 2 ; i++)
+    {
+      struct stat st;
+      memset(&st, 0, sizeof(struct stat));
+
+      // set to 0 to indicate not supported for directory because we cannot (efficiently) get this info for every subdirectory
+      st.st_nlink =  0;
+
+      // setup stat size and acl meta data
+      st.st_size    = 512;
+      st.st_blksize = 512;
+      st.st_blocks  =  1;
+      st.st_mode    = (S_IFDIR | 0777);
+      st.st_uid     = default_id;
+      st.st_gid     = default_id;
+      // todo fix below times
+      st.st_atime   = 0;
+      st.st_mtime   = 0;
+      st.st_ctime   = 0;
+
+      const char *const str = dots[i];
+
+      // flatten the info using fuse's function into a buffer
+      int res = 0;
+      if ((res = filler(buf,str,&st,0)) != 0) {
+        syslog(LOG_ERR, "ERROR: readdir filling the buffer %d %s:%d", res, __FILE__, __LINE__);
+      }
+    }
+  // free the info pointers
+  hdfsFreeFileInfo(info,numEntries);
+  return 0;
+}

src/contrib/fuse-dfs/src/fuse_impls_release.c

@@ -0,0 +1,82 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "fuse_dfs.h"
+#include "fuse_impls.h"
+#include "fuse_file_handle.h"
+
+/**
+ * This mutex is to protect releasing a file handle in case the user calls close in different threads
+ * and fuse passes these calls to here.
+ */
+pthread_mutex_t release_mutex = PTHREAD_MUTEX_INITIALIZER;
+
+int dfs_release (const char *path, struct fuse_file_info *fi) {
+  TRACE1("release", path)
+
+  // retrieve dfs specific data
+  dfs_context *dfs = (dfs_context*)fuse_get_context()->private_data;
+
+  // check params and the context var
+  assert(path);
+  assert(dfs);
+  assert('/' == *path);
+
+  int ret = 0;
+
+  //
+  // Critical section - protect from multiple close calls in different threads.
+  // (no returns until end)
+  //
+
+  pthread_mutex_lock(&release_mutex);
+
+  if (NULL != (void*)fi->fh) {
+
+    dfs_fh *fh = (dfs_fh*)fi->fh;
+    assert(fh);
+
+    hdfsFile file_handle = (hdfsFile)fh->hdfsFH;
+
+    if (NULL != file_handle) {
+      if (hdfsCloseFile(fh->fs, file_handle) != 0) {
+        syslog(LOG_ERR, "ERROR: dfs problem - could not close file_handle(%ld) for %s %s:%d\n",(long)file_handle,path, __FILE__, __LINE__);
+        fprintf(stderr, "ERROR: dfs problem - could not close file_handle(%ld) for %s %s:%d\n",(long)file_handle,path, __FILE__, __LINE__);
+        ret = -EIO;
+      }
+    }
+
+    if (fh->buf != NULL) {
+      free(fh->buf);
+    }
+    // this is always created and initialized, so always destroy it. (see dfs_open)
+    pthread_mutex_destroy(&fh->mutex);
+
+    free(fh);
+
+    fi->fh = (uint64_t)0;
+  }
+
+  pthread_mutex_unlock(&release_mutex);
+
+  //
+  // End critical section 
+  // 
+
+  return ret;
+}

src/contrib/fuse-dfs/src/fuse_impls_rename.c

@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "fuse_dfs.h"
+#include "fuse_impls.h"
+#include "fuse_trash.h"
+#include "fuse_connect.h"
+
+int dfs_rename(const char *from, const char *to)
+{
+  TRACE1("rename", from) 
+
+ // retrieve dfs specific data
+  dfs_context *dfs = (dfs_context*)fuse_get_context()->private_data;
+
+  // check params and the context var
+  assert(from);
+  assert(to);
+  assert(dfs);
+
+  assert('/' == *from);
+  assert('/' == *to);
+
+  if (is_protected(from) || is_protected(to)) {
+    syslog(LOG_ERR,"ERROR: hdfs trying to rename: %s %s", from, to);
+    return -EACCES;
+  }
+
+  if (dfs->read_only) {
+    syslog(LOG_ERR,"ERROR: hdfs is configured as read-only, cannot rename the directory %s\n",from);
+    return -EACCES;
+  }
+
+  hdfsFS userFS;
+  // if not connected, try to connect and fail out if we can't.
+  if ((userFS = doConnectAsUser(dfs->nn_hostname,dfs->nn_port))== NULL) {
+    syslog(LOG_ERR, "ERROR: could not connect to dfs %s:%d\n", __FILE__, __LINE__);
+    return -EIO;
+  }
+
+  if (hdfsRename(userFS, from, to)) {
+    syslog(LOG_ERR,"ERROR: hdfs trying to rename %s to %s",from, to);
+    return -EIO;
+  }
+
+  return 0;
+
+}

src/contrib/fuse-dfs/src/fuse_impls_rmdir.c

@@ -0,0 +1,71 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "fuse_dfs.h"
+#include "fuse_impls.h"
+#include "fuse_trash.h"
+#include "fuse_connect.h"
+
+extern const char *const TrashPrefixDir;
+
+int dfs_rmdir(const char *path)
+{
+  TRACE1("rmdir", path)
+
+  // retrieve dfs specific data
+  dfs_context *dfs = (dfs_context*)fuse_get_context()->private_data;
+
+  // check params and the context var
+  assert(path);
+  assert(dfs);
+  assert('/' == *path);
+
+  if (is_protected(path)) {
+    syslog(LOG_ERR,"ERROR: hdfs trying to delete a protected directory: %s ",path);
+    return -EACCES;
+  }
+
+  if (dfs->read_only) {
+    syslog(LOG_ERR,"ERROR: hdfs is configured as read-only, cannot delete the directory %s\n",path);
+    return -EACCES;
+  }
+
+  hdfsFS userFS;
+  // if not connected, try to connect and fail out if we can't.
+  if ((userFS = doConnectAsUser(dfs->nn_hostname,dfs->nn_port))== NULL) {
+    syslog(LOG_ERR, "ERROR: could not connect to dfs %s:%d\n", __FILE__, __LINE__);
+    return -EIO;
+  }
+
+  int numEntries = 0;
+  hdfsFileInfo *info = hdfsListDirectory(userFS,path,&numEntries);
+
+  // free the info pointers
+  hdfsFreeFileInfo(info,numEntries);
+
+  if (numEntries) {
+    return -ENOTEMPTY;
+  }
+
+  if (hdfsDeleteWithTrash(userFS, path, dfs->usetrash)) {
+    syslog(LOG_ERR,"ERROR: hdfs error trying to delete the directory %s\n",path);
+    return -EIO;
+  }
+
+  return 0;
+}

src/contrib/fuse-dfs/src/fuse_impls_statfs.c

@@ -0,0 +1,85 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "fuse_dfs.h"
+#include "fuse_impls.h"
+#include "fuse_connect.h"
+
+
+int dfs_statfs(const char *path, struct statvfs *st)
+{
+  TRACE1("statfs",path)
+
+  // retrieve dfs specific data
+  dfs_context *dfs = (dfs_context*)fuse_get_context()->private_data;
+
+  // check params and the context var
+  assert(path);
+  assert(st);
+  assert(dfs);
+
+  // init the stat structure
+  memset(st,0,sizeof(struct statvfs));
+
+  hdfsFS userFS;
+  // if not connected, try to connect and fail out if we can't.
+  if ((userFS = doConnectAsUser(dfs->nn_hostname,dfs->nn_port))== NULL) {
+    syslog(LOG_ERR, "ERROR: could not connect to dfs %s:%d\n", __FILE__, __LINE__);
+    return -EIO;
+  }
+
+  const long cap   = hdfsGetCapacity(userFS);
+  const long used  = hdfsGetUsed(userFS);
+  const long bsize = hdfsGetDefaultBlockSize(userFS);
+
+  // fill in the statvfs structure
+
+  /* FOR REFERENCE:
+     struct statvfs {
+     unsigned long  f_bsize;    // file system block size
+     unsigned long  f_frsize;   // fragment size
+     fsblkcnt_t     f_blocks;   // size of fs in f_frsize units
+     fsblkcnt_t     f_bfree;    // # free blocks
+     fsblkcnt_t     f_bavail;   // # free blocks for non-root
+     fsfilcnt_t     f_files;    // # inodes
+     fsfilcnt_t     f_ffree;    // # free inodes
+     fsfilcnt_t     f_favail;   // # free inodes for non-root
+     unsigned long  f_fsid;     // file system id
+     unsigned long  f_flag;     / mount flags
+     unsigned long  f_namemax;  // maximum filename length
+     };
+  */
+
+  st->f_bsize   =  bsize;
+  st->f_frsize  =  bsize;
+
+  st->f_blocks  =  cap/bsize;
+
+  st->f_bfree   =  (cap-used)/bsize;
+  st->f_bavail  =  (cap-used)/bsize;
+
+  st->f_files   =  1000;
+  st->f_ffree   =  500;
+  st->f_favail  =  500;
+  st->f_fsid    =  1023;
+  st->f_flag    =  ST_RDONLY | ST_NOSUID;
+  st->f_namemax =  1023;
+
+  return 0;
+}
+

src/contrib/fuse-dfs/src/fuse_impls_symlink.c

@@ -0,0 +1,30 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "fuse_dfs.h"
+#include "fuse_impls.h"
+
+
+int dfs_symlink(const char *from, const char *to)
+{
+  TRACE1("symlink", from)
+  (void)from;
+  (void)to;
+  // bugbug we need the FileSystem to support this posix API
+  return -ENOTSUP;
+}

src/contrib/fuse-dfs/src/fuse_impls_truncate.c

@@ -0,0 +1,67 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "fuse_dfs.h"
+#include "fuse_impls.h"
+#include "fuse_connect.h"
+
+/**
+ * For now implement truncate here and only for size == 0.
+ * Weak implementation in that we just delete the file and 
+ * then re-create it, but don't set the user, group, and times to the old
+ * file's metadata. 
+ */
+int dfs_truncate(const char *path, off_t size)
+{
+  TRACE1("truncate", path)
+  if (size != 0) {
+    return -ENOTSUP;
+  }
+
+  dfs_context *dfs = (dfs_context*)fuse_get_context()->private_data;
+
+  assert(path);
+  assert('/' == *path);
+  assert(dfs);
+
+  int ret = dfs_unlink(path);
+  if (ret != 0) {
+    return ret;
+  }
+
+  hdfsFS userFS;
+  // if not connected, try to connect and fail out if we can't.
+  if ((userFS = doConnectAsUser(dfs->nn_hostname,dfs->nn_port)) == NULL) {
+    syslog(LOG_ERR, "ERROR: could not connect to dfs %s:%d\n", __FILE__, __LINE__);
+    return -EIO;
+  }
+
+  int flags = O_WRONLY | O_CREAT;
+
+  hdfsFile file;
+  if ((file = (hdfsFile)hdfsOpenFile(userFS, path, flags,  0, 3, 0)) == NULL) {
+    syslog(LOG_ERR, "ERROR: could not connect open file %s:%d\n", __FILE__, __LINE__);
+    return -EIO;
+  }
+
+  if (hdfsCloseFile(userFS, file) != 0) {
+    syslog(LOG_ERR, "ERROR: could not connect close file %s:%d\n", __FILE__, __LINE__);
+    return -EIO;
+  }
+  return 0;
+}

src/contrib/fuse-dfs/src/fuse_impls_unlink.c

@@ -0,0 +1,62 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "fuse_dfs.h"
+#include "fuse_impls.h"
+#include "fuse_connect.h"
+#include "fuse_trash.h"
+extern const char *const TrashPrefixDir;
+
+int dfs_unlink(const char *path)
+{
+  TRACE1("unlink", path)
+
+  // retrieve dfs specific data
+  dfs_context *dfs = (dfs_context*)fuse_get_context()->private_data;
+
+  // check params and the context var
+  assert(path);
+  assert(dfs);
+  assert('/' == *path);
+
+  if (is_protected(path)) {
+    syslog(LOG_ERR,"ERROR: hdfs trying to delete a protected directory: %s ",path);
+    return -EACCES;
+  }
+
+  if (dfs->read_only) {
+    syslog(LOG_ERR,"ERROR: hdfs is configured as read-only, cannot create the directory %s\n",path);
+    return -EACCES;
+  }
+
+  hdfsFS userFS;
+  // if not connected, try to connect and fail out if we can't.
+  if ((userFS = doConnectAsUser(dfs->nn_hostname,dfs->nn_port))== NULL) {
+    syslog(LOG_ERR, "ERROR: could not connect to dfs %s:%d\n", __FILE__, __LINE__);
+    return -EIO;
+  }
+
+
+  if (hdfsDeleteWithTrash(userFS, path, dfs->usetrash)) {
+    syslog(LOG_ERR,"ERROR: hdfs trying to delete the file %s",path);
+    return -EIO;
+  }
+
+  return 0;
+
+}

src/contrib/fuse-dfs/src/fuse_impls_utimens.c

@@ -0,0 +1,52 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "fuse_dfs.h"
+#include "fuse_impls.h"
+#include "fuse_connect.h"
+
+ int dfs_utimens(const char *path, const struct timespec ts[2])
+{
+  TRACE1("utimens", path)
+#if PERMS
+  // retrieve dfs specific data
+  dfs_context *dfs = (dfs_context*)fuse_get_context()->private_data;
+
+  // check params and the context var
+  assert(path);
+  assert(dfs);
+  assert('/' == *path);
+
+  time_t aTime = ts[0].tv_sec;
+  time_t mTime = ts[1].tv_sec;
+
+  hdfsFS userFS;
+  // if not connected, try to connect and fail out if we can't.
+  if ((userFS = doConnectAsUser(dfs->nn_hostname,dfs->nn_port))== NULL) {
+    syslog(LOG_ERR, "ERROR: could not connect to dfs %s:%d\n", __FILE__, __LINE__);
+    return -EIO;
+  }
+
+  if (hdfsUtime(userFS, path, mTime, aTime)) {
+    syslog(LOG_ERR,"ERROR: hdfs trying to utime %s to %ld/%ld",path, (long)mTime, (long)aTime);
+    fprintf(stderr,"ERROR: could not set utime for path %s\n",path);
+    return -EIO;
+  }
+#endif  
+  return 0;
+}

src/contrib/fuse-dfs/src/fuse_impls_write.c

@@ -0,0 +1,75 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "fuse_dfs.h"
+#include "fuse_impls.h"
+#include "fuse_file_handle.h"
+
+int dfs_write(const char *path, const char *buf, size_t size,
+                     off_t offset, struct fuse_file_info *fi)
+{
+  TRACE1("write", path)
+
+  // retrieve dfs specific data
+  dfs_context *dfs = (dfs_context*)fuse_get_context()->private_data;
+  int ret = 0;
+
+  // check params and the context var
+  assert(path);
+  assert(dfs);
+  assert('/' == *path);
+  assert(fi);
+
+  dfs_fh *fh = (dfs_fh*)fi->fh;
+  assert(fh);
+
+  hdfsFile file_handle = (hdfsFile)fh->hdfsFH;
+  assert(file_handle);
+
+  //
+  // Critical section - make the sanity check (tell to see the writes are sequential) and the actual write 
+  // (no returns until end)
+  //
+  pthread_mutex_lock(&fh->mutex);
+
+  tSize length = 0;
+  assert(fh->fs);
+
+  tOffset cur_offset = hdfsTell(fh->fs, file_handle);
+  if (cur_offset != offset) {
+    syslog(LOG_ERR, "ERROR: user trying to random access write to a file %d!=%d for %s %s:%d\n",(int)cur_offset, (int)offset,path, __FILE__, __LINE__);
+    ret =  -EIO;
+  } else {
+    length = hdfsWrite(fh->fs, file_handle, buf, size);
+    if (length <= 0) {
+      syslog(LOG_ERR, "ERROR: fuse problem - could not write all the bytes for %s %d!=%d%s:%d\n",path,length,(int)size, __FILE__, __LINE__);
+      ret = -EIO;
+    } 
+    if (length != size) {
+      syslog(LOG_ERR, "WARN: fuse problem - could not write all the bytes for %s %d!=%d%s:%d\n",path,length,(int)size, __FILE__, __LINE__);
+    }
+  }
+
+  //
+  // Critical section end 
+  //
+
+  pthread_mutex_unlock(&fh->mutex);
+
+  return ret == 0 ? length : ret;
+}

src/contrib/fuse-dfs/src/fuse_init.c

@@ -0,0 +1,139 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include <strings.h>
+
+#include "fuse_dfs.h"
+#include "fuse_init.h"
+#include "fuse_options.h"
+#include "fuse_context_handle.h"
+
+// Hacked up function to basically do:
+//  protectedpaths = split(options.protected,':');
+
+void init_protectedpaths(dfs_context *dfs) {
+
+  char *tmp = options.protected;
+
+
+  // handle degenerate case up front.
+  if (tmp == NULL || 0 == *tmp) {
+    dfs->protectedpaths = (char**)malloc(sizeof(char*));
+    dfs->protectedpaths[0] = NULL;
+    return;
+  }
+  assert(tmp);
+
+  if (options.debug) {
+    print_options();
+  }
+
+
+  int i = 0;
+  while (tmp && (NULL != (tmp = index(tmp,':')))) {
+    tmp++; // pass the ,
+    i++;
+  }
+  i++; // for the last entry
+  i++; // for the final NULL
+  dfs->protectedpaths = (char**)malloc(sizeof(char*)*i);
+  assert(dfs->protectedpaths);
+  tmp = options.protected;
+  int j  = 0;
+  while (NULL != tmp && j < i) {
+    int length;
+    char *eos = index(tmp,':');
+    if (NULL != eos) {
+      length = eos - tmp; // length of this value
+    } else {
+      length = strlen(tmp);
+    }
+    dfs->protectedpaths[j] = (char*)malloc(sizeof(char)*length+1);
+    assert(dfs->protectedpaths[j]);
+    strncpy(dfs->protectedpaths[j], tmp, length);
+    dfs->protectedpaths[j][length] = '\0';
+    if (eos) {
+      tmp = eos + 1;
+    } else {
+      tmp = NULL;
+    }
+    j++;
+  }
+  dfs->protectedpaths[j] = NULL;
+
+  /*
+    j  = 0;
+    while (dfs->protectedpaths[j]) {
+    printf("dfs->protectedpaths[%d]=%s\n",j,dfs->protectedpaths[j]);
+    fflush(stdout);
+    j++;
+    }
+    exit(1);
+  */
+}
+
+void *dfs_init()
+{
+
+  //
+  // Create a private struct of data we will pass to fuse here and which
+  // will then be accessible on every call.
+  //
+  dfs_context *dfs = (dfs_context*)malloc(sizeof (dfs_context));
+
+  if (NULL == dfs) {
+    syslog(LOG_ERR, "FATAL: could not malloc fuse dfs context struct - out of memory %s:%d", __FILE__, __LINE__);
+    exit(1);
+  }
+
+  // initialize the context
+  dfs->debug                 = options.debug;
+  dfs->nn_hostname           = options.server;
+  dfs->nn_port               = options.port;
+  dfs->fs                    = NULL;
+  dfs->read_only             = options.read_only;
+  dfs->usetrash              = options.usetrash;
+  dfs->protectedpaths        = NULL;
+  dfs->rdbuffer_size         = options.rdbuffer_size;
+  dfs->direct_io             = options.direct_io;
+
+  bzero(dfs->dfs_uri,0);
+  sprintf(dfs->dfs_uri,"dfs://%s:%d/",dfs->nn_hostname,dfs->nn_port);
+  dfs->dfs_uri_len = strlen(dfs->dfs_uri);
+
+  // use ERR level to ensure it makes it into the log.
+  syslog(LOG_ERR, "mounting %s", dfs->dfs_uri);
+
+  init_protectedpaths(dfs);
+  assert(dfs->protectedpaths != NULL);
+
+  if (dfs->rdbuffer_size <= 0) {
+    syslog(LOG_DEBUG, "WARN: dfs->rdbuffersize <= 0 = %ld %s:%d", dfs->rdbuffer_size, __FILE__, __LINE__);
+    dfs->rdbuffer_size = 32768;
+  }
+  return (void*)dfs;
+}
+
+
+
+void dfs_destroy (void *ptr)
+{
+  TRACE("destroy")
+  dfs_context *dfs = (dfs_context*)ptr;
+  dfs->fs = NULL;
+}

src/contrib/fuse-dfs/src/fuse_init.h

@@ -0,0 +1,31 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef __FUSE_INIT_H__
+#define __FUSE_INIT_H__
+
+/**
+ * These are responsible for initializing connections to dfs and internal
+ * data structures and then freeing them.
+ * i.e., what happens on mount and unmount.
+ *
+ */
+void *dfs_init();
+void dfs_destroy (void *ptr);
+
+#endif

src/contrib/fuse-dfs/src/fuse_options.c

@@ -0,0 +1,165 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "fuse_dfs.h"
+#include "fuse_options.h"
+#include <getopt.h>
+
+#include "fuse_context_handle.h"
+
+void print_options() {
+  fprintf(stderr,"options:\n");
+  fprintf(stderr, "\tprotected=%s\n",options.protected);
+  fprintf(stderr, "\tserver=%s\n",options.server);
+  fprintf(stderr, "\tport=%d\n",options.port);
+  fprintf(stderr, "\tdebug=%d\n",options.debug);
+  fprintf(stderr, "\tread_only=%d\n",options.read_only);
+  fprintf(stderr, "\tusetrash=%d\n",options.usetrash);
+  fprintf(stderr, "\tentry_timeout=%d\n",options.entry_timeout);
+  fprintf(stderr, "\tattribute_timeout=%d\n",options.attribute_timeout);
+  fprintf(stderr, "\tprivate=%d\n",options.private);
+  fprintf(stderr, "\trdbuffer_size=%d (KBs)\n",(int)options.rdbuffer_size/1024);
+}
+
+const char *program;  
+
+
+/** macro to define options */
+#define DFSFS_OPT_KEY(t, p, v) { t, offsetof(struct options, p), v }
+
+void print_usage(const char *pname)
+{
+  fprintf(stdout,"USAGE: %s [debug] [--help] [--version] [-oprotected=<colon_seped_list_of_paths] [rw] [-onotrash] [-ousetrash] [-obig_writes] [-oprivate (single user)] [ro] [-oserver=<hadoop_servername>] [-oport=<hadoop_port>] [-oentry_timeout=<secs>] [-oattribute_timeout=<secs>] [-odirect_io] [-onopoermissions] [-o<other fuse option>] <mntpoint> [fuse options]\n",pname);
+  fprintf(stdout,"NOTE: debugging option for fuse is -debug\n");
+}
+
+
+/** keys for FUSE_OPT_ options */
+enum
+  {
+    KEY_VERSION,
+    KEY_HELP,
+    KEY_USETRASH,
+    KEY_NOTRASH,
+    KEY_RO,
+    KEY_RW,
+    KEY_PRIVATE,
+    KEY_BIGWRITES,
+    KEY_DEBUG,
+    KEY_INITCHECKS,
+    KEY_NOPERMISSIONS,
+    KEY_DIRECTIO,
+  };
+
+struct fuse_opt dfs_opts[] =
+  {
+    DFSFS_OPT_KEY("server=%s", server, 0),
+    DFSFS_OPT_KEY("entry_timeout=%d", entry_timeout, 0),
+    DFSFS_OPT_KEY("attribute_timeout=%d", attribute_timeout, 0),
+    DFSFS_OPT_KEY("protected=%s", protected, 0),
+    DFSFS_OPT_KEY("port=%d", port, 0),
+    DFSFS_OPT_KEY("rdbuffer=%d", rdbuffer_size,0),
+
+    FUSE_OPT_KEY("private", KEY_PRIVATE),
+    FUSE_OPT_KEY("ro", KEY_RO),
+    FUSE_OPT_KEY("debug", KEY_DEBUG),
+    FUSE_OPT_KEY("initchecks", KEY_INITCHECKS),
+    FUSE_OPT_KEY("nopermissions", KEY_NOPERMISSIONS),
+    FUSE_OPT_KEY("big_writes", KEY_BIGWRITES),
+    FUSE_OPT_KEY("rw", KEY_RW),
+    FUSE_OPT_KEY("usetrash", KEY_USETRASH),
+    FUSE_OPT_KEY("notrash", KEY_NOTRASH),
+    FUSE_OPT_KEY("direct_io", KEY_DIRECTIO),
+    FUSE_OPT_KEY("-v",             KEY_VERSION),
+    FUSE_OPT_KEY("--version",      KEY_VERSION),
+    FUSE_OPT_KEY("-h",             KEY_HELP),
+    FUSE_OPT_KEY("--help",         KEY_HELP),
+    FUSE_OPT_END
+  };
+
+int dfs_options(void *data, const char *arg, int key,  struct fuse_args *outargs)
+{
+  (void) data;
+
+  switch (key) {
+  case FUSE_OPT_KEY_OPT:
+    fprintf(stderr,"fuse-dfs ignoring option %s\n",arg);
+    return 1;
+  case  KEY_VERSION:
+    fprintf(stdout,"%s %s\n",program,_FUSE_DFS_VERSION);
+    exit(0);
+  case KEY_HELP:
+    print_usage(program);
+    exit(0);
+  case KEY_USETRASH:
+    options.usetrash = 1;
+    break;
+  case KEY_NOTRASH:
+    options.usetrash = 1;
+    break;
+  case KEY_RO:
+    options.read_only = 1;
+    break;
+  case KEY_RW:
+    options.read_only = 0;
+    break;
+  case KEY_PRIVATE:
+    options.private = 1;
+    break;
+  case KEY_DEBUG:
+    fuse_opt_add_arg(outargs, "-d");
+    options.debug = 1;
+    break;
+  case KEY_INITCHECKS:
+    options.initchecks = 1;
+    break;
+  case KEY_NOPERMISSIONS:
+    options.no_permissions = 1;
+    break;
+  case KEY_DIRECTIO:
+    options.direct_io = 1;
+    break;
+  case KEY_BIGWRITES:
+#ifdef FUSE_CAP_BIG_WRITES
+    fuse_opt_add_arg(outargs, "-obig_writes");
+#endif
+    break;
+  default: {
+    // try and see if the arg is a URI for DFS
+    int tmp_port;
+    char tmp_server[1024];
+
+    if (!sscanf(arg,"dfs://%1024[a-zA-Z0-9_.-]:%d",tmp_server,&tmp_port)) {
+      if (strcmp(arg,"ro") == 0) {
+        options.read_only = 1;
+      } else if (strcmp(arg,"rw") == 0) {
+        options.read_only = 0;
+      } else {
+        fprintf(stderr,"fuse-dfs didn't recognize %s,%d\n",arg,key);
+        fuse_opt_add_arg(outargs,arg);
+        return 0;
+      }
+    } else {
+      options.port = tmp_port;
+      options.server = strdup(tmp_server);
+      fprintf(stderr, "port=%d,server=%s\n", options.port, options.server);
+    }
+  }
+  }
+  return 0;
+}

src/contrib/fuse-dfs/src/fuse_options.h

@@ -0,0 +1,44 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef __FUSE_OPTIONS_H__
+#define __FUSE_OPTIONS_H__
+
+/** options for fuse_opt.h */
+struct options {
+  char* protected;
+  char* server;
+  int port;
+  int debug;
+  int read_only;
+  int initchecks;
+  int no_permissions;
+  int usetrash;
+  int entry_timeout;
+  int attribute_timeout;
+  int private;
+  size_t rdbuffer_size;
+  int direct_io;
+} options;
+
+extern struct fuse_opt dfs_opts[];
+void print_options();
+void print_usage(const char *pname);
+int dfs_options(void *data, const char *arg, int key,  struct fuse_args *outargs);
+
+#endif

src/contrib/fuse-dfs/src/fuse_stat_struct.c

@@ -0,0 +1,123 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include <math.h>
+#include <pthread.h>
+#include <grp.h>
+#include <pwd.h>
+
+#include "fuse_dfs.h"
+#include "fuse_stat_struct.h"
+#include "fuse_context_handle.h"
+
+#if PERMS
+/**
+ * getpwuid and getgrgid return static structs so we safeguard the contents
+ * while retrieving fields using the 2 structs below.
+ * NOTE: if using both, always get the passwd struct firt!
+ */
+extern pthread_mutex_t passwdstruct_mutex; 
+extern pthread_mutex_t groupstruct_mutex;
+#endif
+
+
+const int default_id       = 99; // nobody  - not configurable since soon uids in dfs, yeah!
+const int blksize = 512;
+
+/**
+ * Converts from a hdfs hdfsFileInfo to a POSIX stat struct
+ *
+ */
+int fill_stat_structure(hdfsFileInfo *info, struct stat *st) 
+{
+  assert(st);
+  assert(info);
+
+  // initialize the stat structure
+  memset(st, 0, sizeof(struct stat));
+
+  // by default: set to 0 to indicate not supported for directory because we cannot (efficiently) get this info for every subdirectory
+  st->st_nlink = (info->mKind == kObjectKindDirectory) ? 0 : 1;
+
+  uid_t owner_id = default_id;
+#if PERMS
+  if (info->mOwner != NULL) {
+    //
+    // Critical section - protect from concurrent calls in different threads since
+    // the struct below is static.
+    // (no returns until end)
+    //
+    pthread_mutex_lock(&passwdstruct_mutex);
+
+    struct passwd *passwd_info = getpwnam(info->mOwner);
+    owner_id = passwd_info == NULL ? default_id : passwd_info->pw_uid;
+
+    //
+    // End critical section 
+    // 
+    pthread_mutex_unlock(&passwdstruct_mutex);
+
+  } 
+#endif
+  gid_t group_id = default_id;
+#if PERMS
+  if (info->mGroup != NULL) {
+    //
+    // Critical section - protect from concurrent calls in different threads since
+    // the struct below is static.
+    // (no returns until end)
+    //
+    pthread_mutex_lock(&groupstruct_mutex);
+
+    struct group *grp = getgrnam(info->mGroup);
+    group_id = grp == NULL ? default_id : grp->gr_gid;
+
+    //
+    // End critical section 
+    // 
+    pthread_mutex_unlock(&groupstruct_mutex);
+
+  }
+#endif
+
+  short perm = (info->mKind == kObjectKindDirectory) ? (S_IFDIR | 0777) :  (S_IFREG | 0666);
+#if PERMS
+  if (info->mPermissions > 0) {
+    perm = (info->mKind == kObjectKindDirectory) ? S_IFDIR:  S_IFREG ;
+    perm |= info->mPermissions;
+  }
+#endif
+
+  // set stat metadata
+  st->st_size     = (info->mKind == kObjectKindDirectory) ? 4096 : info->mSize;
+  st->st_blksize  = blksize;
+  st->st_blocks   =  ceil(st->st_size/st->st_blksize);
+  st->st_mode     = perm;
+  st->st_uid      = owner_id;
+  st->st_gid      = group_id;
+#if PERMS
+  st->st_atime    = info->mLastAccess;
+#else
+  st->st_atime    = info->mLastMod;
+#endif
+  st->st_mtime    = info->mLastMod;
+  st->st_ctime    = info->mLastMod;
+
+  return 0;
+}
+

src/contrib/fuse-dfs/src/fuse_stat_struct.h

@@ -0,0 +1,36 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef __FUSE_STAT_STRUCT_H__
+#define __FUSE_STAT_STRUCT_H__
+
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <unistd.h>
+
+#include "hdfs.h"
+
+/**
+ * Converts from a hdfs hdfsFileInfo to a POSIX stat struct
+ * Should be thread safe.
+ */
+int fill_stat_structure(hdfsFileInfo *info, struct stat *st) ;
+
+extern const int default_id;
+extern const int blksize;
+#endif

src/contrib/fuse-dfs/src/fuse_trash.c

@@ -0,0 +1,125 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+#include <hdfs.h>
+#include <strings.h>
+
+#include "fuse_dfs.h"
+#include "fuse_trash.h"
+#include "fuse_context_handle.h"
+
+
+const char *const TrashPrefixDir = "/user/root/.Trash";
+const char *const TrashDir = "/user/root/.Trash/Current";
+
+#define TRASH_RENAME_TRIES  100
+
+//
+// NOTE: this function is a c implementation of org.apache.hadoop.fs.Trash.moveToTrash(Path path).
+//
+
+int move_to_trash(const char *item, hdfsFS userFS) {
+
+  // retrieve dfs specific data
+  dfs_context *dfs = (dfs_context*)fuse_get_context()->private_data;
+
+  // check params and the context var
+  assert(item);
+  assert(dfs);
+  assert('/' == *item);
+  assert(rindex(item,'/') >= 0);
+
+
+  char fname[4096]; // or last element of the directory path
+  char parent_directory[4096]; // the directory the fname resides in
+
+  if (strlen(item) > sizeof(fname) - strlen(TrashDir)) {
+    syslog(LOG_ERR, "ERROR: internal buffer too small to accomodate path of length %d %s:%d\n", (int)strlen(item), __FILE__, __LINE__);
+    return -EIO;
+  }
+
+  // separate the file name and the parent directory of the item to be deleted
+  {
+    int length_of_parent_dir = rindex(item, '/') - item ;
+    int length_of_fname = strlen(item) - length_of_parent_dir - 1; // the '/'
+
+    // note - the below strncpys should be safe from overflow because of the check on item's string length above.
+    strncpy(parent_directory, item, length_of_parent_dir);
+    parent_directory[length_of_parent_dir ] = 0;
+    strncpy(fname, item + length_of_parent_dir + 1, strlen(item));
+    fname[length_of_fname + 1] = 0;
+  }
+
+  // create the target trash directory
+  char trash_dir[4096];
+  if (snprintf(trash_dir, sizeof(trash_dir), "%s%s",TrashDir,parent_directory) >= sizeof trash_dir) {
+    syslog(LOG_ERR, "move_to_trash error target is not big enough to hold new name for %s %s:%d\n",item, __FILE__, __LINE__);
+    return -EIO;
+  }
+
+  // create the target trash directory in trash (if needed)
+  if ( hdfsExists(userFS, trash_dir)) {
+    // make the directory to put it in in the Trash - NOTE
+    // hdfsCreateDirectory also creates parents, so Current will be created if it does not exist.
+    if (hdfsCreateDirectory(userFS, trash_dir)) {
+      return -EIO;
+    }
+  }
+
+  //
+  // if the target path in Trash already exists, then append with
+  // a number. Start from 1.
+  //
+  char target[4096];
+  int j ;
+  if ( snprintf(target, sizeof target,"%s/%s",trash_dir, fname) >= sizeof target) {
+    syslog(LOG_ERR, "move_to_trash error target is not big enough to hold new name for %s %s:%d\n",item, __FILE__, __LINE__);
+    return -EIO;
+  }
+
+  // NOTE: this loop differs from the java version by capping the #of tries
+  for (j = 1; ! hdfsExists(userFS, target) && j < TRASH_RENAME_TRIES ; j++) {
+    if (snprintf(target, sizeof target,"%s/%s.%d",trash_dir, fname, j) >= sizeof target) {
+      syslog(LOG_ERR, "move_to_trash error target is not big enough to hold new name for %s %s:%d\n",item, __FILE__, __LINE__);
+      return -EIO;
+    }
+  }
+  if (hdfsRename(userFS, item, target)) {
+    syslog(LOG_ERR,"ERROR: hdfs trying to rename %s to %s",item, target);
+    return -EIO;
+  }
+  return 0;
+} 
+
+
+int hdfsDeleteWithTrash(hdfsFS userFS, const char *path, int useTrash) {
+
+  // move the file to the trash if this is enabled and its not actually in the trash.
+  if (useTrash && strncmp(path, TrashPrefixDir, strlen(TrashPrefixDir)) != 0) {
+    int ret= move_to_trash(path, userFS);
+    return ret;
+  }
+
+  if (hdfsDelete(userFS, path, 1)) {
+    syslog(LOG_ERR,"ERROR: hdfs trying to delete the file %s",path);
+    return -EIO;
+  }
+  return 0;
+
+}

src/contrib/fuse-dfs/src/fuse_trash.h

@@ -0,0 +1,26 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef __FUSE_TRASH_H__
+#define __FUSE_TRASH_H__
+
+#include <hdfs.h>
+
+int hdfsDeleteWithTrash(hdfsFS userFS, const char *path, int useTrash);
+
+#endif

src/contrib/fuse-dfs/src/fuse_users.c

@@ -0,0 +1,221 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+#include <pthread.h>
+#include <grp.h>
+#include <pwd.h>
+#include <stdlib.h>
+
+#include "fuse_dfs.h"
+
+
+
+#if PERMS
+/**
+ * getpwuid and getgrgid return static structs so we safeguard the contents
+ * while retrieving fields using the 2 structs below.
+ * NOTE: if using both, always get the passwd struct firt!
+ */
+pthread_mutex_t passwdstruct_mutex = PTHREAD_MUTEX_INITIALIZER;
+pthread_mutex_t groupstruct_mutex = PTHREAD_MUTEX_INITIALIZER;
+#endif
+
+#if PERMS
+
+/**
+ * Utility for getting the user making the fuse call in char * form
+ * NOTE: if non-null return, the return must be freed by the caller.
+ */
+char *getUsername(uid_t uid)
+{
+  //
+  // Critical section - protect from concurrent calls in different threads.
+  // since the struct below is static.
+  // (no returns until end)
+  //
+
+  pthread_mutex_lock(&passwdstruct_mutex);
+
+  struct passwd *userinfo = getpwuid(uid);
+  char * ret = userinfo && userinfo->pw_name ? strdup(userinfo->pw_name) : NULL;
+
+  pthread_mutex_unlock(&passwdstruct_mutex);
+
+  //
+  // End critical section 
+  // 
+  return ret;
+}
+
+/**
+ * Cleans up a char ** group pointer
+ */
+
+void freeGroups(char **groups, int numgroups) {
+  if (groups == NULL) {
+    return;
+  }
+  int i ;
+  for (i = 0; i < numgroups; i++) {
+    free(groups[i]);
+  }
+  free(groups);
+}
+
+#define GROUPBUF_SIZE 5
+
+char *getGroup(gid_t gid) {
+  //
+  // Critical section - protect from concurrent calls in different threads.
+  // since the struct below is static.
+  // (no returns until end)
+  //
+
+  pthread_mutex_lock(&groupstruct_mutex);
+
+  struct group* grp = getgrgid(gid);
+  char * ret = grp && grp->gr_name ? strdup(grp->gr_name) : NULL;
+
+  //
+  // End critical section 
+  // 
+  pthread_mutex_unlock(&groupstruct_mutex);
+
+  return ret;
+}
+
+
+/**
+ * Utility for getting the group from the uid
+ * NOTE: if non-null return, the return must be freed by the caller.
+ */
+char *getGroupUid(uid_t uid) {
+  //
+  // Critical section - protect from concurrent calls in different threads
+  // since the structs below are static.
+  // (no returns until end)
+  //
+
+  pthread_mutex_lock(&passwdstruct_mutex);
+  pthread_mutex_lock(&groupstruct_mutex);
+
+  char *ret = NULL;
+  struct passwd *userinfo = getpwuid(uid);
+  if (NULL != userinfo) {
+    struct group* grp = getgrgid( userinfo->pw_gid);
+    ret = grp && grp->gr_name ? strdup(grp->gr_name) : NULL;
+  }
+
+  //
+  // End critical section 
+  // 
+  pthread_mutex_unlock(&groupstruct_mutex);
+  pthread_mutex_unlock(&passwdstruct_mutex);
+
+  return ret;
+}
+
+
+/**
+ * lookup the gid based on the uid
+ */
+gid_t getGidUid(uid_t uid) {
+  //
+  // Critical section - protect from concurrent calls in different threads
+  // since the struct below is static.
+  // (no returns until end)
+  //
+
+  pthread_mutex_lock(&passwdstruct_mutex);
+
+  struct passwd *userinfo = getpwuid(uid);
+  gid_t gid = userinfo == NULL ? 0 : userinfo->pw_gid;
+
+  //
+  // End critical section 
+  // 
+  pthread_mutex_unlock(&passwdstruct_mutex);
+
+  return gid;
+}
+
+/**
+ * Utility for getting the groups for the user making the fuse call in char * form
+ */
+char ** getGroups(uid_t uid, int *num_groups)
+{
+  char *user = getUsername(uid);
+
+  if (user == NULL)
+    return NULL;
+
+  char **groupnames = NULL;
+
+  // see http://www.openldap.org/lists/openldap-devel/199903/msg00023.html
+
+  //#define GETGROUPS_T 1 
+#ifdef GETGROUPS_T
+  *num_groups = GROUPBUF_SIZE;
+
+  gid_t* grouplist = malloc(GROUPBUF_SIZE * sizeof(gid_t)); 
+  assert(grouplist != NULL);
+  gid_t* tmp_grouplist; 
+  int rtr;
+
+  gid_t gid = getGidUid(uid);
+
+  if ((rtr = getgrouplist(user, gid, grouplist, num_groups)) == -1) {
+    // the buffer we passed in is < *num_groups
+    if ((tmp_grouplist = realloc(grouplist, *num_groups * sizeof(gid_t))) != NULL) {
+      grouplist = tmp_grouplist;
+      getgrouplist(user, gid, grouplist, num_groups);
+    }
+  }
+
+  groupnames = (char**)malloc(sizeof(char*)* (*num_groups) + 1);
+  assert(groupnames);
+  int i;
+  for (i=0; i < *num_groups; i++)  {
+    groupnames[i] = getGroup(grouplist[i]);
+    if (groupnames[i] == NULL) {
+      fprintf(stderr, "error could not lookup group %d\n",(int)grouplist[i]);
+    }
+  } 
+  free(grouplist);
+  assert(user != NULL);
+  groupnames[i] = user;
+  *num_groups = *num_groups + 1;
+#else
+
+  int i = 0;
+  assert(user != NULL);
+  groupnames[i] = user;
+  i++;
+
+  groupnames[i] = getGroupUid(uid);
+  if (groupnames[i]) {
+    i++;
+  }
+
+  *num_groups = i;
+
+#endif
+  return groupnames;
+}
+#endif

src/contrib/fuse-dfs/src/fuse_users.h

@@ -0,0 +1,70 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef __FUSE_USERS_H__
+#define __FUSE_USERS_H__
+
+#include <grp.h>
+#include <pwd.h>
+#include <pthread.h>
+
+/**
+ * Overall Note:
+ * 1. all these functions should be thread safe.
+ * 2. the ones that return char * or char **, generally require
+ * the caller to free the return value.
+ *
+ */
+
+
+/**
+ * Utility for getting the user making the fuse call in char * form
+ * NOTE: if non-null return, the return must be freed by the caller.
+ */
+char *getUsername(uid_t uid);
+
+
+/**
+ * Cleans up a char ** group pointer
+ */
+void freeGroups(char **groups, int numgroups);
+
+/**
+ * Lookup single group. Caller responsible for free of the return value
+ */
+char *getGroup(gid_t gid);
+
+/**
+ * Utility for getting the group from the uid
+ * NOTE: if non-null return, the return must be freed by the caller.
+ */
+char *getGroupUid(uid_t uid) ;
+
+
+/**
+ * lookup the gid based on the uid
+ */
+
+gid_t getGidUid(uid_t uid);
+
+/**
+ * Utility for getting the groups for the user making the fuse call in char * form
+ */
+char ** getGroups(uid_t uid, int *num_groups);
+
+#endif

src/contrib/fuse-dfs/src/test/TestFuseDFS.java

@@ -0,0 +1,626 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import org.apache.hadoop.hdfs.*;
+import junit.framework.TestCase;
+import java.io.*;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.*;
+import org.apache.hadoop.fs.permission.*;
+import java.net.*;
+
+/**
+ * This class tests that the Fuse module for DFS can mount properly
+ * and does a few simple commands:
+ * mkdir
+ * rmdir
+ * ls
+ * cat
+ *
+ * cp and touch are purposely not tested because they won't work with the current module
+
+ *
+ */
+public class TestFuseDFS extends TestCase {
+
+  /**
+   * mount the fuse file system using assumed fuse module library installed in /usr/local/lib or somewhere else on your
+   * pre-existing LD_LIBRARY_PATH
+   *
+   */
+
+  static Process fuse_process;
+  static String fuse_cmd;
+  static private void mount(String mountpoint, URI dfs) throws IOException, InterruptedException  {
+
+    String cp = System.getProperty("java.class.path");
+    Runtime r = Runtime.getRuntime();
+    fuse_cmd = System.getProperty("build.test") + "/../fuse_dfs";
+    String libhdfs = System.getProperty("build.test") + "/../../../libhdfs/";
+    String arch = System.getProperty("os.arch");
+    String jvm = System.getProperty("java.home") + "/lib/" + arch + "/server";
+    String lp = System.getProperty("LD_LIBRARY_PATH") + ":" + "/usr/local/lib:" + libhdfs + ":" + jvm;
+    System.err.println("LD_LIBRARY_PATH=" + lp);
+    String cmd[] =  {  fuse_cmd, "dfs://" + dfs.getHost() + ":" + String.valueOf(dfs.getPort()), 
+                       mountpoint, "-obig_writes", "-odebug", "-oentry_timeout=0.1",  "-oattribute_timeout=0.1", "-ousetrash", "rw", "-oinitchecks",
+                       "-ordbuffer=32768"};
+    final String [] envp = {
+      "CLASSPATH="+  cp,
+      "LD_LIBRARY_PATH=" + lp,
+      "PATH=" + "/usr/bin:/bin"
+
+    };
+
+    // ensure the mount point is not currently mounted
+    Process p = r.exec("fusermount -u " + mountpoint);
+    p.waitFor();
+
+    // clean up the mount point
+    p = r.exec("rm -rf " + mountpoint);
+    assertTrue(p.waitFor() == 0);
+
+    // make the mount point if needed
+    p = r.exec("mkdir -p " + mountpoint);
+    assertTrue(p.waitFor() == 0);
+
+    // mount fuse to the mount point
+    fuse_process = r.exec(cmd, envp);
+
+    // give DFS a chance to come up
+    try { Thread.sleep(3000); } catch(Exception e) { }
+  }
+
+  /**
+   * unmounts fuse for before shutting down.
+   */
+  static private void umount(String mpoint) throws IOException, InterruptedException {
+    Runtime r= Runtime.getRuntime();
+    Process p = r.exec("fusermount -u " + mpoint);
+    p.waitFor();
+  }
+
+  /**
+   * Set things up - create mini dfs cluster and mount the fuse filesystem.
+   */
+  public TestFuseDFS() throws IOException,InterruptedException  {
+  }
+
+  static private MiniDFSCluster cluster;
+  static private DistributedFileSystem fileSys;
+  final static private String mpoint;
+
+  static {
+    mpoint = System.getProperty("build.test") + "/mnt";
+    System.runFinalizersOnExit(true);
+    startStuff();
+  }
+
+
+  static public void startStuff() {
+    try {
+      Configuration conf = new Configuration();
+      conf.setBoolean("dfs.permissions",false);
+      cluster = new MiniDFSCluster(conf, 1, true, null);
+      fileSys = (DistributedFileSystem)cluster.getFileSystem();
+      assertTrue(fileSys.getFileStatus(new Path("/")).isDir());
+      mount(mpoint, fileSys.getUri());
+    } catch(Exception e) {
+      e.printStackTrace();
+    }
+  }
+
+  public void setUp() {
+  }
+
+  /**
+   * use shell to create a dir and then use filesys to see it exists.
+   */
+  public void testMkdir() throws IOException,InterruptedException, Exception  {
+    try {
+      // First create a new directory with mkdirs
+      Path path = new Path("/foo");
+      Runtime r = Runtime.getRuntime();
+      String cmd = "mkdir -p " + mpoint + path.toString();
+      Process p = r.exec(cmd);
+      assertTrue(p.waitFor() == 0);
+
+      // check it is there
+      assertTrue(fileSys.getFileStatus(path).isDir());
+
+      // check again through the shell
+      String lsCmd = "ls " + mpoint + path.toString();
+      p = r.exec(lsCmd);
+      assertTrue(p.waitFor() == 0);
+    } catch(Exception e) {
+      e.printStackTrace();
+      throw e;
+    }
+  }
+
+
+  /**
+   * use shell to create a dir and then use filesys to see it exists.
+   */
+  public void testWrites() throws IOException,InterruptedException  {
+    try {
+
+      // write a hello file
+      File file = new File(mpoint, "hello.txt");
+      FileOutputStream f = new FileOutputStream(file);
+      String s = "hello ";
+      f.write(s.getBytes());
+      s = "world";
+      f.write(s.getBytes());
+      f.flush();
+      f.close();
+
+
+      try {
+        Thread.sleep(1000);
+      } catch(Exception e) {
+      }
+
+      // check the file exists.
+      Path myPath = new Path("/hello.txt");
+      assertTrue(fileSys.exists(myPath));
+
+      // check the data is ok
+      FileInputStream fi = new FileInputStream(new File(mpoint, "hello.txt"));
+      byte b[] = new byte[12];
+      int length = fi.read(b,0,12);
+      assertTrue(length > 0);
+      String s2 = new String( b, 0, length);
+      assertEquals("hello world", s2);
+    } catch(Exception e) {
+      e.printStackTrace();
+    } finally {
+    }
+  }
+
+  /**
+   * Test ls for dir already created in testMkdDir also tests bad ls
+   */
+  public void testLs() throws IOException,InterruptedException  {
+    try {
+      // First create a new directory with mkdirs
+      Runtime r = Runtime.getRuntime();
+
+      // mkdir
+      Process p = r.exec("mkdir -p " + mpoint + "/test/mkdirs");
+      assertTrue(p.waitFor() == 0);
+
+      // ls
+      p = r.exec("ls " + mpoint + "/test/mkdirs");
+      assertTrue(p.waitFor() == 0);
+
+      // ls non-existant directory
+      p = r.exec("ls " + mpoint + "/test/mkdirsNotThere");
+      int res = p.waitFor();
+      assertFalse(res == 0);
+    } catch(Exception e) {
+      e.printStackTrace();
+    }
+
+  }
+
+  /**
+   * Remove a dir using the shell and use filesys to see it no longer exists.
+   */
+  public void testRmdir() throws IOException,InterruptedException  {
+    try {
+      // First create a new directory with mkdirs
+
+      Runtime r = Runtime.getRuntime();
+      Process p = r.exec("mkdir -p " + mpoint + "/test/rmdir");
+      assertTrue(p.waitFor() == 0);
+
+      Path myPath = new Path("/test/rmdir");
+      assertTrue(fileSys.exists(myPath));
+
+      // remove it
+      p = r.exec("rmdir " + mpoint + "/test/rmdir");
+      assertTrue(p.waitFor() == 0);
+
+      // check it is not there
+      assertFalse(fileSys.exists(myPath));
+
+      Path trashPath = new Path("/user/root/.Trash/Current/test/rmdir");
+      assertTrue(fileSys.exists(trashPath));
+
+      // make it again to test trashing same thing twice
+      p = r.exec("mkdir -p " + mpoint + "/test/rmdir");
+      assertTrue(p.waitFor() == 0);
+
+      assertTrue(fileSys.exists(myPath));
+
+      // remove it
+      p = r.exec("rmdir " + mpoint + "/test/rmdir");
+      assertTrue(p.waitFor() == 0);
+
+      // check it is not there
+      assertFalse(fileSys.exists(myPath));
+
+      trashPath = new Path("/user/root/.Trash/Current/test/rmdir.1");
+      assertTrue(fileSys.exists(trashPath));
+
+    } catch(Exception e) {
+      e.printStackTrace();
+    }
+  }
+
+  /**
+   * use shell to create a dir and then use filesys to see it exists.
+   */
+  public void testDF() throws IOException,InterruptedException, Exception  {
+    try {
+      // First create a new directory with mkdirs
+      Path path = new Path("/foo");
+      Runtime r = Runtime.getRuntime();
+      String cmd = "mkdir -p " + mpoint + path.toString();
+      Process p = r.exec(cmd);
+      assertTrue(p.waitFor() == 0);
+      File f = new File(mpoint + "/foo");
+
+      DistributedFileSystem.DiskStatus d = fileSys.getDiskStatus();
+
+      long fileUsedBlocks =  (f.getTotalSpace() - f.getFreeSpace())/(64 * 1024 * 1024);
+      long dfsUsedBlocks = (long)Math.ceil((double)d.getDfsUsed()/(64 * 1024 * 1024));
+
+      assertTrue(fileUsedBlocks == dfsUsedBlocks);
+      assertTrue(d.getCapacity() == f.getTotalSpace());
+
+    } catch(Exception e) {
+      e.printStackTrace();
+      throw e;
+    }
+  }
+
+  /**
+   * use shell to create a dir and then use filesys to see it exists.
+   */
+  public void testChown() throws IOException,InterruptedException, Exception  {
+    try {
+      // First create a new directory with mkdirs
+      Path path = new Path("/foo");
+      Runtime r = Runtime.getRuntime();
+      String cmd = "mkdir -p " + mpoint + path.toString();
+      Process p = r.exec(cmd);
+      assertTrue(p.waitFor() == 0);
+
+      // check it is there
+      assertTrue(fileSys.getFileStatus(path).isDir());
+
+      FileStatus foo = fileSys.getFileStatus(path);
+      System.err.println("DEBUG:owner=" + foo.getOwner());
+
+      cmd = "chown nobody " + mpoint + path.toString();
+      p = r.exec(cmd);
+      assertTrue(p.waitFor() == 0);
+
+      //      cmd = "chgrp nobody " + mpoint + path.toString();
+      //      p = r.exec(cmd);
+      //      assertTrue(p.waitFor() == 0);
+
+      foo = fileSys.getFileStatus(path);
+
+      System.err.println("DEBUG:owner=" + foo.getOwner());
+
+      assertTrue(foo.getOwner().equals("nobody"));
+      assertTrue(foo.getGroup().equals("nobody"));
+
+    } catch(Exception e) {
+      e.printStackTrace();
+      throw e;
+    }
+  }
+
+  /**
+   * use shell to create a dir and then use filesys to see it exists.
+   */
+  public void testChmod() throws IOException,InterruptedException, Exception  {
+    try {
+      // First create a new directory with mkdirs
+      Path path = new Path("/foo");
+      Runtime r = Runtime.getRuntime();
+      String cmd = "mkdir -p " + mpoint + path.toString();
+      Process p = r.exec(cmd);
+      assertTrue(p.waitFor() == 0);
+
+      // check it is there
+      assertTrue(fileSys.getFileStatus(path).isDir());
+
+      cmd = "chmod 777 " + mpoint + path.toString();
+      p = r.exec(cmd);
+      assertTrue(p.waitFor() == 0);
+
+      FileStatus foo = fileSys.getFileStatus(path);
+      FsPermission perm = foo.getPermission();
+      assertTrue(perm.toShort() == 0777);
+
+    } catch(Exception e) {
+      e.printStackTrace();
+      throw e;
+    }
+  }
+
+  /**
+   * use shell to create a dir and then use filesys to see it exists.
+   */
+  public void testUtimes() throws IOException,InterruptedException, Exception  {
+    try {
+      // First create a new directory with mkdirs
+      Path path = new Path("/utimetest");
+      Runtime r = Runtime.getRuntime();
+      String cmd = "touch " + mpoint + path.toString();
+      Process p = r.exec(cmd);
+      assertTrue(p.waitFor() == 0);
+
+      // check it is there
+      assertTrue(fileSys.exists(path));
+
+      FileStatus foo = fileSys.getFileStatus(path);
+      long oldTime = foo.getModificationTime();
+      try { Thread.sleep(1000); } catch(Exception e) {}
+
+      cmd = "touch " + mpoint + path.toString();
+      p = r.exec(cmd);
+      assertTrue(p.waitFor() == 0);
+
+      try { Thread.sleep(1000); } catch(Exception e) {}
+      foo = fileSys.getFileStatus(path);
+      long newTime = foo.getModificationTime();
+
+      assertTrue(newTime > oldTime);
+
+    } catch(Exception e) {
+      e.printStackTrace();
+      throw e;
+    } finally {
+    }
+  }
+
+  /**
+   *
+   * Test dfs_read on a file size that will trigger multiple internal reads. 
+   * First, just check raw size reading is ok and then check with smaller reads
+   * including checking the validity of the data read.
+   *
+   */
+  public void testReads() throws IOException,InterruptedException  {
+    try {
+      // First create a new directory with mkdirs
+      Runtime r = Runtime.getRuntime();
+      Process p;
+
+      // create the file
+      Path myPath = new Path("/test/hello.reads");
+      FSDataOutputStream s = fileSys.create(myPath);
+      String hello = "hello world!";
+      int written = 0;
+      int mycount = 0;
+      while(written < 1024 * 9) {
+        s.writeUTF(hello);
+        s.writeInt(mycount++);
+        written += hello.length() + 4;
+      }
+      s.close();
+
+      // check it exists
+      assertTrue(fileSys.exists(myPath));
+      FileStatus foo = fileSys.getFileStatus(myPath);
+      assertTrue(foo.getLen() >= 9 * 1024);
+
+
+      {
+        // cat the file
+        DataInputStream is = new DataInputStream(new FileInputStream(mpoint + "/test/hello.reads"));
+        byte buf [] = new byte[4096];
+        // test reading 0 length
+        assertTrue(is.read(buf, 0, 0) == 0);
+
+        // test real reads
+        assertTrue(is.read(buf, 0, 1024) == 1024);
+        assertTrue(is.read(buf, 0, 4096) == 4096);
+        assertTrue(is.read(buf, 0, 4096) == 4096);
+        is.close();
+      }
+
+      {
+        DataInputStream is = new DataInputStream(new FileInputStream(mpoint + "/test/hello.reads"));
+        int read = 0;
+        int counter = 0;
+        try {
+          while(true) {
+            String s2 = DataInputStream.readUTF(is);
+            int s3 = is.readInt();
+            assertTrue(s2.equals(hello));
+            assertTrue(s3 == counter++);
+            read += hello.length() + 4;
+          }
+        } catch(EOFException e) {
+          assertTrue(read >= 9 * 1024);
+        }
+      }
+
+      // check reading an empty file for EOF
+      {
+        // create the file
+        myPath = new Path("/test/hello.reads2");
+        s = fileSys.create(myPath);
+        s.close();
+
+        FSDataInputStream fs = fileSys.open(myPath);
+        assertEquals(-1,  fs.read());
+
+        FileInputStream f = new FileInputStream(mpoint + "/test/hello.reads2");
+        assertEquals(-1, f.read());
+      }
+
+    } catch(Exception e) {
+      e.printStackTrace();
+    } finally {
+    }
+  }
+
+
+  /**
+   * Use filesys to create the hello world! file and then cat it and see its contents are correct.
+   */
+  public void testCat() throws IOException,InterruptedException  {
+    try {
+      // First create a new directory with mkdirs
+      Runtime r = Runtime.getRuntime();
+      Process p = r.exec("rm -rf " + mpoint + "/test/hello");
+      assertTrue(p.waitFor() == 0);
+
+      // create the file
+      Path myPath = new Path("/test/hello");
+      FSDataOutputStream s = fileSys.create(myPath);
+      String hello = "hello world!";
+      s.writeUTF(hello);
+      s.writeInt(1033);
+      s.close();
+
+      // check it exists
+      assertTrue(fileSys.exists(myPath));
+
+      // cat the file
+      DataInputStream is = new DataInputStream(new FileInputStream(mpoint + "/test/hello"));
+      String s2 = DataInputStream.readUTF(is);
+      int s3 = is.readInt();
+      assertTrue(s2.equals(hello));
+      assertTrue(s3 == 1033);
+
+    } catch(Exception e) {
+      e.printStackTrace();
+    } finally {
+    }
+  }
+
+
+
+  /**
+   * Use filesys to create the hello world! file and then cat it and see its contents are correct.
+   */
+  public void testAppends() throws IOException,InterruptedException  {
+    try {
+      // First create a new directory with mkdirs
+      Runtime r = Runtime.getRuntime();
+
+      {
+        FileOutputStream os = new FileOutputStream(mpoint + "/appends");
+        String hello = "hello";
+        os.write(hello.getBytes());
+        os.flush();
+        os.close();
+      }
+
+      // check it exists
+      Path myPath = new Path("/appends");
+      assertTrue(fileSys.exists(myPath));
+
+      try {
+        Thread.sleep(1000);
+      } catch(Exception e) {
+      }
+
+      FileStatus foo = fileSys.getFileStatus(myPath);
+
+      File f = new File(mpoint + "/appends");
+      assertTrue(f.length() > 0);
+
+      {
+        FileOutputStream os = new FileOutputStream(mpoint + "/appends", true);
+        String hello = " world!";
+        os.write(hello.getBytes());
+        os.flush();
+        os.close();
+      }
+
+      // cat the file
+      FileInputStream is = new FileInputStream(mpoint + "/appends");
+      byte b[] = new byte[1024];
+      int len = is.read(b);
+      assertTrue(len > 0);
+      String s2 = new String(b,0,len);
+      assertTrue(s2.equals("hello world!"));
+
+    } catch(Exception e) {
+      e.printStackTrace();
+    } finally {
+    }
+  }
+
+
+
+
+  public void testDone() throws IOException {
+      close();
+  }
+
+  /**
+   * Unmount and close
+   */
+  protected void tearDown() throws Exception {
+  }
+
+  /**
+   * Unmount and close
+   */
+  protected void finalize() throws Throwable {
+    close();
+  }
+
+  public void close() {
+    try {
+      int length;
+
+      // print out the fuse debug output
+      {
+      do {
+      InputStream i = fuse_process.getInputStream();
+      byte b[] = new byte[i.available()];
+      length = i.read(b);
+      System.err.println("read x bytes: " + length);
+      System.err.write(b,0,b.length);
+      } while(length > 0) ;
+      }
+
+      do {
+      InputStream i = fuse_process.getErrorStream();
+      byte b[] = new byte[i.available()];
+      length = i.read(b);
+      System.err.println("read x bytes: " + length);
+      System.err.write(b,0,b.length);
+      } while(length > 0) ;
+
+      umount(mpoint);
+
+      fuse_process.destroy();
+      fuse_process = null;
+        if(fileSys != null) {
+        fileSys.close();
+        fileSys = null;
+      }
+      if(cluster != null) {
+        cluster.shutdown();
+        cluster = null;
+      }
+    } catch(Exception e) { }
+  }
+};

src/contrib/hdfsproxy/README

@@ -0,0 +1,51 @@
+HDFSPROXY is an HTTPS proxy server that exposes the same HSFTP interface as a 
+real cluster. It authenticates users via user certificates and enforce access 
+control based on configuration files.
+
+Starting up an HDFSPROXY server is similar to starting up an HDFS cluster. 
+Simply run "hdfsproxy" shell command. The main configuration file is 
+hdfsproxy-default.xml, which should be on the classpath. hdfsproxy-env.sh 
+can be used to set up environmental variables. In particular, JAVA_HOME should 
+be set. Additional configuration files include user-certs.xml, 
+user-permissions.xml and ssl-server.xml, which are used to specify allowed user
+certs, allowed directories/files, and ssl keystore information for the proxy, 
+respectively. The location of these files can be specified in 
+hdfsproxy-default.xml. Environmental variable HDFSPROXY_CONF_DIR can be used to
+point to the directory where these configuration files are located. The 
+configuration files of the proxied HDFS cluster should also be available on the
+classpath (hdfs-default.xml and hdfs-site.xml).
+
+Mirroring those used in HDFS, a few shell scripts are provided to start and 
+stop a group of proxy servers. The hosts to run hdfsproxy on are specified in 
+hdfsproxy-hosts file, one host per line. All hdfsproxy servers are stateless 
+and run independently from each other. Simple load balancing can be set up by 
+mapping all hdfsproxy server IP addresses to a single hostname. Users should 
+use that hostname to access the proxy. If an IP address look up for that 
+hostname returns more than one IP addresses, an HFTP/HSFTP client will randomly
+pick one to use.
+
+Command "hdfsproxy -reloadPermFiles" can be used to trigger reloading of 
+user-certs.xml and user-permissions.xml files on all proxy servers listed in 
+the hdfsproxy-hosts file. Similarly, "hdfsproxy -clearUgiCache" command can be 
+used to clear the UGI caches on all proxy servers.
+
+For tomcat based installation.
+1. set up the environment and configuration files. 
+	 a) export HADOOP_CONF_DIR=${user.home}/devel/source-conf
+	 	source-conf directory should point to the source cluster's configuration directory, 
+	 	where core-site.xml, and hdfs-site.xml should already be correctly configured for 
+	 	the source cluster settings.
+	 b) export HDFSPROXY_CONF_DIR=${user.home}/devel/proxy-conf
+	  proxy-conf directory should point to the proxy's configuration directory, where 
+	  hdfsproxy-default.xml, etc, should already be properly configured.
+
+2. cd ==> hdfsproxy directory,  ant war
+	 
+3. download and install tomcat6, change tomcat conf/server.xml file to include https support. 
+	 uncomment item below SSL HTTP/1.1 Connector and add paths, resulting something look like this:
+	 <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
+               maxThreads="150" scheme="https" secure="true" keystoreFile="${user.home}/grid/hdfsproxy-conf/server2.keystore" 
+               keystorePass="changeme" keystoreType="JKS"  clientAuth="true" sslProtocol="TLS" />
+4. copy war file in step 2 to tomcat's webapps directory and rename it to ROOT.war
+5. export JAVA_OPTS="-Djavax.net.ssl.trustStore=${user.home}/grid/hdfsproxy-conf/server2.keystore -Djavax.net.ssl.trustStorePassword=changeme"
+6. start up tomcat with tomcat's bin/startup.sh 

src/contrib/hdfsproxy/bin/hdfsproxy

@@ -0,0 +1,165 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+# The HdfsProxy command script
+#
+# Environment Variables
+#
+#   JAVA_HOME        The java implementation to use.  Overrides JAVA_HOME.
+#
+#   HDFSPROXY_CLASSPATH Extra Java CLASSPATH entries.
+#
+#   HDFSPROXY_HEAPSIZE  The maximum amount of heap to use, in MB. 
+#                    Default is 1000.
+#
+#   HDFSPROXY_OPTS      Extra Java runtime options.
+#   
+#   HDFSPROXY_NAMENODE_OPTS       These options are added to HDFSPROXY_OPTS 
+#   HDFSPROXY_CLIENT_OPTS         when the respective command is run.
+#   HDFSPROXY_{COMMAND}_OPTS etc  HDFSPROXY_JT_OPTS applies to JobTracker 
+#                              for e.g.  HDFSPROXY_CLIENT_OPTS applies to 
+#                              more than one command (fs, dfs, fsck, 
+#                              dfsadmin etc)  
+#
+#   HDFSPROXY_CONF_DIR  Alternate conf dir. Default is ${HDFSPROXY_HOME}/conf.
+#
+#   HDFSPROXY_ROOT_LOGGER The root appender. Default is INFO,console
+#
+
+bin=`dirname "$0"`
+bin=`cd "$bin"; pwd`
+
+. "$bin"/hdfsproxy-config.sh
+
+cygwin=false
+case "`uname`" in
+CYGWIN*) cygwin=true;;
+esac
+
+if [ -f "${HDFSPROXY_CONF_DIR}/hdfsproxy-env.sh" ]; then
+  . "${HDFSPROXY_CONF_DIR}/hdfsproxy-env.sh"
+fi
+
+# some Java parameters
+if [ "$JAVA_HOME" != "" ]; then
+  #echo "run java in $JAVA_HOME"
+  JAVA_HOME=$JAVA_HOME
+fi
+  
+if [ "$JAVA_HOME" = "" ]; then
+  echo "Error: JAVA_HOME is not set."
+  exit 1
+fi
+
+JAVA=$JAVA_HOME/bin/java
+JAVA_HEAP_MAX=-Xmx1000m 
+
+# check envvars which might override default args
+if [ "$HDFSPROXY_HEAPSIZE" != "" ]; then
+  #echo "run with heapsize $HDFSPROXY_HEAPSIZE"
+  JAVA_HEAP_MAX="-Xmx""$HDFSPROXY_HEAPSIZE""m"
+  #echo $JAVA_HEAP_MAX
+fi
+
+# CLASSPATH initially contains $HDFSPROXY_CONF_DIR
+CLASSPATH="${HDFSPROXY_CONF_DIR}"
+CLASSPATH=${CLASSPATH}:$JAVA_HOME/lib/tools.jar
+
+# for developers, add HdfsProxy classes to CLASSPATH
+if [ -d "$HDFSPROXY_HOME/build/classes" ]; then
+  CLASSPATH=${CLASSPATH}:$HDFSPROXY_HOME/build/classes
+fi
+if [ -d "$HDFSPROXY_HOME/build/webapps" ]; then
+  CLASSPATH=${CLASSPATH}:$HDFSPROXY_HOME/build
+fi
+if [ -d "$HDFSPROXY_HOME/build/test/classes" ]; then
+  CLASSPATH=${CLASSPATH}:$HDFSPROXY_HOME/build/test/classes
+fi
+
+# so that filenames w/ spaces are handled correctly in loops below
+IFS=
+
+# for releases, add hdfsproxy jar & webapps to CLASSPATH
+if [ -d "$HDFSPROXY_HOME/webapps" ]; then
+  CLASSPATH=${CLASSPATH}:$HDFSPROXY_HOME
+fi
+for f in $HDFSPROXY_HOME/hdfsproxy-*.jar; do
+  CLASSPATH=${CLASSPATH}:$f;
+done
+
+# add libs to CLASSPATH
+if [ -d "$HDFSPROXY_HOME/lib" ]; then
+	for f in $HDFSPROXY_HOME/lib/*.jar; do
+	  CLASSPATH=${CLASSPATH}:$f;
+	done
+fi
+
+if [ -d "$HDFSPROXY_HOME/../../" ]; then
+	for f in $HDFSPROXY_HOME/../../*.jar; do
+    CLASSPATH=${CLASSPATH}:$f;
+  done
+fi
+
+if [ -d "$HDFSPROXY_HOME/../../lib" ]; then
+	for f in $HDFSPROXY_HOME/../../lib/*.jar; do
+    CLASSPATH=${CLASSPATH}:$f;
+  done
+fi
+
+# add user-specified CLASSPATH last
+if [ "$HDFSPROXY_CLASSPATH" != "" ]; then
+  CLASSPATH=${CLASSPATH}:${HDFSPROXY_CLASSPATH}
+fi
+
+# default log directory & file
+if [ "$HDFSPROXY_LOG_DIR" = "" ]; then
+  HDFSPROXY_LOG_DIR="$HDFSPROXY_HOME/logs"
+fi
+if [ "$HDFSPROXY_LOGFILE" = "" ]; then
+  HDFSPROXY_LOGFILE='hdfsproxy.log'
+fi
+
+# restore ordinary behaviour
+unset IFS
+
+# figure out which class to run
+CLASS='org.apache.hadoop.hdfsproxy.HdfsProxy'
+
+# cygwin path translation
+if $cygwin; then
+  CLASSPATH=`cygpath -p -w "$CLASSPATH"`
+  HDFSPROXY_HOME=`cygpath -d "$HDFSPROXY_HOME"`
+  HDFSPROXY_LOG_DIR=`cygpath -d "$HDFSPROXY_LOG_DIR"`
+fi
+
+# cygwin path translation
+if $cygwin; then
+  JAVA_LIBRARY_PATH=`cygpath -p "$JAVA_LIBRARY_PATH"`
+fi
+
+HDFSPROXY_OPTS="$HDFSPROXY_OPTS -Dhdfsproxy.log.dir=$HDFSPROXY_LOG_DIR"
+HDFSPROXY_OPTS="$HDFSPROXY_OPTS -Dhdfsproxy.log.file=$HDFSPROXY_LOGFILE"
+HDFSPROXY_OPTS="$HDFSPROXY_OPTS -Dhdfsproxy.home.dir=$HDFSPROXY_HOME"
+HDFSPROXY_OPTS="$HDFSPROXY_OPTS -Dhdfsproxy.id.str=$HDFSPROXY_IDENT_STRING"
+HDFSPROXY_OPTS="$HDFSPROXY_OPTS -Dhdfsproxy.root.logger=${HDFSPROXY_ROOT_LOGGER:-INFO,console}"
+if [ "x$JAVA_LIBRARY_PATH" != "x" ]; then
+  HDFSPROXY_OPTS="$HDFSPROXY_OPTS -Djava.library.path=$JAVA_LIBRARY_PATH"
+fi  
+
+# run it
+exec "$JAVA" $JAVA_HEAP_MAX $HDFSPROXY_OPTS -classpath "$CLASSPATH" $CLASS "$@"

src/contrib/hdfsproxy/bin/hdfsproxy-config.sh

@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# included in all the hadoop scripts with source command
+# should not be executable directly
+# also should not be passed any arguments, since we need original $*
+
+# resolve links - $0 may be a softlink
+
+this="$0"
+while [ -h "$this" ]; do
+  ls=`ls -ld "$this"`
+  link=`expr "$ls" : '.*-> \(.*\)$'`
+  if expr "$link" : '.*/.*' > /dev/null; then
+    this="$link"
+  else
+    this=`dirname "$this"`/"$link"
+  fi
+done
+
+# convert relative path to absolute path
+bin=`dirname "$this"`
+script=`basename "$this"`
+bin=`cd "$bin"; pwd`
+this="$bin/$script"
+
+# the root of the HdfsProxy installation
+export HDFSPROXY_HOME=`dirname "$this"`/..
+
+#check to see if the conf dir is given as an optional argument
+if [ $# -gt 1 ]
+then
+    if [ "--config" = "$1" ]
+	  then
+	      shift
+	      confdir=$1
+	      shift
+	      HDFSPROXY_CONF_DIR=$confdir
+    fi
+fi
+ 
+# Allow alternate conf dir location.
+HDFSPROXY_CONF_DIR="${HDFSPROXY_CONF_DIR:-$HDFSPROXY_HOME/conf}"
+
+#check to see it is specified whether to use the slaves file
+if [ $# -gt 1 ]
+then
+    if [ "--hosts" = "$1" ]
+    then
+        shift
+        slavesfile=$1
+        shift
+        export HDFSPROXY_SLAVES="${HDFSPROXY_CONF_DIR}/$slavesfile"
+    fi
+fi

src/contrib/hdfsproxy/bin/hdfsproxy-daemon.sh

@@ -0,0 +1,141 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+# Runs a HdfsProxy as a daemon.
+#
+# Environment Variables
+#
+#   HDFSPROXY_CONF_DIR  Alternate conf dir. Default is ${HDFSPROXY_HOME}/conf.
+#   HDFSPROXY_LOG_DIR   Where log files are stored.  PWD by default.
+#   HDFSPROXY_MASTER    host:path where hdfsproxy code should be rsync'd from
+#   HDFSPROXY_PID_DIR   The pid files are stored. /tmp by default.
+#   HDFSPROXY_IDENT_STRING   A string representing this instance of hdfsproxy. $USER by default
+#   HDFSPROXY_NICENESS The scheduling priority for daemons. Defaults to 0.
+##
+
+usage="Usage: hdfsproxy-daemon.sh [--config <conf-dir>] [--hosts hostlistfile] (start|stop) "
+
+# if no args specified, show usage
+if [ $# -le 1 ]; then
+  echo $usage
+  exit 1
+fi
+
+bin=`dirname "$0"`
+bin=`cd "$bin"; pwd`
+
+. "$bin"/hdfsproxy-config.sh
+
+# get arguments
+startStop=$1
+shift
+
+hdfsproxy_rotate_log ()
+{
+    log=$1;
+    num=5;
+    if [ -n "$2" ]; then
+	num=$2
+    fi
+    if [ -f "$log" ]; then # rotate logs
+	while [ $num -gt 1 ]; do
+	    prev=`expr $num - 1`
+	    [ -f "$log.$prev" ] && mv "$log.$prev" "$log.$num"
+	    num=$prev
+	done
+	mv "$log" "$log.$num";
+    fi
+}
+
+if [ -f "${HDFSPROXY_CONF_DIR}/hdfsproxy-env.sh" ]; then
+  . "${HDFSPROXY_CONF_DIR}/hdfsproxy-env.sh"
+fi
+
+# get log directory
+if [ "$HDFSPROXY_LOG_DIR" = "" ]; then
+  export HDFSPROXY_LOG_DIR="$HDFSPROXY_HOME/logs"
+fi
+mkdir -p "$HDFSPROXY_LOG_DIR"
+
+if [ "$HDFSPROXY_PID_DIR" = "" ]; then
+  HDFSPROXY_PID_DIR=/tmp
+fi
+
+if [ "$HDFSPROXY_IDENT_STRING" = "" ]; then
+  export HDFSPROXY_IDENT_STRING="$USER"
+fi
+
+# some variables
+export HDFSPROXY_LOGFILE=hdfsproxy-$HDFSPROXY_IDENT_STRING-$HOSTNAME.log
+export HDFSPROXY_ROOT_LOGGER="INFO,DRFA"
+log=$HDFSPROXY_LOG_DIR/hdfsproxy-$HDFSPROXY_IDENT_STRING-$HOSTNAME.out
+pid=$HDFSPROXY_PID_DIR/hdfsproxy-$HDFSPROXY_IDENT_STRING.pid
+
+# Set default scheduling priority
+if [ "$HDFSPROXY_NICENESS" = "" ]; then
+    export HDFSPROXY_NICENESS=0
+fi
+
+case $startStop in
+
+  (start)
+
+    mkdir -p "$HDFSPROXY_PID_DIR"
+
+    if [ -f $pid ]; then
+      if kill -0 `cat $pid` > /dev/null 2>&1; then
+        echo hdfsproxy running as process `cat $pid`.  Stop it first.
+        exit 1
+      fi
+    fi
+
+    if [ "$HDFSPROXY_MASTER" != "" ]; then
+      echo rsync from $HDFSPROXY_MASTER
+      rsync -a -e ssh --delete --exclude=.svn --exclude='logs/*' --exclude='contrib/hod/logs/*' $HDFSPROXY_MASTER/ "$HDFSPROXY_HOME"
+    fi
+
+    hdfsproxy_rotate_log $log
+    echo starting hdfsproxy, logging to $log
+    cd "$HDFSPROXY_HOME"
+    nohup nice -n $HDFSPROXY_NICENESS "$HDFSPROXY_HOME"/bin/hdfsproxy --config $HDFSPROXY_CONF_DIR "$@" > "$log" 2>&1 < /dev/null &
+    echo $! > $pid
+    sleep 1; head "$log"
+    ;;
+          
+  (stop)
+
+    if [ -f $pid ]; then
+      if kill -0 `cat $pid` > /dev/null 2>&1; then
+        echo stopping hdfsproxy
+        kill `cat $pid`
+      else
+        echo no hdfsproxy to stop
+      fi
+    else
+      echo no hdfsproxy to stop
+    fi
+    ;;
+
+  (*)
+    echo $usage
+    exit 1
+    ;;
+
+esac
+
+

src/contrib/hdfsproxy/bin/hdfsproxy-daemons.sh

@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+# Run a HdfsProxy command on all slave hosts.
+
+usage="Usage: hdfsproxy-daemons.sh [--config confdir] [--hosts hostlistfile] [start|stop] "
+
+# if no args specified, show usage
+if [ $# -le 1 ]; then
+  echo $usage
+  exit 1
+fi
+
+bin=`dirname "$0"`
+bin=`cd "$bin"; pwd`
+
+. $bin/hdfsproxy-config.sh
+
+exec "$bin/hdfsproxy-slaves.sh" --config $HDFSPROXY_CONF_DIR cd "$HDFSPROXY_HOME" \; "$bin/hdfsproxy-daemon.sh" --config $HDFSPROXY_CONF_DIR "$@"

src/contrib/hdfsproxy/bin/hdfsproxy-slaves.sh

@@ -0,0 +1,68 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+# Run a shell command on all slave hosts.
+#
+# Environment Variables
+#
+#   HDFSPROXY_SLAVES    File naming remote hosts.
+#     Default is ${HDFSPROXY_CONF_DIR}/hdfsproxy-hosts.
+#   HDFSPROXY_CONF_DIR  Alternate conf dir. Default is ${HDFSPROXY_HOME}/conf.
+#   HDFSPROXY_SLAVE_SLEEP Seconds to sleep between spawning remote commands.
+#   HDFSPROXY_SSH_OPTS Options passed to ssh when running remote commands.
+##
+
+usage="Usage: hdfsproxy-slaves.sh [--config confdir] command..."
+
+# if no args specified, show usage
+if [ $# -le 0 ]; then
+  echo $usage
+  exit 1
+fi
+
+bin=`dirname "$0"`
+bin=`cd "$bin"; pwd`
+
+. "$bin"/hdfsproxy-config.sh
+
+# If the slaves file is specified in the command line,
+# then it takes precedence over the definition in 
+# hdfsproxy-env.sh. Save it here.
+HOSTLIST=$HDFSPROXY_SLAVES
+
+if [ -f "${HDFSPROXY_CONF_DIR}/hdfsproxy-env.sh" ]; then
+  . "${HDFSPROXY_CONF_DIR}/hdfsproxy-env.sh"
+fi
+
+if [ "$HOSTLIST" = "" ]; then
+  if [ "$HDFSPROXY_SLAVES" = "" ]; then
+    export HOSTLIST="${HDFSPROXY_CONF_DIR}/hdfsproxy-hosts"
+  else
+    export HOSTLIST="${HDFSPROXY_SLAVES}"
+  fi
+fi
+
+for slave in `cat "$HOSTLIST"`; do
+ ssh $HDFSPROXY_SSH_OPTS $slave $"${@// /\\ }" \
+   2>&1 | sed "s/^/$slave: /" &
+ if [ "$HDFSPROXY_SLAVE_SLEEP" != "" ]; then
+   sleep $HDFSPROXY_SLAVE_SLEEP
+ fi
+done
+
+wait

src/contrib/hdfsproxy/bin/hdfsproxy-tomcat-server.sh

@@ -0,0 +1,92 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+# Runs a HdfsProxy as a daemon.
+#
+# Environment Variables
+#
+#   HDFSPROXY_CONF_DIR  Alternate conf dir. Default is ${HDFSPROXY_HOME}/conf.
+#   HDFSPROXY_MASTER    host:path where hdfsproxy code should be rsync'd from
+#   HDFSPROXY_PID_DIR   The pid files are stored. /tmp by default.
+#   HDFSPROXY_IDENT_STRING   A string representing this instance of hdfsproxy. $USER by default
+#   HDFSPROXY_NICENESS The scheduling priority for daemons. Defaults to 0.
+#		TOMCAT_HOME_DIR tomcat home directory.
+##
+
+usage="Usage: hdfsproxy-tomcat-server.sh [--config <conf-dir>] [--hosts hostlistfile] (start|stop) "
+
+# if no args specified, show usage
+if [ $# -le 1 ]; then
+  echo $usage
+  exit 1
+fi
+
+bin=`dirname "$0"`
+bin=`cd "$bin"; pwd`
+
+. "$bin"/hdfsproxy-config.sh
+
+# get arguments
+startStop=$1
+shift
+
+
+if [ -f "${HDFSPROXY_CONF_DIR}/hdfsproxy-env.sh" ]; then
+  . "${HDFSPROXY_CONF_DIR}/hdfsproxy-env.sh"
+fi
+
+
+if [ "$HDFSPROXY_IDENT_STRING" = "" ]; then
+  export HDFSPROXY_IDENT_STRING="$USER"
+fi
+
+
+# Set default scheduling priority
+if [ "$HDFSPROXY_NICENESS" = "" ]; then
+    export HDFSPROXY_NICENESS=0
+fi
+
+case $startStop in
+
+  (start)
+    if [ "$HDFSPROXY_MASTER" != "" ]; then
+      echo rsync from $HDFSPROXY_MASTER
+      rsync -a -e ssh --delete --exclude=.svn --exclude='logs/*' --exclude='contrib/hod/logs/*' $HDFSPROXY_MASTER/ "$HDFSPROXY_HOME"
+    fi
+
+    echo starting hdfsproxy tomcat server
+    cd "$HDFSPROXY_HOME"
+    nohup nice -n $HDFSPROXY_NICENESS "$TOMCAT_HOME_DIR"/bin/startup.sh >& /dev/null &
+    sleep 1
+    ;;
+          
+  (stop)
+
+    echo stopping hdfsproxy tomcat server
+    cd "$HDFSPROXY_HOME"
+    nohup nice -n $HDFSPROXY_NICENESS "$TOMCAT_HOME_DIR"/bin/shutdown.sh >& /dev/null &
+    ;;
+
+  (*)
+    echo $usage
+    exit 1
+    ;;
+
+esac
+
+

src/contrib/hdfsproxy/bin/hdfsproxy-tomcat-servers.sh

@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+# Run a HdfsProxy command on all slave hosts.
+
+usage="Usage: hdfsproxy-tomcat-servers.sh [--config confdir] [--hosts hostlistfile] [start|stop] "
+
+# if no args specified, show usage
+if [ $# -le 1 ]; then
+  echo $usage
+  exit 1
+fi
+
+bin=`dirname "$0"`
+bin=`cd "$bin"; pwd`
+
+. $bin/hdfsproxy-config.sh
+
+exec "$bin/hdfsproxy-tomcat-slaves.sh" --config $HDFSPROXY_CONF_DIR cd "$HDFSPROXY_HOME" \; "$bin/hdfsproxy-tomcat-server.sh" --config $HDFSPROXY_CONF_DIR "$@"

src/contrib/hdfsproxy/bin/hdfsproxy-tomcat-slaves.sh

@@ -0,0 +1,68 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+# Run a shell command on all slave hosts.
+#
+# Environment Variables
+#
+#   HDFSPROXY_SLAVES    File naming remote hosts.
+#     Default is ${HDFSPROXY_CONF_DIR}/hdfsproxy-hosts.
+#   HDFSPROXY_CONF_DIR  Alternate conf dir. Default is ${HDFSPROXY_HOME}/conf.
+#   HDFSPROXY_SLAVE_SLEEP Seconds to sleep between spawning remote commands.
+#   HDFSPROXY_SSH_OPTS Options passed to ssh when running remote commands.
+##
+
+usage="Usage: hdfsproxy-tomcat-slaves.sh [--config confdir] command..."
+
+# if no args specified, show usage
+if [ $# -le 0 ]; then
+  echo $usage
+  exit 1
+fi
+
+bin=`dirname "$0"`
+bin=`cd "$bin"; pwd`
+
+. "$bin"/hdfsproxy-config.sh
+
+# If the slaves file is specified in the command line,
+# then it takes precedence over the definition in 
+# hdfsproxy-env.sh. Save it here.
+HOSTLIST=$HDFSPROXY_SLAVES
+
+if [ -f "${HDFSPROXY_CONF_DIR}/hdfsproxy-env.sh" ]; then
+  . "${HDFSPROXY_CONF_DIR}/hdfsproxy-env.sh"
+fi
+
+if [ "$HOSTLIST" = "" ]; then
+  if [ "$HDFSPROXY_SLAVES" = "" ]; then
+    export HOSTLIST="${HDFSPROXY_CONF_DIR}/hdfsproxy-hosts"
+  else
+    export HOSTLIST="${HDFSPROXY_SLAVES}"
+  fi
+fi
+
+for slave in `cat "$HOSTLIST"`; do
+ ssh $HDFSPROXY_SSH_OPTS $slave $"${@// /\\ }" \
+   2>&1 | sed "s/^/$slave: /" & 
+ if [ "$HDFSPROXY_SLAVE_SLEEP" != "" ]; then
+   sleep $HDFSPROXY_SLAVE_SLEEP
+ fi
+done
+
+wait

src/contrib/hdfsproxy/bin/proxy-util

@@ -0,0 +1,152 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+# The Proxy command utility script
+#
+# Environment Variables
+#
+#   JAVA_HOME        The java implementation to use.  Overrides JAVA_HOME.
+#
+#   HDFSPROXY_CLASSPATH Extra Java CLASSPATH entries.
+#
+#   HDFSPROXY_HEAPSIZE  The maximum amount of heap to use, in MB. 
+#                    Default is 1000.
+#
+#   HDFSPROXY_OPTS      Extra Java runtime options.
+#   
+#   HDFSPROXY_NAMENODE_OPTS       These options are added to HDFSPROXY_OPTS 
+#   HDFSPROXY_CLIENT_OPTS         when the respective command is run.
+#   HDFSPROXY_{COMMAND}_OPTS etc  HDFSPROXY_JT_OPTS applies to JobTracker 
+#                              for e.g.  HDFSPROXY_CLIENT_OPTS applies to 
+#                              more than one command (fs, dfs, fsck, 
+#                              dfsadmin etc)  
+#
+#   HDFSPROXY_CONF_DIR  Alternate conf dir. Default is ${HDFSPROXY_HOME}/conf.
+#
+#   HDFSPROXY_ROOT_LOGGER The root appender. Default is INFO,console
+#
+
+bin=`dirname "$0"`
+bin=`cd "$bin"; pwd`
+
+. "$bin"/hdfsproxy-config.sh
+
+cygwin=false
+case "`uname`" in
+CYGWIN*) cygwin=true;;
+esac
+
+if [ -f "${HDFSPROXY_CONF_DIR}/hdfsproxy-env.sh" ]; then
+  . "${HDFSPROXY_CONF_DIR}/hdfsproxy-env.sh"
+fi
+
+# some Java parameters
+if [ "$JAVA_HOME" != "" ]; then
+  #echo "run java in $JAVA_HOME"
+  JAVA_HOME=$JAVA_HOME
+fi
+  
+if [ "$JAVA_HOME" = "" ]; then
+  echo "Error: JAVA_HOME is not set."
+  exit 1
+fi
+
+JAVA=$JAVA_HOME/bin/java
+JAVA_HEAP_MAX=-Xmx1000m 
+
+# check envvars which might override default args
+if [ "$HDFSPROXY_HEAPSIZE" != "" ]; then
+  #echo "run with heapsize $HDFSPROXY_HEAPSIZE"
+  JAVA_HEAP_MAX="-Xmx""$HDFSPROXY_HEAPSIZE""m"
+  #echo $JAVA_HEAP_MAX
+fi
+
+# CLASSPATH initially contains $HDFSPROXY_CONF_DIR
+CLASSPATH="${HADOOP_CONF_DIR}"
+CLASSPATH="${CLASSPATH}:${HDFSPROXY_CONF_DIR}"
+CLASSPATH=${CLASSPATH}:$JAVA_HOME/lib/tools.jar
+
+# for developers, add HdfsProxy classes to CLASSPATH
+if [ -d "$HDFSPROXY_HOME/build/classes" ]; then
+  CLASSPATH=${CLASSPATH}:$HDFSPROXY_HOME/build/classes
+fi
+if [ -d "$HDFSPROXY_HOME/build/webapps" ]; then
+  CLASSPATH=${CLASSPATH}:$HDFSPROXY_HOME/build
+fi
+if [ -d "$HDFSPROXY_HOME/build/test/classes" ]; then
+  CLASSPATH=${CLASSPATH}:$HDFSPROXY_HOME/build/test/classes
+fi
+
+# so that filenames w/ spaces are handled correctly in loops below
+IFS=
+
+# for releases, add hdfsproxy jar & webapps to CLASSPATH
+if [ -d "$HDFSPROXY_HOME/webapps" ]; then
+  CLASSPATH=${CLASSPATH}:$HDFSPROXY_HOME
+fi
+for f in $HDFSPROXY_HOME/hdfsproxy-*.jar; do
+  CLASSPATH=${CLASSPATH}:$f;
+done
+
+# add libs to CLASSPATH
+for f in $HDFSPROXY_HOME/lib/*.jar; do
+  CLASSPATH=${CLASSPATH}:$f;
+done
+
+# add user-specified CLASSPATH last
+if [ "$HDFSPROXY_CLASSPATH" != "" ]; then
+  CLASSPATH=${CLASSPATH}:${HDFSPROXY_CLASSPATH}
+fi
+
+# default log directory & file
+if [ "$HDFSPROXY_LOG_DIR" = "" ]; then
+  HDFSPROXY_LOG_DIR="$HDFSPROXY_HOME/logs"
+fi
+if [ "$HDFSPROXY_LOGFILE" = "" ]; then
+  HDFSPROXY_LOGFILE='proxy-util.log'
+fi
+
+# restore ordinary behaviour
+unset IFS
+
+# figure out which class to run
+CLASS='org.apache.hadoop.hdfsproxy.ProxyUtil'
+
+# cygwin path translation
+if $cygwin; then
+  CLASSPATH=`cygpath -p -w "$CLASSPATH"`
+  HDFSPROXY_HOME=`cygpath -d "$HDFSPROXY_HOME"`
+  HDFSPROXY_LOG_DIR=`cygpath -d "$HDFSPROXY_LOG_DIR"`
+fi
+
+# cygwin path translation
+if $cygwin; then
+  JAVA_LIBRARY_PATH=`cygpath -p "$JAVA_LIBRARY_PATH"`
+fi
+
+HDFSPROXY_OPTS="$HDFSPROXY_OPTS -Dhdfsproxy.log.dir=$HDFSPROXY_LOG_DIR"
+HDFSPROXY_OPTS="$HDFSPROXY_OPTS -Dhdfsproxy.log.file=$HDFSPROXY_LOGFILE"
+HDFSPROXY_OPTS="$HDFSPROXY_OPTS -Dhdfsproxy.home.dir=$HDFSPROXY_HOME"
+HDFSPROXY_OPTS="$HDFSPROXY_OPTS -Dhdfsproxy.id.str=$HDFSPROXY_IDENT_STRING"
+HDFSPROXY_OPTS="$HDFSPROXY_OPTS -Dhdfsproxy.root.logger=${HDFSPROXY_ROOT_LOGGER:-INFO,console}"
+if [ "x$JAVA_LIBRARY_PATH" != "x" ]; then
+  HDFSPROXY_OPTS="$HDFSPROXY_OPTS -Djava.library.path=$JAVA_LIBRARY_PATH"
+fi  
+
+# run it
+exec "$JAVA" $JAVA_HEAP_MAX $HDFSPROXY_OPTS -classpath "$CLASSPATH" $CLASS "$@"

src/contrib/hdfsproxy/bin/start-hdfsproxy-tomcat.sh

@@ -0,0 +1,36 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+# Start hdfsproxy tomcat servers.
+# Run this on master node.
+
+usage="Usage: start-hdfsproxy-tomcat.sh"
+
+bin=`dirname "$0"`
+bin=`cd "$bin"; pwd`
+
+. "$bin"/hdfsproxy-config.sh
+
+# get arguments
+if [ $# -ge 1 ]; then
+  echo $usage
+  exit 1
+fi
+
+# start hdfsproxy tomcat servers
+"$bin"/hdfsproxy-tomcat-servers.sh --config $HDFSPROXY_CONF_DIR --hosts hdfsproxy-hosts start

src/contrib/hdfsproxy/bin/start-hdfsproxy.sh

@@ -0,0 +1,37 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+# Start hdfsproxy daemons.
+# Run this on master node.
+
+usage="Usage: start-hdfsproxy.sh"
+
+bin=`dirname "$0"`
+bin=`cd "$bin"; pwd`
+
+. "$bin"/hdfsproxy-config.sh
+
+# get arguments
+if [ $# -ge 1 ]; then
+  echo $usage
+  exit 1
+fi
+
+# start hdfsproxy daemons
+# "$bin"/hdfsproxy-daemon.sh --config $HDFSPROXY_CONF_DIR start
+"$bin"/hdfsproxy-daemons.sh --config $HDFSPROXY_CONF_DIR --hosts hdfsproxy-hosts start

src/contrib/hdfsproxy/bin/stop-hdfsproxy-tomcat.sh

@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+# Stop hdfsproxy tomcat servers.  Run this on master node.
+
+bin=`dirname "$0"`
+bin=`cd "$bin"; pwd`
+
+. "$bin"/hdfsproxy-config.sh
+
+# "$bin"/hdfsproxy-daemon.sh --config $HDFSPROXY_CONF_DIR stop
+"$bin"/hdfsproxy-tomcat-servers.sh --config $HDFSPROXY_CONF_DIR --hosts hdfsproxy-hosts stop
+

src/contrib/hdfsproxy/bin/stop-hdfsproxy.sh

@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+# Stop hdfsproxy daemons.  Run this on master node.
+
+bin=`dirname "$0"`
+bin=`cd "$bin"; pwd`
+
+. "$bin"/hdfsproxy-config.sh
+
+# "$bin"/hdfsproxy-daemon.sh --config $HDFSPROXY_CONF_DIR stop
+"$bin"/hdfsproxy-daemons.sh --config $HDFSPROXY_CONF_DIR --hosts hdfsproxy-hosts stop
+

src/contrib/hdfsproxy/build.xml

@@ -0,0 +1,458 @@
+<?xml version="1.0" ?>
+
+<!--
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+-->
+
+<project name="hdfsproxy" default="jar" xmlns:ivy="antlib:org.apache.ivy.ant">
+	<property name="hdfsproxyVersion" value="2.0"/>
+	<property name="final.name" value="${ant.project.name}-${hdfsproxyVersion}"/>
+	<property name="javac.debug" value="on"/>
+	<property name="javac.optimize" value="on"/>
+	<import file="../build-contrib.xml"/>
+	
+	<property name="bin.dir" value="${basedir}/bin"/>
+	<property name="lib.dir" value="${basedir}/lib"/>
+	<property name="hadoop.jars.dir" value="${basedir}/hadoopjars"/>
+	
+	<property name="docs.dir" value="${basedir}/docs"/>
+	<property name="test.build.dir" value="${build.dir}/test"/>
+	<property name="test.build.classes" value="${test.build.dir}/classes"/>	
+	<property name="src.test.resources" value="${basedir}/src/test/resources"/>
+	<property name="ssl.keystore.proxy" value="${src.test.resources}/ssl-keys/proxy.keystore"/>
+	<property name="ssl.keystore.client" value="${src.test.resources}/ssl-keys/client.keystore"/>
+	<property name="ssl.client.cert" value="${src.test.resources}/ssl-keys/test.crt"/>
+	<property name="proxy.conf.test" value="${src.test.resources}/proxy-config"/>
+	<property name="tomcat.conf.test" value="${src.test.resources}/tomcat-config"/>
+	<property name="target.dir" value="${build.dir}/target"/>
+	<property name="logs.dir" value="${target.dir}/logs"/>
+	<property name="reports.dir" value="${target.dir}/reports"/>
+	<property name="tomcatconfig.dir" value="${target.dir}/tomcat-config"/>
+	<property name="tomcat.container.id" value="tomcat5x"/>
+	<property name="cargo.logging" value="high"/>
+	<property name="cactus.formatter.type" value="xml"/>
+	<property name="cactus.warfile.name" value="test"/>	
+  
+	<available file="${hadoop.root}/build/classes" type="dir" property="test.available"/>
+	<property environment="env"/>
+	<!-- check if environment has been set -->
+  <condition property="proxy.conf.dir" value="${env.HDFSPROXY_CONF_DIR}" else="${basedir}/conf">
+    <and>
+        <isset property="env.HDFSPROXY_CONF_DIR"/>
+        <available file="${env.HDFSPROXY_CONF_DIR}/hdfsproxy-default.xml"/>
+    </and>
+  </condition> 
+  
+  <condition property="startCactus">
+  	<and>
+	    <or>
+	    	<equals arg1="${testcase}" arg2="TestProxyFilter" />
+				<equals arg1="${testcase}" arg2="TestProxyUtil" />
+				<equals arg1="${testcase}" arg2="TestProxyForwardServlet" />
+				<not>
+					<isset property="testcase"/>
+				</not>
+	    </or>
+    	<isset property="test.available"/>
+    </and>
+  </condition>
+  
+  <condition property="useClover">
+  	<and>
+    	<isset property="clover.home"/>
+    	<available file="${clover.home}/lib/clover.jar"/>
+    </and>
+  </condition>
+
+	<property name="ivy.settings.file" location="${hadoop.root}/ivy/ivysettings.xml"/>
+  
+  <target name="ivy-init" depends="ivy-init-antlib">
+    		<ivy:settings id="${ant.project.name}.ivy.settings"/>
+	</target>
+	
+	<!-- Define the Cactus tasks -->	
+	<target name="load-tasks" depends="ivy-retrieve-common">
+		<taskdef resource="cactus.tasks"
+			 classpathref="cactus.classpath">
+		</taskdef>		
+	</target>
+	
+
+	<target name="jar" depends="compile" description="Create jar">
+		<echo>
+            Building the .jar files.
+        </echo>
+		<jar jarfile="${build.dir}/${final.name}.jar" basedir="${build.classes}" includes="org/apache/hadoop/hdfsproxy/**/*.class" >
+                        <manifest>
+                            <section name="org/apache/hadoop/hdfsproxy">
+                                <attribute name="Implementation-Title" value="HdfsProxy"/>
+                                <attribute name="Implementation-Version" value="${hdfsproxyVersion}"/>
+                                <attribute name="Implementation-Vendor" value="Apache"/>
+                            </section>
+                        </manifest>
+		</jar>
+	</target>
+	
+	
+	<!-- ================================================================== -->
+	<!-- Make war file                                              -->
+	<!-- ================================================================== -->
+	
+	<target name="war" depends="compile" description="Create war">
+		<echo>
+			Building the .war file
+		</echo>
+	  <war destfile="${build.dir}/${final.name}.war" webxml="${basedir}/conf/tomcat-web.xml">
+	    <lib dir="${common.ivy.lib.dir}">
+	      <include name="commons-logging-${commons-logging.version}.jar"/>
+        <include name="junit-${junit.version}.jar"/>
+        <include name="log4j-${log4j.version}.jar"/>
+        <include name="slf4j-api-${slf4j-api.version}.jar"/>
+        <include name="slf4j-log4j12-${slf4j-log4j12.version}.jar"/>
+        <include name="xmlenc-${xmlenc.version}.jar"/>
+        <include name="core-${core.version}.jar"/> 
+	    </lib>
+	    <classes dir="${proxy.conf.dir}" excludes="**/*.example **/*.template **/*.sh hadoop-site.xml"/>
+	    <classes dir="${build.classes}"/>
+	    <classes dir="${hadoop.root}/build/classes"/>
+	  </war>
+	</target>
+	
+	<target name="forward" depends="compile" description="Create forward war">
+		<echo>
+			Building the forward war file
+		</echo>
+	  <war destfile="${build.dir}/${final.name}-forward.war" webxml="${basedir}/conf/tomcat-forward-web.xml">
+	    <lib dir="${common.ivy.lib.dir}">
+	      <include name="commons-logging-${commons-logging.version}.jar"/>
+        <include name="junit-${junit.version}.jar"/>
+        <include name="log4j-${log4j.version}.jar"/>
+        <include name="slf4j-api-${slf4j-api.version}.jar"/>
+        <include name="slf4j-log4j12-${slf4j-log4j12.version}.jar"/>
+        <include name="xmlenc-${xmlenc.version}.jar"/>
+        <include name="core-${core.version}.jar"/> 
+	    </lib>
+	    <classes dir="${proxy.conf.dir}" excludes="**/*.example **/*.template **/*.sh hadoop-site.xml"/>
+	    <classes dir="${build.classes}"/>
+	    <classes dir="${hadoop.root}/build/classes"/>
+	  </war>
+	</target>	
+	
+	<target name="cactifywar" depends="war,load-tasks,cactifywar-pure,cactifywar-clover" description="To include clover coverage test use -Dclover.home ..."/>
+	
+	<target name="cactifywar-pure" depends="war,load-tasks" unless="useClover">
+		<mkdir dir="${target.dir}" />
+		<echo> no clover found ...</echo>
+    <cactifywar srcfile="${build.dir}/${final.name}.war"
+        destfile="${target.dir}/${cactus.warfile.name}.war"
+        mergewebxml="${src.test.resources}/cactus-web.xml">
+      <servletredirector/>
+      <servletredirector name="ServletRedirectorSecure"
+          mapping="/ServletRedirectorSecure" roles="test"/>
+      <filterredirector mapping="/test/filterRedirector.jsp"/>
+      <classes dir="${proxy.conf.test}" excludes="**/*.template **/*.sh"/>
+      <classes dir="${test.build.dir}"/>
+    </cactifywar>    	
+	</target>
+
+	<target name="cactifywar-clover" depends="war,load-tasks" if="useClover">
+		<mkdir dir="${target.dir}" />
+		<echo> Including clover.jar in the war file ...</echo>
+    <cactifywar srcfile="${build.dir}/${final.name}.war"
+        destfile="${target.dir}/${cactus.warfile.name}.war"
+        mergewebxml="${src.test.resources}/cactus-web.xml">
+      <servletredirector/>
+      <servletredirector name="ServletRedirectorSecure"
+          mapping="/ServletRedirectorSecure" roles="test"/>
+      <filterredirector mapping="/test/filterRedirector.jsp"/>
+      <classes dir="${proxy.conf.test}" excludes="**/*.template **/*.sh"/>
+      <classes dir="${test.build.dir}"/>
+      <lib dir="${clover.home}/lib">
+      	<include name="clover.jar"/> 
+      </lib>
+    </cactifywar>    	
+	</target>
+	
+	<target name="test" depends="compile,compile-test,test-junit,test-cactus" description="Automated Test Framework" if="test.available"/>
+	
+	<target name="test-junit" depends="compile,compile-test" if="test.available">
+		<junit fork="yes" printsummary="yes" errorProperty="tests.failed" failureProperty="tests.failed">
+        <classpath refid="test.classpath"/>
+        <sysproperty key="test.build.data" value="${build.test}/data"/>
+	      <sysproperty key="build.test" value="${build.test}"/>
+	      <sysproperty key="user.dir" value="${build.test}/data"/>
+	      <sysproperty key="fs.default.name" value="${fs.default.name}"/>
+	      <sysproperty key="hadoop.test.localoutputfile" value="${hadoop.test.localoutputfile}"/>
+	      <sysproperty key="hadoop.log.dir" value="${hadoop.log.dir}"/> 
+        <sysproperty key="test.src.dir" value="${test.src.dir}"/>
+        <sysproperty key="javax.net.ssl.trustStore" value="${ssl.keystore.proxy}"/>
+				<sysproperty key="javax.net.ssl.trustStorePassword" value="changeme"/>
+				<sysproperty key="javax.net.ssl.keyStore.proxy" value="${ssl.keystore.proxy}"/>
+				<sysproperty key="javax.net.ssl.keyStore" value="${ssl.keystore.client}"/>
+				<sysproperty key="javax.net.ssl.keyStorePassword" value="changeme"/>
+				<sysproperty key="javax.net.ssl.keyPassword" value="changeme"/>			
+				<sysproperty key="javax.net.ssl.clientCert" value="${ssl.client.cert}"/>
+        <formatter type="xml" />
+        <batchtest todir="${test.build.dir}" unless="testcase">
+           <fileset dir="${src.test}">
+             <include name="**/TestHdfsProxy.java"/>
+             <include name="**/TestProxyUgiManager.java"/>
+           </fileset>
+        </batchtest>
+        <batchtest todir="${test.build.dir}" if="testcase">
+            <fileset dir="${src.test}">
+            	<include name="**/${testcase}.java"/>
+            	<exclude name="**/TestProxyFilter.java"/>
+            	<exclude name="**/TestProxyUtil.java"/>
+            	<exclude name="**/TestProxyForwardServlet.java"/>
+            </fileset>
+         </batchtest>
+    </junit>    	
+    <fail if="tests.failed">Tests failed!</fail>
+	</target>
+
+
+	<target name="test-cactus" depends="compile,compile-test,cactifywar" if="startCactus">
+		<exec executable="${env.JAVA_HOME}/bin/java" outputproperty="cargo.servlet.admin.port">
+	    <arg line="-cp ${build.test} org.apache.hadoop.hdfsproxy.FindFreePort -random"/>
+		</exec>
+		<exec executable="${env.JAVA_HOME}/bin/java" outputproperty="cargo.servlet.http.port">
+	    <arg line="-cp ${build.test} org.apache.hadoop.hdfsproxy.FindFreePort ${cargo.servlet.admin.port}"/>
+		</exec>
+		<exec executable="${env.JAVA_HOME}/bin/java" outputproperty="cargo.servlet.https.port">
+	    <arg line="-cp ${build.test} org.apache.hadoop.hdfsproxy.FindFreePort ${cargo.servlet.http.port}"/>
+		</exec>
+		
+		<echo> Free Ports: startup-${cargo.servlet.admin.port} / http-${cargo.servlet.http.port} / https-${cargo.servlet.https.port}</echo>
+	  <echo>Please take a deep breath while Cargo gets the Tomcat for running the servlet tests...</echo>
+	  
+	  <mkdir dir="${tomcatconfig.dir}" />
+	  <mkdir dir="${tomcatconfig.dir}/conf" />
+	  <mkdir dir="${tomcatconfig.dir}/webapps" />
+	  <mkdir dir="${tomcatconfig.dir}/temp" />
+	  <mkdir dir="${logs.dir}" />
+	  <mkdir dir="${reports.dir}" />	  
+	  <copy file="${tomcat.conf.test}/server.xml" tofile="${tomcatconfig.dir}/conf/server.xml" overwrite="true">
+		  <filterset>
+		    <filter token="ADMIN.PORT" value="${cargo.servlet.admin.port}"/>
+		    <filter token="HTTP.PORT" value="${cargo.servlet.http.port}"/>
+		    <filter token="HTTPS.PORT" value="${cargo.servlet.https.port}"/>
+		  </filterset>
+		</copy>		
+		<copy file="${tomcat.conf.test}/web.xml" tofile="${tomcatconfig.dir}/conf/web.xml"/>
+		<copy file="${tomcat.conf.test}/tomcat-users.xml" tofile="${tomcatconfig.dir}/conf/tomcat-users.xml"/>
+	
+		<cactus warfile="${target.dir}/${cactus.warfile.name}.war" fork="yes" haltonfailure="no" printsummary="yes" failureproperty="tests.failed">
+			<classpath>
+				<path refid="cactus.classpath"/>
+				<pathelement location="${build.classes}"/>
+				<pathelement location="${proxy.conf.dir}"/>
+				<pathelement location="${src.test.resources}"/>
+				<pathelement location="${src.test.resources}/proxy-config"/>
+			</classpath>			
+			<containerset>
+				<cargo containerId="${tomcat.container.id}" timeout="30000" output="${logs.dir}/output.log" log="${logs.dir}/cargo.log">
+				 <zipUrlInstaller
+            installUrl="http://apache.osuosl.org/tomcat/tomcat-6/v6.0.18/bin/apache-tomcat-6.0.18.zip"
+            installDir="${target.dir}/${tomcat.container.id}"/>
+				  <configuration type="existing" home="${tomcatconfig.dir}">
+						<property name="cargo.servlet.port" value="${cargo.servlet.http.port}"/>
+						<property name="cargo.logging" value="${cargo.logging}"/>
+						<property name="cactus.toDir" value="${build.test}"/>
+						<deployable type="war" file="${target.dir}/${cactus.warfile.name}.war"/>
+					</configuration>
+				</cargo>
+			</containerset>
+			<sysproperty key="test.build.data" value="${build.test}/data"/>
+      <sysproperty key="build.test" value="${build.test}"/>
+      <sysproperty key="build.target" value="${target.dir}"/>
+			<sysproperty key="javax.net.ssl.trustStore" value="${ssl.keystore.proxy}"/>
+			<sysproperty key="javax.net.ssl.trustStorePassword" value="changeme"/>
+			<sysproperty key="javax.net.ssl.keyStore.proxy" value="${ssl.keystore.proxy}"/>
+			<sysproperty key="javax.net.ssl.keyStore" value="${ssl.keystore.client}"/>
+			<sysproperty key="javax.net.ssl.keyStorePassword" value="changeme"/>
+			<sysproperty key="javax.net.ssl.keyPassword" value="changeme"/>			
+			<sysproperty key="javax.net.ssl.clientCert" value="${ssl.client.cert}"/>     
+      <sysproperty key="test.proxy.conf.dir" value="${proxy.conf.test}"/>
+      <sysproperty key="test.proxy.https.port" value="${cargo.servlet.https.port}"/>
+      
+			<formatter type="${cactus.formatter.type}"/>
+			<batchtest todir="${reports.dir}" unless="testcase">
+				<fileset dir="${src.test}">
+					<include name="**/Test*.java"/>
+					<exclude name="**/TestHdfsProxy.java"/>
+					<exclude name="**/TestProxyUgiManager.java"/>
+				</fileset>
+			</batchtest>
+			<batchtest todir="${reports.dir}" if="testcase">
+        <fileset dir="${src.test}">
+        	<include name="**/${testcase}.java"/>
+        	<exclude name="**/TestHdfsProxy.java"/>
+        	<exclude name="**/TestProxyUgiManager.java"/>
+        </fileset>
+      </batchtest>
+		</cactus>		
+    <fail if="tests.failed">Tests failed!</fail>
+	</target>
+	<!-- ====================================================== -->
+	<!-- Macro definitions                                      -->
+	<!-- ====================================================== -->
+	<macrodef name="macro_tar" description="Worker Macro for tar">
+		<attribute name="param.destfile"/>
+		<element name="param.listofitems"/>
+		<sequential>
+			<tar compression="gzip" longfile="gnu"
+          destfile="@{param.destfile}">
+				<param.listofitems/>
+			</tar>
+		</sequential>
+	</macrodef>
+
+	<!-- ================================================================== -->
+	<!-- D I S T R I B U T I O N                                            -->
+	<!-- ================================================================== -->
+	<!--                                                                    -->
+	<!-- ================================================================== -->
+	<target name="local-package" depends="jar,war" description="Package in local build directory">
+		<mkdir dir="${build.dir}/${final.name}"/>
+		<mkdir dir="${build.dir}/${final.name}/logs"/>
+		<copy todir="${build.dir}/${final.name}" includeEmptyDirs="false">
+			<fileset dir="${build.dir}">
+				<include name="*.jar" />
+				<include name="*.war" />
+			</fileset>
+		</copy>
+		<copy todir="${build.dir}/${final.name}/lib" includeEmptyDirs="false">
+			<fileset dir="${common.ivy.lib.dir}">
+        <include name="commons-logging-${commons-logging.version}.jar"/>
+        <include name="commons-logging-api-${commons-logging-api.version}.jar"/>
+        <include name="junit-${junit.version}.jar"/>
+        <include name="log4j-${log4j.version}.jar"/>
+        <include name="slf4j-api-${slf4j-api.version}.jar"/>
+        <include name="slf4j-log4j12-${slf4j-log4j12.version}.jar"/>
+        <include name="xmlenc-${xmlenc.version}.jar"/>
+        <include name="jetty-util-${jetty-util.version}.jar"/>
+        <include name="jetty-${jetty.version}.jar"/>
+        <include name="servlet-api-2.5-${servlet-api-2.5.version}.jar"/>
+        <include name="core-${core.version}.jar"/> 
+        <include name="jsp-${jsp.version}-${jetty.version}.jar"/> 
+        <include name="jsp-api-${jsp.version}-${jetty.version}.jar"/> 
+		       </fileset>
+		</copy>
+
+		<copy todir="${build.dir}/${final.name}/lib" includeEmptyDirs="false">
+                  	<fileset dir="${hadoop.root}/build">
+                          	<include name="*-core.jar"/>
+                          	<include name="*-tools.jar"/>
+			</fileset>
+		</copy>
+
+		<copy todir="${build.dir}/${final.name}/bin">
+			<fileset dir="${bin.dir}"/>
+		</copy>
+	
+
+		<copy todir="${build.dir}/${final.name}/conf">
+			<fileset dir="${proxy.conf.dir}"/>
+		</copy>
+		
+
+		<copy todir="${build.dir}/${final.name}">
+			<fileset dir="${basedir}">
+				<include name="README" />
+				<include name="build.xml" />
+				<include name="*.txt" />
+			</fileset>
+		</copy>
+
+		<copy todir="${build.dir}/${final.name}/src" includeEmptyDirs="true">
+			<fileset dir="${src.dir}" excludes="**/*.template **/docs/build/**/*"/>
+		</copy>
+
+		<chmod perm="ugo+x" type="file" parallel="false">
+			<fileset dir="${build.dir}/${final.name}/bin"/>
+		</chmod>
+
+	</target>
+	<target name="package" depends="local-package" description="Build distribution">
+    <mkdir dir="${dist.dir}/contrib/${name}"/>
+    <copy todir="${dist.dir}/contrib/${name}">
+      <fileset dir="${build.dir}/${final.name}">
+      	<exclude name="**/lib/**" />
+      	<exclude name="**/src/**" />
+      	<exclude name="*.war" />
+      </fileset>
+    </copy>
+    <chmod dir="${dist.dir}/contrib/${name}/bin" perm="a+x" includes="*"/>
+	</target>
+
+	<!-- ================================================================== -->
+	<!-- Make release tarball                                               -->
+	<!-- ================================================================== -->
+	<target name="tar" depends="local-package,war" description="Make release tarball">
+		<macro_tar param.destfile="${build.dir}/${final.name}.tar.gz">
+			<param.listofitems>
+				<tarfileset dir="${build.dir}" mode="664">
+					<exclude name="${final.name}/bin/*" />
+					<include name="${final.name}/**" />
+				</tarfileset>
+				<tarfileset dir="${build.dir}" mode="755">
+					<include name="${final.name}/bin/*" />
+				</tarfileset>
+			</param.listofitems>
+		</macro_tar>
+	</target>
+
+	<target name="binary" depends="local-package,war" description="Make tarball without source and documentation">
+		<macro_tar param.destfile="${build.dir}/${final.name}-bin.tar.gz">
+			<param.listofitems>
+				<tarfileset dir="${build.dir}" mode="664">
+					<exclude name="${final.name}/bin/*" />
+					<exclude name="${final.name}/src/**" />
+					<exclude name="${final.name}/docs/**" />
+					<include name="${final.name}/**" />
+				</tarfileset>
+				<tarfileset dir="${build.dir}" mode="755">
+					<include name="${final.name}/bin/*" />
+				</tarfileset>
+			</param.listofitems>
+		</macro_tar>
+	</target>
+
+
+	 <!-- the unit test classpath -->
+  <path id="test.classpath">
+  	<pathelement location="${proxy.conf.test}" />
+    <pathelement location="${test.build.dir}" />
+    <pathelement location="${hadoop.root}/build/test/classes"/>
+    <pathelement location="${hadoop.root}/build/test/core/classes"/>
+    <pathelement location="${hadoop.root}/build/test/hdfs/classes"/>
+    <pathelement location="${hadoop.root}/build/test/mapred/classes"/>
+    <!--<pathelement location="${hadoop.root}/src/contrib/test"/>-->
+    <pathelement location="${hadoop.root}/conf"/>
+    <pathelement location="${hadoop.root}/build"/>
+    <pathelement location="${hadoop.root}/build/classes"/>
+    <pathelement location="${hadoop.root}/build/tools"/>
+    <pathelement location="${build.examples}"/>
+    <pathelement path="${clover.jar}"/>
+    <path refid="contrib-classpath"/>
+  </path>
+  
+  <path id="cactus.classpath">
+    <path refid="test.classpath"/>
+  </path>
+
+</project>

src/contrib/hdfsproxy/conf/configuration.xsl

@@ -0,0 +1,24 @@
+<?xml version="1.0"?>
+<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
+<xsl:output method="html"/>
+<xsl:template match="configuration">
+<html>
+<body>
+<table border="1">
+<tr>
+ <td>name</td>
+ <td>value</td>
+ <td>description</td>
+</tr>
+<xsl:for-each select="property">
+<tr>
+  <td><a name="{name}"><xsl:value-of select="name"/></a></td>
+  <td><xsl:value-of select="value"/></td>
+  <td><xsl:value-of select="description"/></td>
+</tr>
+</xsl:for-each>
+</table>
+</body>
+</html>
+</xsl:template>
+</xsl:stylesheet>

src/contrib/hdfsproxy/conf/hdfsproxy-default.xml

@@ -0,0 +1,59 @@
+<?xml version="1.0"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+
+<!-- Put hdfsproxy specific properties in this file. -->
+
+<configuration>
+
+<property>
+  <name>hdfsproxy.https.address</name>
+  <value>0.0.0.0:8443</value>
+  <description>the SSL port that hdfsproxy listens on
+  </description>
+</property>
+
+<property>
+  <name>hdfsproxy.hosts</name>
+  <value>hdfsproxy-hosts</value>
+  <description>location of hdfsproxy-hosts file
+  </description>
+</property>
+
+<property>
+  <name>hdfsproxy.dfs.namenode.address</name>
+  <value>localhost:54321</value>
+  <description>namenode address of the HDFS cluster being proxied
+  </description>
+</property>
+
+<property>
+  <name>hdfsproxy.https.server.keystore.resource</name>
+  <value>ssl-server.xml</value>
+  <description>location of the resource from which ssl server keystore
+  information will be extracted
+  </description>
+</property>
+
+<property>
+  <name>hdfsproxy.user.permissions.file.location</name>
+  <value>user-permissions.xml</value>
+  <description>location of the user permissions file
+  </description>
+</property>
+
+<property>
+  <name>hdfsproxy.user.certs.file.location</name>
+  <value>user-certs.xml</value>
+  <description>location of the user certs file
+  </description>
+</property>
+
+<property>
+  <name>hdfsproxy.ugi.cache.ugi.lifetime</name>
+  <value>15</value>
+  <description> The lifetime (in minutes) of a cached ugi
+  </description>
+</property>
+
+</configuration>
+

src/contrib/hdfsproxy/conf/hdfsproxy-env.sh

@@ -0,0 +1,44 @@
+# Set HdfsProxy-specific environment variables here.
+
+# The only required environment variable is JAVA_HOME.  All others are
+# optional.  When running a distributed configuration it is best to
+# set JAVA_HOME in this file, so that it is correctly defined on
+# remote nodes.
+
+# The java implementation to use.  Required.
+# export JAVA_HOME=/usr/lib/j2sdk1.5-sun
+
+# Extra Java CLASSPATH elements.  Optional.
+# export HDFSPROXY_CLASSPATH=
+
+# The maximum amount of heap to use, in MB. Default is 1000.
+# export HDFSPROXY_HEAPSIZE=2000
+
+# Extra Java runtime options.  Empty by default.
+# export HDFSPROXY_OPTS=
+
+# Extra ssh options.  Empty by default.
+# export HDFSPROXY_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HDFSPROXY_CONF_DIR"
+
+# Where log files are stored.  $HDFSPROXY_HOME/logs by default.
+# export HDFSPROXY_LOG_DIR=${HDFSPROXY_HOME}/logs
+
+# File naming remote slave hosts.  $HDFSPROXY_HOME/conf/slaves by default.
+# export HDFSPROXY_SLAVES=${HDFSPROXY_HOME}/conf/slaves
+
+# host:path where hdfsproxy code should be rsync'd from.  Unset by default.
+# export HDFSPROXY_MASTER=master:/home/$USER/src/hdfsproxy
+
+# Seconds to sleep between slave commands.  Unset by default.  This
+# can be useful in large clusters, where, e.g., slave rsyncs can
+# otherwise arrive faster than the master can service them.
+# export HDFSPROXY_SLAVE_SLEEP=0.1
+
+# The directory where pid files are stored. /tmp by default.
+# export HDFSPROXY_PID_DIR=/var/hdfsproxy/pids
+
+# A string representing this instance of hdfsproxy. $USER by default.
+# export HDFSPROXY_IDENT_STRING=$USER
+
+# The scheduling priority for daemon processes.  See 'man nice'.
+# export HDFSPROXY_NICENESS=10

src/contrib/hdfsproxy/conf/hdfsproxy-env.sh.template

@@ -0,0 +1,44 @@
+# Set HdfsProxy-specific environment variables here.
+
+# The only required environment variable is JAVA_HOME.  All others are
+# optional.  When running a distributed configuration it is best to
+# set JAVA_HOME in this file, so that it is correctly defined on
+# remote nodes.
+
+# The java implementation to use.  Required.
+# export JAVA_HOME=/usr/lib/j2sdk1.5-sun
+
+# Extra Java CLASSPATH elements.  Optional.
+# export HDFSPROXY_CLASSPATH=
+
+# The maximum amount of heap to use, in MB. Default is 1000.
+# export HDFSPROXY_HEAPSIZE=2000
+
+# Extra Java runtime options.  Empty by default.
+# export HDFSPROXY_OPTS=
+
+# Extra ssh options.  Empty by default.
+# export HDFSPROXY_SSH_OPTS="-o ConnectTimeout=1 -o SendEnv=HDFSPROXY_CONF_DIR"
+
+# Where log files are stored.  $HDFSPROXY_HOME/logs by default.
+# export HDFSPROXY_LOG_DIR=${HDFSPROXY_HOME}/logs
+
+# File naming remote slave hosts.  $HDFSPROXY_HOME/conf/slaves by default.
+# export HDFSPROXY_SLAVES=${HDFSPROXY_HOME}/conf/slaves
+
+# host:path where hdfsproxy code should be rsync'd from.  Unset by default.
+# export HDFSPROXY_MASTER=master:/home/$USER/src/hdfsproxy
+
+# Seconds to sleep between slave commands.  Unset by default.  This
+# can be useful in large clusters, where, e.g., slave rsyncs can
+# otherwise arrive faster than the master can service them.
+# export HDFSPROXY_SLAVE_SLEEP=0.1
+
+# The directory where pid files are stored. /tmp by default.
+# export HDFSPROXY_PID_DIR=/var/hdfsproxy/pids
+
+# A string representing this instance of hdfsproxy. $USER by default.
+# export HDFSPROXY_IDENT_STRING=$USER
+
+# The scheduling priority for daemon processes.  See 'man nice'.
+# export HDFSPROXY_NICENESS=10

src/contrib/hdfsproxy/conf/hdfsproxy-hosts

@@ -0,0 +1 @@
+localhost

src/contrib/hdfsproxy/conf/log4j.properties

@@ -0,0 +1,61 @@
+# Define some default values that can be overridden by system properties
+hdfsproxy.root.logger=INFO,console
+hdfsproxy.log.dir=.
+hdfsproxy.log.file=hdfsproxy.log
+
+# Define the root logger to the system property "hdfsproxy.root.logger".
+log4j.rootLogger=${hdfsproxy.root.logger}
+
+# Logging Threshold
+log4j.threshhold=ALL
+
+#
+# Daily Rolling File Appender
+#
+
+log4j.appender.DRFA=org.apache.log4j.DailyRollingFileAppender
+log4j.appender.DRFA.File=${hdfsproxy.log.dir}/${hdfsproxy.log.file}
+
+# Rollver at midnight
+log4j.appender.DRFA.DatePattern=.yyyy-MM-dd
+
+# 30-day backup
+#log4j.appender.DRFA.MaxBackupIndex=30
+log4j.appender.DRFA.layout=org.apache.log4j.PatternLayout
+
+# Pattern format: Date LogLevel LoggerName LogMessage
+log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
+# Debugging Pattern format
+#log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n
+
+
+#
+# console
+# Add "console" to rootlogger above if you want to use this 
+#
+
+log4j.appender.console=org.apache.log4j.ConsoleAppender
+log4j.appender.console.target=System.err
+log4j.appender.console.layout=org.apache.log4j.PatternLayout
+log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n
+
+#
+# Rolling File Appender
+#
+
+#log4j.appender.RFA=org.apache.log4j.RollingFileAppender
+#log4j.appender.RFA.File=${hdfsproxy.log.dir}/${hdfsproxy.log.file}
+
+# Logfile size and and 30-day backups
+#log4j.appender.RFA.MaxFileSize=1MB
+#log4j.appender.RFA.MaxBackupIndex=30
+
+#log4j.appender.RFA.layout=org.apache.log4j.PatternLayout
+#log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} - %m%n
+#log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n
+
+# Custom Logging levels
+
+#log4j.logger.org.apache.hadoop.hdfsproxy.HttpsProxy=DEBUG
+#log4j.logger.org.apache.hadoop.hdfsproxy.ProxyFilter=DEBUG
+

src/contrib/hdfsproxy/conf/ssl-server.xml

@@ -0,0 +1,48 @@
+<?xml version="1.0"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+
+<!--
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+-->
+
+<configuration>
+
+<property>
+  <name>ssl.server.truststore.location</name>
+  <value>${javax.net.ssl.keyStore.proxy}</value>
+</property>
+
+<property>
+  <name>ssl.server.truststore.password</name>
+  <value>changeme</value>
+</property>
+
+<property>
+  <name>ssl.server.keystore.location</name>
+  <value>${javax.net.ssl.keyStore.proxy}</value>
+</property>
+
+<property>
+  <name>ssl.server.keystore.password</name>
+  <value>changeme</value>
+</property>
+
+<property>
+  <name>ssl.server.keystore.keypassword</name>
+  <value>changeme</value>
+</property>
+
+</configuration>

src/contrib/hdfsproxy/conf/tomcat-forward-web.xml

@@ -0,0 +1,110 @@
+<?xml version="1.0" encoding="ISO-8859-1"?>
+<!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+
+<!DOCTYPE web-app 
+    PUBLIC "-//Sun Microsystems, Inc.//DTD Web Application 2.3//EN" 
+    "http://java.sun.com/dtd/web-app_2_3.dtd">
+
+<web-app>
+
+
+    <!-- General description of your web application -->
+
+    <display-name>HDFS Proxy</display-name>
+    <description>
+      get data from grid forward war
+    </description>
+
+    <context-param>
+      <param-name>webmaster</param-name>
+      <param-value>zhiyong1@yahoo-inc.com</param-value>
+      <description>
+        The EMAIL address of the administrator to whom questions
+        and comments about this application should be addressed.
+      </description>
+    </context-param>
+    
+    <filter>
+	   	<filter-name>proxyFilter</filter-name>
+	   	<filter-class>org.apache.hadoop.hdfsproxy.ProxyFilter</filter-class>
+	   	<init-param>
+	      <param-name>filteraddress</param-name>
+	      <param-value>10</param-value>
+	   	</init-param>
+		</filter>
+
+		<filter-mapping>
+        <filter-name>proxyFilter</filter-name>
+				<url-pattern>/*</url-pattern>
+    </filter-mapping>
+
+    
+    <servlet>
+    	<servlet-name>proxyForward</servlet-name>
+      <description>forward data access to specifc servlets</description>
+      <servlet-class>org.apache.hadoop.hdfsproxy.ProxyForwardServlet</servlet-class>
+    </servlet>
+    
+    <servlet-mapping>
+        <servlet-name>proxyForward</servlet-name>
+        <url-pattern>/listPaths/*</url-pattern>
+    </servlet-mapping>
+	  <servlet-mapping>
+        <servlet-name>proxyForward</servlet-name>
+        <url-pattern>/data/*</url-pattern>
+    </servlet-mapping>
+    <servlet-mapping>
+        <servlet-name>proxyForward</servlet-name>
+        <url-pattern>/streamFile/*</url-pattern>
+    </servlet-mapping>
+    
+    <servlet>
+    	<servlet-name>fileForward</servlet-name>
+      <description>forward file data access to streamFile</description>
+      <servlet-class>org.apache.hadoop.hdfsproxy.ProxyFileForward</servlet-class>
+    </servlet>
+    
+    <servlet-mapping>
+        <servlet-name>fileForward</servlet-name>
+        <url-pattern>/file/*</url-pattern>
+    </servlet-mapping>
+    
+
+		<welcome-file-list>
+		  <welcome-file>index.html</welcome-file>
+		</welcome-file-list>
+
+    <!-- Define the default session timeout for your application,
+         in minutes.  From a servlet or JSP page, you can modify
+         the timeout for a particular session dynamically by using
+         HttpSession.getMaxInactiveInterval(). -->
+
+    <session-config>
+      <session-timeout>30</session-timeout>    <!-- 30 minutes -->
+    </session-config>    
+
+
+</web-app>
+
+
+
+
+
+
+
+

src/contrib/hdfsproxy/conf/tomcat-web.xml

@@ -0,0 +1,158 @@
+<?xml version="1.0" encoding="ISO-8859-1"?>
+<!--
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+
+<!DOCTYPE web-app 
+    PUBLIC "-//Sun Microsystems, Inc.//DTD Web Application 2.3//EN" 
+    "http://java.sun.com/dtd/web-app_2_3.dtd">
+
+<web-app>
+
+
+    <!-- General description of your web application -->
+
+    <display-name>HDFS Proxy</display-name>
+    <description>
+      get data from grid 
+    </description>
+
+
+    <!-- Context initialization parameters that define shared
+         String constants used within your application, which
+         can be customized by the system administrator who is
+         installing your application.  The values actually
+         assigned to these parameters can be retrieved in a
+         servlet or JSP page by calling:
+
+             String value =
+               getServletContext().getInitParameter("name");
+
+         where "name" matches the <param-name> element of
+         one of these initialization parameters.
+
+         You can define any number of context initialization
+         parameters, including zero.
+    -->
+
+    <context-param>
+      <param-name>webmaster</param-name>
+      <param-value>zhiyong1@yahoo-inc.com</param-value>
+      <description>
+        The EMAIL address of the administrator to whom questions
+        and comments about this application should be addressed.
+      </description>
+    </context-param>
+    
+    <filter>
+	   	<filter-name>proxyFilter</filter-name>
+	   	<filter-class>org.apache.hadoop.hdfsproxy.ProxyFilter</filter-class>
+	   	<init-param>
+	      <param-name>filteraddress</param-name>
+	      <param-value>10</param-value>
+	   	</init-param>
+		</filter>
+
+		<filter-mapping>
+        <filter-name>proxyFilter</filter-name>
+				<url-pattern>/*</url-pattern>
+    </filter-mapping>
+    	
+
+
+    <!-- Servlet definitions for the servlets that make up
+         your web application, including initialization
+         parameters.  With Tomcat, you can also send requests
+         to servlets not listed here with a request like this:
+
+           http://localhost:8080/{context-path}/servlet/{classname}
+
+         but this usage is not guaranteed to be portable.  It also
+         makes relative references to images and other resources
+         required by your servlet more complicated, so defining
+         all of your servlets (and defining a mapping to them with
+         a servlet-mapping element) is recommended.
+
+         Servlet initialization parameters can be retrieved in a
+         servlet or JSP page by calling:
+
+             String value =
+               getServletConfig().getInitParameter("name");
+
+         where "name" matches the <param-name> element of
+         one of these initialization parameters.
+
+         You can define any number of servlets, including zero.
+    -->
+
+    
+    <servlet>
+    	<servlet-name>listPaths</servlet-name>
+      <description>list paths data access</description>
+      <servlet-class>org.apache.hadoop.hdfsproxy.ProxyListPathsServlet</servlet-class>
+    </servlet>
+    
+    <servlet-mapping>
+        <servlet-name>listPaths</servlet-name>
+        <url-pattern>/listPaths/*</url-pattern>
+    </servlet-mapping>
+
+		<servlet>
+    	<servlet-name>data</servlet-name>
+      <description>data access</description>
+      <servlet-class>org.apache.hadoop.hdfsproxy.ProxyFileDataServlet</servlet-class>
+    </servlet>
+    
+	  <servlet-mapping>
+        <servlet-name>data</servlet-name>
+        <url-pattern>/data/*</url-pattern>
+    </servlet-mapping>
+    
+    <servlet>
+    	<servlet-name>streamFile</servlet-name>
+      <description>stream file access</description>
+      <servlet-class>org.apache.hadoop.hdfsproxy.ProxyStreamFile</servlet-class>
+    </servlet>
+    
+    <servlet-mapping>
+        <servlet-name>streamFile</servlet-name>
+        <url-pattern>/streamFile/*</url-pattern>
+    </servlet-mapping>
+    
+
+		<welcome-file-list>
+		  <welcome-file>index.html</welcome-file>
+		</welcome-file-list>
+
+    <!-- Define the default session timeout for your application,
+         in minutes.  From a servlet or JSP page, you can modify
+         the timeout for a particular session dynamically by using
+         HttpSession.getMaxInactiveInterval(). -->
+
+    <session-config>
+      <session-timeout>30</session-timeout>    <!-- 30 minutes -->
+    </session-config>    
+
+
+</web-app>
+
+
+
+
+
+
+
+

src/contrib/hdfsproxy/conf/user-certs.xml

@@ -0,0 +1,32 @@
+<?xml version="1.0"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+
+<!-- 
+
+This file defines the mappings from username to comma seperated list
+of certificate serial numbers that the user is allowed to use. One mapping
+per user. Wildcard characters, such as "*" and "?", are not recognized. 
+Any leading or trailing whitespaces are stripped/ignored.
+
+-->
+
+<configuration>
+<property>
+  <name> nobody </name>
+  <value> ,6  ,,  3 , 9a2cf0be9ddf8280
+
+
+
+         </value>
+</property>
+
+<property>
+  <name> Admin </name>
+  <value>, 6,  ,,  3 , 9a2cf0be9ddf8280
+
+
+
+         </value>
+</property>
+
+</configuration>

src/contrib/hdfsproxy/conf/user-permissions.xml

@@ -0,0 +1,26 @@
+<?xml version="1.0"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+
+<!-- 
+
+This file defines the mappings from user name to comma seperated list
+of directories/files that the user is allowed to access. One mapping
+per user. Wildcard characters, such as "*" and "?", are not recognized. 
+For example, to match "/output" directory, one can use "/output" or 
+"/output/", but not "/output/*". Note that any leading or trailing
+whitespaces are stripped/ignored for the name field.
+
+-->
+
+<configuration>
+<property>
+  <name> nobody </name>
+  <value> ,
+
+
+
+        /input, /user, /data </value>
+</property>
+
+
+</configuration>

src/contrib/hdfsproxy/ivy.xml

@@ -0,0 +1,90 @@
+<?xml version="1.0" ?>
+<ivy-module version="1.0">
+  <info organisation="org.apache.hadoop" module="${ant.project.name}">
+    <license name="Apache 2.0"/>
+    <ivyauthor name="Apache Hadoop Team" url="http://hadoop.apache.org"/>
+    <description>
+        Apache Hadoop contrib
+    </description>
+  </info>
+  <configurations defaultconfmapping="default">
+    <!--these match the Maven configurations-->
+    <conf name="default" extends="master,runtime"/>
+    <conf name="master" description="contains the artifact but no dependencies"/>
+    <conf name="runtime" description="runtime but not the artifact" />
+
+    <conf name="common" visibility="private" 
+      description="artifacts needed to compile/test the application"/>
+  </configurations>
+
+  <publications>
+    <!--get the artifact from our module name-->
+    <artifact conf="master"/>
+  </publications>
+  <dependencies>
+    <dependency org="log4j"
+      name="log4j"
+      rev="${log4j.version}"
+      conf="common->master"/>
+    <dependency org="commons-logging"
+      name="commons-logging"
+      rev="${commons-logging.version}"
+      conf="common->master"/>
+    <dependency org="commons-logging"
+      name="commons-logging-api"
+      rev="${commons-logging-api.version}"
+      conf="common->master"/>
+    <dependency org="junit"
+      name="junit"
+      rev="${junit.version}"
+      conf="common->master"/>
+    <dependency org="org.slf4j"
+      name="slf4j-api"
+      rev="${slf4j-api.version}"
+      conf="common->master"/>
+    <dependency org="org.slf4j"
+      name="slf4j-log4j12"
+      rev="${slf4j-log4j12.version}"
+      conf="common->master"/>
+    <dependency org="xmlenc"
+      name="xmlenc"
+      rev="${xmlenc.version}"
+      conf="common->master"/>
+    <dependency org="org.mortbay.jetty"
+      name="jetty"
+      rev="${jetty.version}"
+      conf="common->master"/>
+    <dependency org="org.mortbay.jetty"
+      name="jetty-util"
+      rev="${jetty-util.version}"
+      conf="common->master"/>
+    <dependency org="org.mortbay.jetty"
+      name="servlet-api-2.5"
+      rev="${servlet-api-2.5.version}"
+      conf="common->master"/>
+    <dependency org="org.eclipse.jdt"
+      name="core"
+      rev="${core.version}"
+      conf="common->master"/>
+    <dependency org="org.apache.cactus" name="cactus.core.framework.uberjar.javaEE.14" rev="${cactus.version}" conf="common->master"/>
+		<dependency org="org.apache.cactus" name="cactus.integration.ant" rev="${cactus.version}" conf="common->master"/>
+		<dependency org="org.apache.cactus" name="cactus.integration.shared.api" rev="${cactus.version}" conf="common->master"/>
+				
+		<dependency org="commons-httpclient" name="commons-httpclient" rev="3.1" conf="common->master"/>
+		<dependency org="commons-io" name="commons-io" rev="1.4" conf="common->master"/>
+		<dependency org="commons-lang" name="commons-lang" rev="2.3" conf="common->master"/>
+		<dependency org="commons-codec" name="commons-codec" rev="1.3" conf="common->master"/>
+		
+		<dependency org="aspectj" name="aspectjrt" rev="1.5.3" conf="common->master"/>
+		
+		<dependency org="org.codehaus.cargo" name="cargo-core-uberjar" rev="0.9" conf="common->master"/>
+		<dependency org="org.codehaus.cargo" name="cargo-ant" rev="0.9" conf="common->master"/>
+
+		<dependency org="javax.servlet" name="jsp-api" rev="2.0" conf="common->master"/>
+		<dependency org="javax.servlet" name="servlet-api" rev="2.5" conf="common->master"/>
+		<dependency org="javax.servlet" name="jstl" rev="1.1.2" conf="common->master"/>
+		<dependency org="taglibs" name="standard" rev="1.1.2" conf="common->master"/>
+	  
+		<dependency org="junitperf" name="junitperf" rev="1.8" conf="common->master"/>
+  </dependencies>
+</ivy-module>

src/contrib/hdfsproxy/ivy/libraries.properties

@@ -0,0 +1,18 @@
+#This properties file lists the versions of the various artifacts used by hadoop.
+#It drives ivy and the generation of a maven POM
+#These are the versions of our dependencies (in alphabetical order)
+ivy.version=2.0.0-rc2
+
+log4j.version=1.2.15
+slf4j-api.version=1.4.3
+slf4j-log4j12.version=1.4.3
+jetty.version=6.1.14
+jetty-util.version=6.1.14
+servlet-api-2.5.version=6.1.14
+cactus.version=1.8.0
+commons-logging.version=1.1
+commons-logging-api.version=1.0.4
+junit.version=3.8.2
+jsp.version=2.1
+core.version=3.1.1
+xmlenc.version=0.52

src/contrib/hdfsproxy/src/java/org/apache/hadoop/hdfsproxy/HdfsProxy.java

@@ -0,0 +1,152 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfsproxy;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.net.NetUtils;
+import org.apache.hadoop.util.StringUtils;
+
+/**
+ * A HTTPS/SSL proxy to HDFS, implementing certificate based access control.
+ */
+public class HdfsProxy {
+  public static final Log LOG = LogFactory.getLog(HdfsProxy.class);
+
+  private ProxyHttpServer server;
+  private InetSocketAddress sslAddr;
+  
+  /** Construct a proxy from the given configuration */
+  public HdfsProxy(Configuration conf) throws IOException {
+    try {
+      initialize(conf);
+    } catch (IOException e) {
+      this.stop();
+      throw e;
+    }
+  }
+  
+  private void initialize(Configuration conf) throws IOException {
+    sslAddr = getSslAddr(conf);
+    String nn = conf.get("hdfsproxy.dfs.namenode.address");
+    if (nn == null)
+      throw new IOException("HDFS NameNode address is not specified");
+    InetSocketAddress nnAddr = NetUtils.createSocketAddr(nn);
+    LOG.info("HDFS NameNode is at: " + nnAddr.getHostName() + ":" + nnAddr.getPort());
+
+    Configuration sslConf = new Configuration(false);
+    sslConf.addResource(conf.get("hdfsproxy.https.server.keystore.resource",
+        "ssl-server.xml"));
+    // unit testing
+    sslConf.set("proxy.http.test.listener.addr",
+                conf.get("proxy.http.test.listener.addr"));
+
+    this.server = new ProxyHttpServer(sslAddr, sslConf);
+    this.server.setAttribute("proxy.https.port", server.getPort());
+    this.server.setAttribute("name.node.address", nnAddr);
+    this.server.setAttribute("name.conf", new Configuration());
+    this.server.addGlobalFilter("ProxyFilter", ProxyFilter.class.getName(), null);
+    this.server.addServlet("listPaths", "/listPaths/*", ProxyListPathsServlet.class);
+    this.server.addServlet("data", "/data/*", ProxyFileDataServlet.class);
+    this.server.addServlet("streamFile", "/streamFile/*", ProxyStreamFile.class);
+  }
+
+  /** return the http port if any, only for testing purposes */
+  int getPort() throws IOException {
+    return server.getPort();
+  }
+  
+  /**
+   * Start the server.
+   */
+  public void start() throws IOException {
+    this.server.start();
+    LOG.info("HdfsProxy server up at: " + sslAddr.getHostName() + ":"
+        + sslAddr.getPort());
+  }
+  
+  /**
+   * Stop all server threads and wait for all to finish.
+   */
+  public void stop() {
+    try {
+      if (server != null) {
+        server.stop();
+        server.join();
+      }
+    } catch (Exception e) {
+      LOG.warn("Got exception shutting down proxy", e);
+    }
+  }
+  
+  /**
+   * Wait for service to finish.
+   * (Normally, it runs forever.)
+   */
+  public void join() {
+    try {
+      this.server.join();
+    } catch (InterruptedException ie) {
+    }
+  }
+  
+  static InetSocketAddress getSslAddr(Configuration conf) throws IOException {
+    String addr = conf.get("hdfsproxy.https.address");
+    if (addr == null)
+      throw new IOException("HdfsProxy address is not specified");
+    return NetUtils.createSocketAddr(addr);
+  }
+
+ 
+
+  public static HdfsProxy createHdfsProxy(String argv[], Configuration conf)
+      throws IOException {
+    if (argv.length > 0) {
+      System.err.println("Usage: HdfsProxy");
+      return null;
+    }
+    if (conf == null) {
+      conf = new Configuration(false);
+      conf.addResource("hdfsproxy-default.xml");
+    }
+   
+    StringUtils.startupShutdownMessage(HdfsProxy.class, argv, LOG);
+    HdfsProxy proxy = new HdfsProxy(conf);
+    proxy.start();
+    return proxy;
+  }
+
+  public static void main(String[] argv) throws Exception {
+    try {
+      HdfsProxy proxy = createHdfsProxy(argv, null);
+      if (proxy != null)
+        proxy.join();
+    } catch (Throwable e) {
+      LOG.error(StringUtils.stringifyException(e));
+      System.exit(-1);
+    }
+  }
+}

src/contrib/hdfsproxy/src/java/org/apache/hadoop/hdfsproxy/ProxyFileDataServlet.java

@@ -0,0 +1,66 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfsproxy;
+
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+
+import javax.servlet.ServletContext;
+import javax.servlet.ServletException;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.hdfs.protocol.ClientProtocol;
+import org.apache.hadoop.hdfs.server.namenode.FileDataServlet;
+import org.apache.hadoop.security.UnixUserGroupInformation;
+
+/** {@inheritDoc} */
+public class ProxyFileDataServlet extends FileDataServlet {
+  /** For java.io.Serializable */
+  private static final long serialVersionUID = 1L;
+  
+  /** {@inheritDoc} */
+  @Override
+  public void init() throws ServletException {
+    ServletContext context = getServletContext();
+    if (context.getAttribute("name.conf") == null) { 
+      context.setAttribute("name.conf", new Configuration());
+    }    
+  }
+
+  /** {@inheritDoc} */
+  @Override
+  protected URI createUri(FileStatus i, UnixUserGroupInformation ugi,
+      ClientProtocol nnproxy, HttpServletRequest request) throws IOException,
+      URISyntaxException {
+    return new URI(request.getScheme(), null, request.getServerName(), request
+        .getServerPort(), "/streamFile", "filename=" + i.getPath() + "&ugi="
+        + ugi, null);
+  }
+
+  /** {@inheritDoc} */
+  @Override
+  protected UnixUserGroupInformation getUGI(HttpServletRequest request) {
+    String userID = (String) request.getAttribute("org.apache.hadoop.hdfsproxy.authorized.userID");
+    UnixUserGroupInformation ugi = ProxyUgiManager.getUgiForUser(userID);
+    return ugi;
+  }
+}

src/contrib/hdfsproxy/src/java/org/apache/hadoop/hdfsproxy/ProxyFileForward.java

@@ -0,0 +1,41 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfsproxy;
+
+import javax.servlet.http.HttpServletRequest;
+
+import org.apache.hadoop.security.UnixUserGroupInformation;
+
+
+public class ProxyFileForward extends ProxyForwardServlet {
+  /** For java.io.Serializable */
+  private static final long serialVersionUID = 1L;
+
+  /** {@inheritDoc} */
+  @Override
+  protected String buildForwardPath(HttpServletRequest request, String pathInfo) {
+    String path = "/streamFile";
+    path += "?filename=" + request.getPathInfo();
+    UnixUserGroupInformation ugi = (UnixUserGroupInformation)request.getAttribute("authorized.ugi");
+    if (ugi != null) {
+      path += "&ugi=" + ugi.toString();
+    }
+    return path;
+  }
+  
+}

src/contrib/hdfsproxy/src/java/org/apache/hadoop/hdfsproxy/ProxyFilter.java

@@ -0,0 +1,384 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfsproxy;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.math.BigInteger;
+import java.security.cert.CertificateFactory;
+import java.security.cert.X509Certificate;
+import java.security.cert.CertificateExpiredException;
+import java.security.cert.CertificateNotYetValidException;
+import java.util.Enumeration;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
+import java.net.InetSocketAddress;
+
+import javax.servlet.Filter;
+import javax.servlet.FilterChain;
+import javax.servlet.FilterConfig;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import javax.servlet.ServletException;
+import javax.servlet.ServletRequest;
+import javax.servlet.ServletResponse;
+import javax.servlet.ServletContext;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.security.UnixUserGroupInformation;
+import org.apache.hadoop.net.NetUtils;
+
+public class ProxyFilter implements Filter {
+  public static final Log LOG = LogFactory.getLog(ProxyFilter.class);
+
+  /** Pattern for triggering reload of user permissions */
+  protected static final Pattern RELOAD_PATTERN = Pattern
+      .compile("^(/reloadPermFiles)$");
+  /** Pattern for triggering clearing of ugi Cache */
+  protected static final Pattern CLEAR_PATTERN = Pattern
+      .compile("^(/clearUgiCache)$");
+  /** Pattern for a filter to find out if a request is HFTP/HSFTP request */
+  protected static final Pattern HFTP_PATTERN = Pattern
+      .compile("^(/listPaths|/data|/streamFile|/file)$");
+  /**
+   * Pattern for a filter to find out if an HFTP/HSFTP request stores its file
+   * path in the extra path information associated with the URL; if not, the
+   * file path is stored in request parameter "filename"
+   */
+  protected static final Pattern FILEPATH_PATTERN = Pattern
+      .compile("^(/listPaths|/data|/file)$");
+
+  private static volatile Map<String, Set<Path>> permsMap;
+  private static volatile Map<String, Set<BigInteger>> certsMap;
+  static {
+    Configuration conf = new Configuration(false);
+    conf.addResource("hdfsproxy-default.xml");
+    Map<String, Set<Path>> pMap = getPermMap(conf);
+    permsMap = pMap != null ? pMap : new HashMap<String, Set<Path>>();
+    Map<String, Set<BigInteger>> cMap = getCertsMap(conf);
+    certsMap = cMap != null ? cMap : new HashMap<String, Set<BigInteger>>();
+  }
+  
+
+  /** {@inheritDoc} */
+  public void init(FilterConfig filterConfig) throws ServletException {
+    ServletContext context = filterConfig.getServletContext();
+    Configuration conf = new Configuration(false);
+    conf.addResource("hdfsproxy-default.xml");
+    conf.addResource("ssl-server.xml");
+    conf.addResource("hdfsproxy-site.xml");
+    String nn = conf.get("hdfsproxy.dfs.namenode.address");
+    if (nn == null) {
+      throw new ServletException("Proxy source cluster name node address not speficied");
+    }
+    InetSocketAddress nAddr = NetUtils.createSocketAddr(nn);
+    context.setAttribute("name.node.address", nAddr);
+    context.setAttribute("name.conf", new Configuration());   
+    
+    context.setAttribute("org.apache.hadoop.hdfsproxy.conf", conf);
+    LOG.info("proxyFilter initialization success: " + nn);
+  }
+
+  private static Map<String, Set<Path>> getPermMap(Configuration conf) {
+    String permLoc = conf.get("hdfsproxy.user.permissions.file.location",
+        "user-permissions.xml");
+    if (conf.getResource(permLoc) == null) {
+      LOG.warn("HdfsProxy user permissions file not found");
+      return null;
+    }
+    Configuration permConf = new Configuration(false);
+    permConf.addResource(permLoc);
+    Map<String, Set<Path>> map = new HashMap<String, Set<Path>>();
+    for (Map.Entry<String, String> e : permConf) {
+      String k = e.getKey();
+      String v = e.getValue();
+      if (k != null && k.length() != 0 && v != null && v.length() != 0) {
+        Set<Path> pathSet = new HashSet<Path>();
+        String[] paths = v.split(",\\s*");
+        for (String p : paths) {
+          if (p.length() != 0) {
+            pathSet.add(new Path(p));
+          }
+        }
+        map.put(k, pathSet);
+      }
+    }
+    return map;
+  }
+
+  private static Map<String, Set<BigInteger>> getCertsMap(Configuration conf) {
+    String certsLoc = conf.get("hdfsproxy.user.certs.file.location",
+        "user-certs.xml");
+    if (conf.getResource(certsLoc) == null) {
+      LOG.warn("HdfsProxy user certs file not found");
+      return null;
+    }
+    Configuration certsConf = new Configuration(false);
+    certsConf.addResource(certsLoc);
+    Map<String, Set<BigInteger>> map = new HashMap<String, Set<BigInteger>>();
+    for (Map.Entry<String, String> e : certsConf) {
+      String k = e.getKey();
+      String v = e.getValue().trim();
+      if (k != null && k.length() != 0 && v != null && v.length() != 0) {
+        Set<BigInteger> numSet = new HashSet<BigInteger>();
+        String[] serialnumbers = v.split("\\s*,\\s*");
+        for (String num : serialnumbers) {
+          if (num.length() != 0) {
+            numSet.add(new BigInteger(num, 16));
+          }
+        }
+        map.put(k, numSet);
+      }
+    }
+    return map;
+  }
+
+  /** {@inheritDoc} */
+  public void destroy() {
+  }
+  
+  
+
+  /** {@inheritDoc} */
+  public void doFilter(ServletRequest request, ServletResponse response,
+      FilterChain chain) throws IOException, ServletException {
+
+    HttpServletRequest rqst = (HttpServletRequest) request;
+    HttpServletResponse rsp = (HttpServletResponse) response;
+    
+    if (LOG.isDebugEnabled()) {
+      StringBuilder b = new StringBuilder("Request from ").append(
+          rqst.getRemoteHost()).append("/").append(rqst.getRemoteAddr())
+          .append(":").append(rqst.getRemotePort());
+
+      @SuppressWarnings("unchecked")
+      Enumeration<String> e = rqst.getAttributeNames();
+      for (; e.hasMoreElements();) {
+        String attribute = e.nextElement();
+        b.append("\n  " + attribute + " => " + rqst.getAttribute(attribute));
+      }
+
+      X509Certificate[] userCerts = (X509Certificate[]) rqst
+          .getAttribute("javax.servlet.request.X509Certificate");
+      if (userCerts != null)
+        for (X509Certificate cert : userCerts)
+          b.append("\n Client certificate Subject Name is "
+              + cert.getSubjectX500Principal().getName());
+
+      b.append("\n The Scheme is " + rqst.getScheme());
+      b.append("\n The Auth Type is " + rqst.getAuthType());
+      b.append("\n The Path Info is " + rqst.getPathInfo());
+      b.append("\n The Translated Path Info is " + rqst.getPathTranslated());
+      b.append("\n The Context Path is " + rqst.getContextPath());
+      b.append("\n The Query String is " + rqst.getQueryString());
+      b.append("\n The Remote User is " + rqst.getRemoteUser());
+      b.append("\n The User Principal is " + rqst.getUserPrincipal());
+      b.append("\n The Request URI is " + rqst.getRequestURI());
+      b.append("\n The Request URL is " + rqst.getRequestURL());
+      b.append("\n The Servlet Path is " + rqst.getServletPath());
+
+      LOG.debug(b.toString());
+    }
+    
+    boolean unitTest = false;
+    if (rqst.getScheme().equalsIgnoreCase("http") && rqst.getParameter("UnitTest") != null) unitTest = true;
+    
+    if (rqst.getScheme().equalsIgnoreCase("https") || unitTest) {
+      boolean isAuthorized = false;
+      X509Certificate[] certs = (X509Certificate[]) rqst.getAttribute("javax.servlet.request.X509Certificate");
+      
+      if (unitTest) {
+        try {
+          LOG.debug("==> Entering https unit test");
+          String SslPath = rqst.getParameter("SslPath");
+          InputStream inStream = new FileInputStream(SslPath);
+          CertificateFactory cf = CertificateFactory.getInstance("X.509");
+          X509Certificate cert = (X509Certificate)cf.generateCertificate(inStream);
+          inStream.close();          
+          certs = new X509Certificate[] {cert};
+        } catch (Exception e) {
+          // do nothing here
+        }
+      } 
+      
+      if (certs == null || certs.length == 0) {
+        rsp.sendError(HttpServletResponse.SC_BAD_REQUEST,
+          "No client SSL certificate received");
+        LOG.info("No Client SSL certificate received");
+        return;       
+      }
+      for (X509Certificate cert : certs) {
+        try {
+          cert.checkValidity();
+        } catch (CertificateExpiredException e) {
+          LOG.info("Received cert for "
+              + cert.getSubjectX500Principal().getName() + " expired");
+          rsp
+              .sendError(HttpServletResponse.SC_FORBIDDEN,
+                  "Certificate expired");
+          return;
+        } catch (CertificateNotYetValidException e) {
+          LOG.info("Received cert for "
+              + cert.getSubjectX500Principal().getName() + " is not yet valid");
+          rsp.sendError(HttpServletResponse.SC_FORBIDDEN,
+              "Certificate is not yet valid");
+          return;
+        }
+      }
+      
+      String[] tokens = certs[0].getSubjectX500Principal().getName().split(
+          "\\s*,\\s*");
+      String userID = null;
+      for (String s : tokens) {
+        if (s.startsWith("CN=")) {
+          userID = s;
+          break;
+        }
+      }
+      if (userID == null || userID.length() < 4) {
+        LOG.info("Can't retrieve user ID from SSL certificate");
+        rsp.sendError(HttpServletResponse.SC_FORBIDDEN,
+            "Can't retrieve user ID from SSL certificate");
+        return;
+      }
+      userID = userID.substring(3);
+      
+      String servletPath = rqst.getServletPath();
+      if (unitTest) { 
+        servletPath = rqst.getParameter("TestSevletPathInfo");
+        LOG.info("this is for unit test purpose only");
+      }
+      
+      if (HFTP_PATTERN.matcher(servletPath).matches()) {
+        // request is an HSFTP request
+        if (FILEPATH_PATTERN.matcher(servletPath).matches()) {
+          // file path as part of the URL
+          isAuthorized = checkPath(userID, certs[0],
+              rqst.getPathInfo() != null ? rqst.getPathInfo() : "/");
+        } else {
+          // file path is stored in "filename" parameter
+          isAuthorized = checkPath(userID, certs[0], rqst
+              .getParameter("filename"));
+        }
+      } else if (RELOAD_PATTERN.matcher(servletPath).matches()
+          && checkUser("Admin", certs[0])) {
+        Configuration conf = new Configuration(false);
+        conf.addResource("hdfsproxy-default.xml");
+        Map<String, Set<Path>> permsMap = getPermMap(conf);
+        Map<String, Set<BigInteger>> certsMap = getCertsMap(conf);
+        if (permsMap == null || certsMap == null) {
+          LOG.warn("Permission files reloading failed");
+          rsp.sendError(HttpServletResponse.SC_INTERNAL_SERVER_ERROR,
+              "Permission files reloading failed");
+          return;
+        }
+        ProxyFilter.permsMap = permsMap;
+        ProxyFilter.certsMap = certsMap;
+        LOG.info("User permissions and user certs files reloaded");
+        rsp.setStatus(HttpServletResponse.SC_OK);
+        return;
+      } else if (CLEAR_PATTERN.matcher(servletPath).matches()
+          && checkUser("Admin", certs[0])) {
+        ProxyUgiManager.clearCache();
+        LOG.info("Ugi cache cleared");
+        rsp.setStatus(HttpServletResponse.SC_OK);
+        return;
+      } 
+
+      if (!isAuthorized) {
+        rsp.sendError(HttpServletResponse.SC_FORBIDDEN, "Unauthorized access");
+        return;
+      }
+      
+      // request is authorized, set ugi for servlets
+      UnixUserGroupInformation ugi = ProxyUgiManager
+          .getUgiForUser(userID);
+      if (ugi == null) {
+        LOG.info("Can't retrieve ugi for user " + userID);
+        rsp.sendError(HttpServletResponse.SC_INTERNAL_SERVER_ERROR,
+            "Can't retrieve ugi for user " + userID);
+        return;
+      }
+      rqst.setAttribute("authorized.ugi", ugi);
+      rqst.setAttribute("org.apache.hadoop.hdfsproxy.authorized.userID", userID);
+    } else if(rqst.getScheme().equalsIgnoreCase("http")) { // http request, set ugi for servlets, only for testing purposes
+      String ugi = rqst.getParameter("ugi");
+      if (ugi != null) {
+        rqst.setAttribute("authorized.ugi", new UnixUserGroupInformation(ugi
+          .split(",")));
+        String[] ugiStr = ugi.split(",");
+        if(ugiStr.length > 0) {
+          rqst.setAttribute("org.apache.hadoop.hdfsproxy.authorized.userID", ugiStr[0]);
+        }
+      } 
+    }
+    chain.doFilter(request, response);
+  }
+
+  /** check that client's cert is listed in the user certs file */
+  private boolean checkUser(String userID, X509Certificate cert) {
+    Set<BigInteger> numSet = certsMap.get(userID);
+    if (numSet == null) {
+      LOG.info("User " + userID + " is not configured in the user certs file");
+      return false;
+    }
+    if (!numSet.contains(cert.getSerialNumber())) {
+      LOG.info("Cert with serial number " + cert.getSerialNumber()
+          + " is not listed for user " + userID);
+      return false;
+    }
+    return true;
+  }
+
+  /** check that the requested path is listed in the user permissions file */
+  private boolean checkPath(String userID, X509Certificate cert, String pathInfo) {
+    if (!checkUser(userID, cert)) {
+      return false;
+    }
+
+    Set<Path> pathSet = permsMap.get(userID);
+    if (pathSet == null) {
+      LOG.info("User " + userID
+              + " is not listed in the user permissions file");
+      return false;
+    }
+    if (pathInfo == null || pathInfo.length() == 0) {
+      LOG.info("Can't get file path from HTTPS request; user is " + userID);
+      return false;
+    }
+    
+    Path userPath = new Path(pathInfo);
+    while (userPath != null) {
+      if (LOG.isDebugEnabled()) {
+        LOG.debug("\n Checking file path " + userPath);
+      }
+      if (pathSet.contains(userPath))
+        return true;
+      userPath = userPath.getParent();
+    }
+    LOG.info("User " + userID + " is not authorized to access " + pathInfo);
+    return false;
+  }
+}

src/contrib/hdfsproxy/src/java/org/apache/hadoop/hdfsproxy/ProxyForwardServlet.java

@@ -0,0 +1,99 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfsproxy;
+
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import java.io.IOException;
+import javax.servlet.ServletException;
+import javax.servlet.ServletContext;
+import javax.servlet.RequestDispatcher;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.security.UnixUserGroupInformation;
+
+/**
+ * 
+ *
+ */
+public class ProxyForwardServlet extends HttpServlet {
+  /**
+   * 
+   */
+  private static final long serialVersionUID = 1L;
+  private static Configuration configuration = null;
+  public static final Log LOG = LogFactory.getLog(ProxyForwardServlet.class);
+  
+  /** {@inheritDoc} */
+  @Override
+  public void init() throws ServletException {
+    ServletContext context = getServletContext();
+    configuration = (Configuration) context.getAttribute("org.apache.hadoop.hdfsproxy.conf");
+  }
+  
+  /** {@inheritDoc} */
+  @Override
+  public void doGet(HttpServletRequest request, HttpServletResponse response)
+    throws IOException, ServletException {  
+    String hostname = request.getServerName(); 
+    
+    String version = configuration.get(hostname);
+    if (version != null) {
+      ServletContext curContext = getServletContext();
+      ServletContext dstContext = curContext.getContext(version);
+      
+      if (dstContext == null) {
+        LOG.info("Context non-exist or restricted from access: " + version);
+        response.sendError(HttpServletResponse.SC_NOT_FOUND);
+        return;
+      }
+      LOG.debug("Request to " + hostname + " is forwarded to version " + version);
+      forwardRequest(request, response, dstContext, request.getServletPath());
+
+    } else {
+      LOG.info("not a valid context path");
+      response.sendError(HttpServletResponse.SC_NOT_IMPLEMENTED); 
+    }
+  } 
+  /** {@inheritDoc} */
+  public void forwardRequest(HttpServletRequest request, HttpServletResponse response, ServletContext context, String pathInfo) 
+    throws IOException, ServletException{
+    String path = buildForwardPath(request, pathInfo);    
+    RequestDispatcher dispatcher = context.getRequestDispatcher(path);
+    if (dispatcher == null) {
+      LOG.info("There was no such dispatcher");
+      response.sendError(HttpServletResponse.SC_NO_CONTENT);
+      return;
+    }
+    dispatcher.forward(request, response);
+  }
+  
+  /** {@inheritDoc} */
+  protected String buildForwardPath(HttpServletRequest request, String pathInfo) {
+    String path = pathInfo;
+    if (request.getPathInfo() != null) {
+      path += request.getPathInfo();
+    }
+    if (request.getQueryString() != null) {
+      path += "?" + request.getQueryString();
+    }
+    return path;
+  }
+}

src/contrib/hdfsproxy/src/java/org/apache/hadoop/hdfsproxy/ProxyHttpServer.java

@@ -0,0 +1,76 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfsproxy;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.util.Map;
+
+import javax.servlet.http.HttpServlet;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.http.HttpServer;
+import org.apache.hadoop.net.NetUtils;
+
+import org.mortbay.jetty.Connector;
+import org.mortbay.jetty.nio.SelectChannelConnector;
+import org.mortbay.jetty.security.SslSocketConnector;
+
+/**
+ * Create a Jetty embedded server to answer http/https requests.
+ */
+public class ProxyHttpServer extends HttpServer {
+  public static final Log LOG = LogFactory.getLog(ProxyHttpServer.class);
+
+  public ProxyHttpServer(InetSocketAddress addr, Configuration conf)
+      throws IOException {
+    super("", addr.getHostName(), addr.getPort(), 0 <= addr.getPort(), conf);
+  }
+
+  /** {@inheritDoc} */
+  protected Connector createBaseListener(Configuration conf)
+      throws IOException {
+    final String sAddr;
+    if (null == (sAddr = conf.get("proxy.http.test.listener.addr"))) {
+      SslSocketConnector sslListener = new SslSocketConnector();
+      sslListener.setKeystore(conf.get("ssl.server.keystore.location"));
+      sslListener.setPassword(conf.get("ssl.server.keystore.password", ""));
+      sslListener.setKeyPassword(conf.get("ssl.server.keystore.keypassword", ""));
+      sslListener.setKeystoreType(conf.get("ssl.server.keystore.type", "jks"));
+      sslListener.setNeedClientAuth(true);
+      System.setProperty("javax.net.ssl.trustStore",
+          conf.get("ssl.server.truststore.location", ""));
+      System.setProperty("javax.net.ssl.trustStorePassword",
+          conf.get("ssl.server.truststore.password", ""));
+      System.setProperty("javax.net.ssl.trustStoreType",
+          conf.get("ssl.server.truststore.type", "jks"));
+      return sslListener;
+    }
+    // unit test
+    InetSocketAddress proxyAddr = NetUtils.createSocketAddr(sAddr);
+    SelectChannelConnector testlistener = new SelectChannelConnector();
+    testlistener.setUseDirectBuffers(false);
+    testlistener.setHost(proxyAddr.getHostName());
+    testlistener.setPort(proxyAddr.getPort());
+    return testlistener;
+  }
+
+}

src/contrib/hdfsproxy/src/java/org/apache/hadoop/hdfsproxy/ProxyListPathsServlet.java

@@ -0,0 +1,49 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfsproxy;
+
+import javax.servlet.ServletContext;
+import javax.servlet.ServletException;
+import javax.servlet.http.HttpServletRequest;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.server.namenode.ListPathsServlet;
+import org.apache.hadoop.security.UnixUserGroupInformation;
+
+/** {@inheritDoc} */
+public class ProxyListPathsServlet extends ListPathsServlet {
+  /** For java.io.Serializable */
+  private static final long serialVersionUID = 1L;
+  
+  /** {@inheritDoc} */
+  @Override
+  public void init() throws ServletException {
+    ServletContext context = getServletContext();
+    if (context.getAttribute("name.conf") == null) { 
+      context.setAttribute("name.conf", new Configuration());
+    }    
+  }
+
+  /** {@inheritDoc} */
+  @Override
+  protected UnixUserGroupInformation getUGI(HttpServletRequest request) {
+    String userID = (String) request.getAttribute("org.apache.hadoop.hdfsproxy.authorized.userID");
+    UnixUserGroupInformation ugi = ProxyUgiManager.getUgiForUser(userID);
+    return ugi;
+  }
+}

src/contrib/hdfsproxy/src/java/org/apache/hadoop/hdfsproxy/ProxyStreamFile.java

@@ -0,0 +1,66 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfsproxy;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+
+import javax.servlet.ServletContext;
+import javax.servlet.ServletException;
+import javax.servlet.http.HttpServletRequest;
+
+import org.apache.hadoop.hdfs.DFSClient;
+import org.apache.hadoop.hdfs.server.namenode.StreamFile;
+import org.apache.hadoop.security.UnixUserGroupInformation;
+import org.apache.hadoop.conf.Configuration;
+
+/** {@inheritDoc} */
+public class ProxyStreamFile extends StreamFile {
+  /** For java.io.Serializable */
+  private static final long serialVersionUID = 1L;
+  /** {@inheritDoc} */
+  @Override
+  public void init() throws ServletException {
+    ServletContext context = getServletContext();
+    if (context.getAttribute("name.conf") == null) { 
+      context.setAttribute("name.conf", new Configuration());
+    }    
+  }
+
+  /** {@inheritDoc} */
+  @Override
+  protected DFSClient getDFSClient(HttpServletRequest request)
+      throws IOException {
+    ServletContext context = getServletContext();
+    Configuration conf = new Configuration((Configuration) context
+        .getAttribute("name.conf"));
+    UnixUserGroupInformation.saveToConf(conf,
+        UnixUserGroupInformation.UGI_PROPERTY_NAME, getUGI(request));
+    InetSocketAddress nameNodeAddr = (InetSocketAddress) context
+        .getAttribute("name.node.address");
+    return new DFSClient(nameNodeAddr, conf);
+  }
+
+  /** {@inheritDoc} */
+  @Override
+  protected UnixUserGroupInformation getUGI(HttpServletRequest request) {
+    String userID = (String) request.getAttribute("org.apache.hadoop.hdfsproxy.authorized.userID");
+    UnixUserGroupInformation ugi = ProxyUgiManager.getUgiForUser(userID);
+    return ugi;
+  }
+}

src/contrib/hdfsproxy/src/java/org/apache/hadoop/hdfsproxy/ProxyUgiManager.java

@@ -0,0 +1,152 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfsproxy;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.regex.Pattern;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.security.UnixUserGroupInformation;
+import org.apache.hadoop.util.Shell;
+
+/** An ugi manager that maintains a temporary ugi cache */
+public class ProxyUgiManager {
+  private static final Map<String, CachedUgi> ugiCache = new HashMap<String, CachedUgi>();
+  private static long ugiLifetime;
+  /** username can only comprise of 0-9a-zA-Z and underscore, i.e. \w */
+  private static final Pattern USERNAME_PATTERN = Pattern.compile("^\\w+$");
+  static final int CLEANUP_THRESHOLD = 1000;
+
+  static {
+    Configuration conf = new Configuration(false);
+    conf.addResource("hdfsproxy-default.xml");
+    ugiLifetime = conf.getLong("hdfsproxy.ugi.cache.ugi.lifetime", 15) * 60 * 1000L;
+  }
+
+  /**
+   * retrieve an ugi for a user. try the cache first, if not found, get it by
+   * running a shell command
+   */
+  public static synchronized UnixUserGroupInformation getUgiForUser(
+      String userName) {
+    long now = System.currentTimeMillis();
+    long cutoffTime = now - ugiLifetime;
+    CachedUgi cachedUgi = ugiCache.get(userName);
+    if (cachedUgi != null && cachedUgi.getInitTime() > cutoffTime)
+      return cachedUgi.getUgi();
+    UnixUserGroupInformation ugi = null;
+    try {
+      ugi = getUgi(userName);
+    } catch (IOException e) {
+      return null;
+    }
+    if (ugiCache.size() > CLEANUP_THRESHOLD) { // remove expired ugi's first
+      for (Iterator<Map.Entry<String, CachedUgi>> it = ugiCache.entrySet()
+          .iterator(); it.hasNext();) {
+        Map.Entry<String, CachedUgi> e = it.next();
+        if (e.getValue().getInitTime() < cutoffTime) {
+          it.remove();
+        }
+      }
+    }
+    ugiCache.put(ugi.getUserName(), new CachedUgi(ugi, now));
+    return ugi;
+  }
+
+  /** clear the ugi cache */
+  public static synchronized void clearCache() {
+    ugiCache.clear();
+  }
+
+  /** set ugi lifetime, only for junit testing purposes */
+  static synchronized void setUgiLifetime(long lifetime) {
+    ugiLifetime = lifetime;
+  }
+
+  /** save an ugi to cache, only for junit testing purposes */
+  static synchronized void saveToCache(UnixUserGroupInformation ugi) {
+    ugiCache.put(ugi.getUserName(), new CachedUgi(ugi, System
+        .currentTimeMillis()));
+  }
+
+  /** get cache size, only for junit testing purposes */
+  static synchronized int getCacheSize() {
+    return ugiCache.size();
+  }
+
+  /**
+   * Get the ugi for a user by running shell command "id -Gn"
+   * 
+   * @param userName name of the user
+   * @return ugi of the user
+   * @throws IOException if encounter any error while running the command
+   */
+  private static UnixUserGroupInformation getUgi(String userName)
+      throws IOException {
+    if (userName == null || !USERNAME_PATTERN.matcher(userName).matches())
+      throw new IOException("Invalid username=" + userName);
+    String[] cmd = new String[] { "bash", "-c", "id -Gn '" + userName + "'"};
+    String[] groups = Shell.execCommand(cmd).split("\\s+");
+    return new UnixUserGroupInformation(userName, groups);
+  }
+
+  /** cached ugi object with its associated init time */
+  private static class CachedUgi {
+    final UnixUserGroupInformation ugi;
+    final long initTime;
+
+    CachedUgi(UnixUserGroupInformation ugi, long initTime) {
+      this.ugi = ugi;
+      this.initTime = initTime;
+    }
+
+    UnixUserGroupInformation getUgi() {
+      return ugi;
+    }
+
+    long getInitTime() {
+      return initTime;
+    }
+
+    /** {@inheritDoc} */
+    public int hashCode() {
+      return ugi.hashCode();
+    }
+
+    static boolean isEqual(Object a, Object b) {
+      return a == b || (a != null && a.equals(b));
+    }
+
+    /** {@inheritDoc} */
+    public boolean equals(Object obj) {
+      if (obj == this) {
+        return true;
+      }
+      if (obj != null && obj instanceof CachedUgi) {
+        CachedUgi that = (CachedUgi) obj;
+        return isEqual(this.ugi, that.ugi) && this.initTime == that.initTime;
+      }
+      return false;
+    }
+
+  }
+}

src/contrib/hdfsproxy/src/java/org/apache/hadoop/hdfsproxy/ProxyUtil.java

@@ -0,0 +1,270 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfsproxy;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.HttpURLConnection;
+import java.net.InetSocketAddress;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.security.cert.X509Certificate;
+import java.util.Date;
+import java.util.Set;
+
+import javax.net.ssl.HttpsURLConnection;
+import javax.net.ssl.HostnameVerifier;
+import javax.net.ssl.SSLSession;
+import javax.servlet.http.HttpServletResponse;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSInputStream;
+import org.apache.hadoop.io.IOUtils;
+import org.apache.hadoop.net.NetUtils;
+import org.apache.hadoop.util.HostsFileReader;
+
+
+/**
+ * Proxy Utility .
+ */
+public class ProxyUtil {
+  public static final Log LOG = LogFactory.getLog(ProxyUtil.class);
+  private static final long MM_SECONDS_PER_DAY = 1000 * 60 * 60 * 24;
+  private static final int CERT_EXPIRATION_WARNING_THRESHOLD = 30; // 30 days warning
+  
+  private static enum UtilityOption {
+    RELOAD("-reloadPermFiles"), CLEAR("-clearUgiCache"), GET("-get"), CHECKCERTS("-checkcerts");
+
+    private String name = null;
+
+    private UtilityOption(String arg) {
+      this.name = arg;
+    }
+
+    public String getName() {
+      return name;
+    }
+  }
+  
+  /**
+   * Dummy hostname verifier that is used to bypass hostname checking
+   */
+  private static class DummyHostnameVerifier implements HostnameVerifier {
+    public boolean verify(String hostname, SSLSession session) {
+      return true;
+    }
+  }
+
+  private static HttpsURLConnection openConnection(String hostname, int port,
+      String path) throws IOException {
+    try {
+      final URL url = new URI("https", null, hostname, port, path, null, null)
+          .toURL();
+      HttpsURLConnection conn = (HttpsURLConnection) url.openConnection();
+      // bypass hostname verification
+      conn.setHostnameVerifier(new DummyHostnameVerifier());
+      conn.setRequestMethod("GET");
+      return conn;
+    } catch (URISyntaxException e) {
+      throw (IOException) new IOException().initCause(e);
+    }
+  }
+
+  private static void setupSslProps(Configuration conf) {
+    System.setProperty("javax.net.ssl.trustStore", conf
+        .get("ssl.client.truststore.location"));
+    System.setProperty("javax.net.ssl.trustStorePassword", conf.get(
+        "ssl.client.truststore.password", ""));
+    System.setProperty("javax.net.ssl.trustStoreType", conf.get(
+        "ssl.client.truststore.type", "jks"));
+    System.setProperty("javax.net.ssl.keyStore", conf
+        .get("ssl.client.keystore.location"));
+    System.setProperty("javax.net.ssl.keyStorePassword", conf.get(
+        "ssl.client.keystore.password", ""));
+    System.setProperty("javax.net.ssl.keyPassword", conf.get(
+        "ssl.client.keystore.keypassword", ""));
+    System.setProperty("javax.net.ssl.keyStoreType", conf.get(
+        "ssl.client.keystore.type", "jks"));
+  }
+
+  static InetSocketAddress getSslAddr(Configuration conf) throws IOException {
+    String addr = conf.get("hdfsproxy.https.address");
+    if (addr == null)
+      throw new IOException("HdfsProxy address is not specified");
+    return NetUtils.createSocketAddr(addr);
+  }
+
+  static boolean sendCommand(Configuration conf, String path)
+      throws IOException {
+    setupSslProps(conf);
+    int sslPort = getSslAddr(conf).getPort();
+    int err = 0;
+    StringBuilder b = new StringBuilder();
+
+    HostsFileReader hostsReader = new HostsFileReader(conf.get("hdfsproxy.hosts",
+        "hdfsproxy-hosts"), "");
+    Set<String> hostsList = hostsReader.getHosts();
+    for (String hostname : hostsList) {
+      HttpsURLConnection connection = null;
+      try {
+        connection = openConnection(hostname, sslPort, path);  
+        connection.connect(); 
+        if (LOG.isDebugEnabled()) {
+          StringBuffer sb = new StringBuffer();
+          X509Certificate[] clientCerts = (X509Certificate[]) connection.getLocalCertificates();
+          if (clientCerts != null) {
+            for (X509Certificate cert : clientCerts)
+              sb.append("\n Client certificate Subject Name is "
+                  + cert.getSubjectX500Principal().getName());
+          } else {
+            sb.append("\n No client certificates were found");  
+          }
+          X509Certificate[] serverCerts = (X509Certificate[]) connection.getServerCertificates();
+          if (serverCerts != null) {
+            for (X509Certificate cert : serverCerts)
+              sb.append("\n Server certificate Subject Name is "
+                  + cert.getSubjectX500Principal().getName());
+          } else {
+            sb.append("\n No server certificates were found");  
+          }
+          LOG.debug(sb.toString());
+        }
+        if (connection.getResponseCode() != HttpServletResponse.SC_OK) {
+          b.append("\n\t" + hostname + ": " + connection.getResponseCode()
+              + " " + connection.getResponseMessage());
+          err++;
+        }
+      } catch (IOException e) {
+        b.append("\n\t" + hostname + ": " + e.getLocalizedMessage());
+        if (LOG.isDebugEnabled()) e.printStackTrace();
+        err++;
+      } finally {
+        if (connection != null)
+          connection.disconnect();
+      }
+    }
+    if (err > 0) {
+      System.err.print("Command failed on the following "
+          + err + " host" + (err==1?":":"s:") + b.toString() + "\n");
+      return false;
+    }
+    return true;
+  }
+  
+  
+  static FSDataInputStream open(Configuration conf, String hostname, int port, String path) throws IOException {
+    setupSslProps(conf);
+    HttpURLConnection connection = null;
+    connection = openConnection(hostname, port, path);
+    connection.connect();
+    final InputStream in = connection.getInputStream();
+    return new FSDataInputStream(new FSInputStream() {
+        public int read() throws IOException {
+          return in.read();
+        }
+        public int read(byte[] b, int off, int len) throws IOException {
+          return in.read(b, off, len);
+        }
+
+        public void close() throws IOException {
+          in.close();
+        }
+
+        public void seek(long pos) throws IOException {
+          throw new IOException("Can't seek!");
+        }
+        public long getPos() throws IOException {
+          throw new IOException("Position unknown!");
+        }
+        public boolean seekToNewSource(long targetPos) throws IOException {
+          return false;
+        }
+      });
+  }
+  
+  static void checkServerCertsExpirationDays(Configuration conf, String hostname, int port) throws IOException {
+    setupSslProps(conf);
+    HttpsURLConnection connection = null;
+    connection = openConnection(hostname, port, null);
+    connection.connect();
+    X509Certificate[] serverCerts = (X509Certificate[]) connection.getServerCertificates();
+    Date curDate = new Date();
+    long curTime = curDate.getTime();
+    if (serverCerts != null) {
+      for (X509Certificate cert : serverCerts) {
+        StringBuffer sb = new StringBuffer();
+        sb.append("\n Server certificate Subject Name: " + cert.getSubjectX500Principal().getName());
+        Date expDate = cert.getNotAfter();
+        long expTime = expDate.getTime();
+        int dayOffSet = (int) ((expTime - curTime)/MM_SECONDS_PER_DAY);
+        sb.append(" have " + dayOffSet + " days to expire");
+        if (dayOffSet < CERT_EXPIRATION_WARNING_THRESHOLD) LOG.warn(sb.toString());
+        else LOG.info(sb.toString());
+      }
+    } else {
+      LOG.info("\n No Server certs was found");  
+    }
+
+    if (connection != null) {
+      connection.disconnect();
+    }
+  }
+
+  public static void main(String[] args) throws Exception {
+    if(args.length < 1 || 
+        (!UtilityOption.RELOAD.getName().equalsIgnoreCase(args[0]) 
+            && !UtilityOption.CLEAR.getName().equalsIgnoreCase(args[0])
+            && !UtilityOption.GET.getName().equalsIgnoreCase(args[0])
+            && !UtilityOption.CHECKCERTS.getName().equalsIgnoreCase(args[0])) ||
+            (UtilityOption.GET.getName().equalsIgnoreCase(args[0]) && args.length != 4) ||
+            (UtilityOption.CHECKCERTS.getName().equalsIgnoreCase(args[0]) && args.length != 3)) {
+      System.err.println("Usage: ProxyUtil ["
+          + UtilityOption.RELOAD.getName() + "] | ["
+          + UtilityOption.CLEAR.getName() + "] | ["
+          + UtilityOption.GET.getName() + " <hostname> <#port> <path> ] | ["
+          + UtilityOption.CHECKCERTS.getName() + " <hostname> <#port> ]");
+      System.exit(0);      
+    }
+    Configuration conf = new Configuration(false);   
+    conf.addResource("ssl-client.xml");
+    conf.addResource("hdfsproxy-default.xml");
+     
+    if (UtilityOption.RELOAD.getName().equalsIgnoreCase(args[0])) {
+      // reload user-certs.xml and user-permissions.xml files
+      sendCommand(conf, "/reloadPermFiles");
+    } else if (UtilityOption.CLEAR.getName().equalsIgnoreCase(args[0])) {
+      // clear UGI caches
+      sendCommand(conf, "/clearUgiCache");
+    } else if (UtilityOption.CHECKCERTS.getName().equalsIgnoreCase(args[0])) {
+      checkServerCertsExpirationDays(conf, args[1], Integer.parseInt(args[2]));
+    } else {
+      String hostname = args[1];
+      int port = Integer.parseInt(args[2]);
+      String path = args[3];
+      InputStream in = open(conf, hostname, port, path);
+      IOUtils.copyBytes(in, System.out, conf, false);
+      in.close();
+    }
+  }
+        
+}

src/contrib/hdfsproxy/src/test/org/apache/hadoop/hdfsproxy/FindFreePort.java

@@ -0,0 +1,87 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfsproxy;
+
+import java.net.ServerSocket;
+import java.io.IOException;
+import java.net.BindException;
+import java.util.Random;
+
+public class FindFreePort {
+  private static final int MIN_AVAILABLE_PORT = 10000;
+  private static final int MAX_AVAILABLE_PORT = 65535;
+  private static Random random = new Random();
+  /**
+   * 
+   * @param num <= 0, find a single free port
+   * @return free port next to port (>port)
+   * @throws IOException
+   */
+  public static int findFreePort(int port) throws IOException {
+    ServerSocket server;
+    if (port < 0) {
+      server =  new ServerSocket(0);      
+    } else {
+      int freePort = port+1;
+      while (true) {
+        try {
+          server =  new ServerSocket(freePort);
+          break;
+        } catch (IOException e) {
+          if (e instanceof BindException) {
+            if (freePort >= MAX_AVAILABLE_PORT || 
+                freePort < MIN_AVAILABLE_PORT) {
+              throw e;
+            }
+          } else {
+            throw e;
+          }
+          freePort += 1;
+        }
+      }
+    }
+    int fport = server.getLocalPort();
+    server.close();
+    return fport;    
+  }
+ /**
+  * 
+  * @return
+  * @throws IOException
+  */
+  public static int findFreePortRandom() throws IOException {
+    return findFreePort(MIN_AVAILABLE_PORT + random.nextInt(MAX_AVAILABLE_PORT - MIN_AVAILABLE_PORT + 1));
+  }
+   
+
+  public static void main(String[] args) throws Exception {
+    if(args.length < 1) {       
+      System.err.println("Usage: FindFreePort < -random / <#port> >");        
+      System.exit(0);      
+    }
+    int j = 0;
+    String cmd = args[j++];
+    if ("-random".equals(cmd)) {
+      System.out.println(findFreePortRandom());
+    } else {
+      System.out.println(findFreePort(Integer.parseInt(cmd)));
+    }   
+  }
+        
+}

src/contrib/hdfsproxy/src/test/org/apache/hadoop/hdfsproxy/SimpleServlet.java

@@ -0,0 +1,51 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfsproxy;
+
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+
+import java.io.PrintWriter;
+import java.io.IOException;
+
+
+/**
+ * simple servlet for forward testing purpose
+ */
+
+public class SimpleServlet extends HttpServlet {
+ 
+  /**
+   * 
+   */
+  private static final long serialVersionUID = 1L;
+
+  public void doGet(HttpServletRequest request, HttpServletResponse response)
+    throws IOException {
+	  response.setContentType("text/html");
+	  PrintWriter out = response.getWriter();	  
+	  out.print("<html><head/><body>");
+    out.print("A GET request");
+    out.print("</body></html>");
+	  out.close();
+	  return;
+  }
+
+}

src/contrib/hdfsproxy/src/test/org/apache/hadoop/hdfsproxy/TestHdfsProxy.java

@@ -0,0 +1,262 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfsproxy;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.net.URI;
+import java.util.Random;
+
+import junit.framework.TestCase;
+
+import org.apache.commons.logging.LogFactory;
+import org.apache.commons.logging.impl.Log4JLogger;
+import org.apache.log4j.Level;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hadoop.hdfs.server.datanode.DataNode;
+import org.apache.hadoop.hdfs.server.namenode.FSNamesystem;
+import org.apache.hadoop.net.NetUtils;
+import org.apache.hadoop.tools.DistCp;
+import org.apache.hadoop.util.ToolRunner;
+
+/**
+ * A JUnit test for HdfsProxy
+ */
+public class TestHdfsProxy extends TestCase {
+  {
+    ((Log4JLogger) LogFactory.getLog("org.apache.hadoop.hdfs.StateChange"))
+        .getLogger().setLevel(Level.OFF);
+    ((Log4JLogger) DataNode.LOG).getLogger().setLevel(Level.OFF);
+    ((Log4JLogger) FSNamesystem.LOG).getLogger().setLevel(Level.OFF);
+    ((Log4JLogger) DistCp.LOG).getLogger().setLevel(Level.ALL);
+  }
+
+  static final URI LOCAL_FS = URI.create("file:///");
+
+  private static final int NFILES = 10;
+  private static String TEST_ROOT_DIR = new Path(System.getProperty(
+      "test.build.data", "/tmp")).toString().replace(' ', '+');
+
+  /**
+   * class MyFile contains enough information to recreate the contents of a
+   * single file.
+   */
+  private static class MyFile {
+    private static Random gen = new Random();
+    private static final int MAX_LEVELS = 3;
+    private static final int MAX_SIZE = 8 * 1024;
+    private static String[] dirNames = { "zero", "one", "two", "three", "four",
+        "five", "six", "seven", "eight", "nine" };
+    private final String name;
+    private int size = 0;
+    private long seed = 0L;
+
+    MyFile() {
+      this(gen.nextInt(MAX_LEVELS));
+    }
+
+    MyFile(int nLevels) {
+      String xname = "";
+      if (nLevels != 0) {
+        int[] levels = new int[nLevels];
+        for (int idx = 0; idx < nLevels; idx++) {
+          levels[idx] = gen.nextInt(10);
+        }
+        StringBuffer sb = new StringBuffer();
+        for (int idx = 0; idx < nLevels; idx++) {
+          sb.append(dirNames[levels[idx]]);
+          sb.append("/");
+        }
+        xname = sb.toString();
+      }
+      long fidx = gen.nextLong() & Long.MAX_VALUE;
+      name = xname + Long.toString(fidx);
+      reset();
+    }
+
+    void reset() {
+      final int oldsize = size;
+      do {
+        size = gen.nextInt(MAX_SIZE);
+      } while (oldsize == size);
+      final long oldseed = seed;
+      do {
+        seed = gen.nextLong() & Long.MAX_VALUE;
+      } while (oldseed == seed);
+    }
+
+    String getName() {
+      return name;
+    }
+
+    int getSize() {
+      return size;
+    }
+
+    long getSeed() {
+      return seed;
+    }
+  }
+
+  private static MyFile[] createFiles(URI fsname, String topdir)
+      throws IOException {
+    return createFiles(FileSystem.get(fsname, new Configuration()), topdir);
+  }
+
+  /**
+   * create NFILES with random names and directory hierarchies with random (but
+   * reproducible) data in them.
+   */
+  private static MyFile[] createFiles(FileSystem fs, String topdir)
+      throws IOException {
+    Path root = new Path(topdir);
+    MyFile[] files = new MyFile[NFILES];
+    for (int i = 0; i < NFILES; i++) {
+      files[i] = createFile(root, fs);
+    }
+    return files;
+  }
+
+  private static MyFile createFile(Path root, FileSystem fs, int levels)
+      throws IOException {
+    MyFile f = levels < 0 ? new MyFile() : new MyFile(levels);
+    Path p = new Path(root, f.getName());
+    FSDataOutputStream out = fs.create(p);
+    byte[] toWrite = new byte[f.getSize()];
+    new Random(f.getSeed()).nextBytes(toWrite);
+    out.write(toWrite);
+    out.close();
+    FileSystem.LOG.info("created: " + p + ", size=" + f.getSize());
+    return f;
+  }
+
+  private static MyFile createFile(Path root, FileSystem fs) throws IOException {
+    return createFile(root, fs, -1);
+  }
+
+  private static boolean checkFiles(FileSystem fs, String topdir, MyFile[] files)
+      throws IOException {
+    return checkFiles(fs, topdir, files, false);
+  }
+
+  private static boolean checkFiles(FileSystem fs, String topdir,
+      MyFile[] files, boolean existingOnly) throws IOException {
+    Path root = new Path(topdir);
+
+    for (int idx = 0; idx < files.length; idx++) {
+      Path fPath = new Path(root, files[idx].getName());
+      try {
+        fs.getFileStatus(fPath);
+        FSDataInputStream in = fs.open(fPath);
+        byte[] toRead = new byte[files[idx].getSize()];
+        byte[] toCompare = new byte[files[idx].getSize()];
+        Random rb = new Random(files[idx].getSeed());
+        rb.nextBytes(toCompare);
+        assertEquals("Cannnot read file.", toRead.length, in.read(toRead));
+        in.close();
+        for (int i = 0; i < toRead.length; i++) {
+          if (toRead[i] != toCompare[i]) {
+            return false;
+          }
+        }
+        toRead = null;
+        toCompare = null;
+      } catch (FileNotFoundException fnfe) {
+        if (!existingOnly) {
+          throw fnfe;
+        }
+      }
+    }
+
+    return true;
+  }
+
+  /** delete directory and everything underneath it. */
+  private static void deldir(FileSystem fs, String topdir) throws IOException {
+    fs.delete(new Path(topdir), true);
+  }
+
+  /** verify hdfsproxy implements the hftp interface */
+  public void testHdfsProxyInterface() throws Exception {
+    MiniDFSCluster cluster = null;
+    HdfsProxy proxy = null;
+    try {
+
+      final Configuration dfsConf = new Configuration();
+      cluster = new MiniDFSCluster(dfsConf, 2, true, null);
+      cluster.waitActive();
+
+      final DistCp distcp = new DistCp(dfsConf);
+      final FileSystem localfs = FileSystem.get(LOCAL_FS, dfsConf);
+      final FileSystem hdfs = cluster.getFileSystem();
+      final Configuration proxyConf = new Configuration(false);
+      proxyConf.set("hdfsproxy.dfs.namenode.address", hdfs.getUri().getHost() + ":"
+          + hdfs.getUri().getPort());
+      proxyConf.set("hdfsproxy.https.address", "localhost:0");
+      final String namenode = hdfs.getUri().toString();
+      if (namenode.startsWith("hdfs://")) {
+        MyFile[] files = createFiles(LOCAL_FS, TEST_ROOT_DIR + "/srcdat");
+        ToolRunner.run(distcp, new String[] { "-log", namenode + "/logs",
+            "file:///" + TEST_ROOT_DIR + "/srcdat", namenode + "/destdat" });
+        assertTrue("Source and destination directories do not match.",
+            checkFiles(hdfs, "/destdat", files));
+        assertTrue("Log directory does not exist.", hdfs.exists(new Path(
+            namenode + "/logs")));
+
+        proxyConf.set("proxy.http.test.listener.addr", "localhost:0");
+        proxy = new HdfsProxy(proxyConf);
+        proxy.start();
+        InetSocketAddress proxyAddr = NetUtils.createSocketAddr("localhost:0");
+        final String realProxyAddr = proxyAddr.getHostName() + ":"
+            + proxy.getPort();
+
+        ToolRunner.run(distcp, new String[] {
+            "hftp://" + realProxyAddr + "/destdat", namenode + "/copied1" });
+        assertTrue("Source and copied directories do not match.", checkFiles(
+            hdfs, "/copied1", files));
+
+        ToolRunner.run(distcp, new String[] {
+            "hftp://" + realProxyAddr + "/destdat",
+            "file:///" + TEST_ROOT_DIR + "/copied2" });
+        assertTrue("Source and copied directories do not match.", checkFiles(
+            localfs, TEST_ROOT_DIR + "/copied2", files));
+
+        deldir(hdfs, "/destdat");
+        deldir(hdfs, "/logs");
+        deldir(hdfs, "/copied1");
+        deldir(localfs, TEST_ROOT_DIR + "/srcdat");
+        deldir(localfs, TEST_ROOT_DIR + "/copied2");
+      }
+    } finally {
+      if (cluster != null) {
+        cluster.shutdown();
+      }
+      if (proxy != null) {
+        proxy.stop();
+      }
+    }
+  }
+}

src/contrib/hdfsproxy/src/test/org/apache/hadoop/hdfsproxy/TestProxyFilter.java

@@ -0,0 +1,120 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfsproxy;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+
+import javax.servlet.FilterChain;
+import javax.servlet.FilterConfig;
+import javax.servlet.ServletException;
+import javax.servlet.ServletRequest;
+import javax.servlet.ServletResponse;
+import javax.servlet.ServletContext;
+
+import org.apache.cactus.FilterTestCase;
+import org.apache.cactus.WebRequest;
+import org.apache.cactus.WebResponse;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+
+public class TestProxyFilter extends FilterTestCase {
+  
+  public static final Log LOG = LogFactory.getLog(TestProxyFilter.class);
+  
+  private static String TEST_CLIENT_SSL_CERT = System.getProperty("javax.net.ssl.clientCert", 
+  "./src/test/resources/ssl-keys/test.crt");
+  
+  private class DummyFilterChain implements FilterChain {
+    public void doFilter(ServletRequest theRequest, ServletResponse theResponse) 
+      throws IOException, ServletException  {
+      PrintWriter writer = theResponse.getWriter();
+  
+      writer.print("<p>some content</p>");
+      writer.close();
+    }
+  
+    public void init(FilterConfig theConfig) {
+    }
+  
+    public void destroy() {
+    }
+  }
+   
+  public void beginDoFilterHttp(WebRequest theRequest) {
+    theRequest.addParameter("ugi", "nobody,test");
+  }  
+  
+  public void testDoFilterHttp() throws ServletException, IOException  {    
+    ProxyFilter filter = new ProxyFilter();
+    
+    ServletContext context = config.getServletContext();
+    context.removeAttribute("name.node.address");
+    context.removeAttribute("name.conf");
+    assertNull(context.getAttribute("name.node.address"));
+    assertNull(context.getAttribute("name.conf"));
+    
+    filter.init(config);
+    
+    assertNotNull(context.getAttribute("name.node.address"));
+    assertNotNull(context.getAttribute("name.conf"));
+    
+    request.removeAttribute("authorized.ugi");
+    assertNull(request.getAttribute("authorized.ugi"));
+        
+    FilterChain mockFilterChain = new DummyFilterChain();
+    filter.doFilter(request, response, mockFilterChain);    
+    assertEquals(request.getAttribute("authorized.ugi").toString(), "nobody,test");
+    
+  }
+
+  public void endDoFilterHttp(WebResponse theResponse)  {
+    assertEquals("<p>some content</p>", theResponse.getText());    
+  }
+  
+  public void beginDoFilterHttps(WebRequest theRequest) throws Exception{
+    theRequest.addParameter("UnitTest", "true");
+    theRequest.addParameter("SslPath", TEST_CLIENT_SSL_CERT);
+    theRequest.addParameter("ugi", "nobody,test");    
+    theRequest.addParameter("TestSevletPathInfo", "/streamFile");
+    theRequest.addParameter("filename", "/user");
+  }  
+  
+  public void testDoFilterHttps() throws Exception  {    
+    ProxyFilter filter = new ProxyFilter();
+    
+    request.removeAttribute("authorized.ugi");
+    assertNull(request.getAttribute("authorized.ugi"));        
+    
+    FilterChain mockFilterChain = new DummyFilterChain();
+    filter.init(config);
+    filter.doFilter(request, response, mockFilterChain);
+    
+    LOG.info("Finish setting up X509Certificate");  
+    assertEquals(request.getAttribute("authorized.ugi").toString().substring(0, 6), "nobody");
+    
+  }
+
+  public void endDoFilterHttps(WebResponse theResponse)  {
+    assertEquals("<p>some content</p>", theResponse.getText());    
+  }
+  
+    
+}
+

src/contrib/hdfsproxy/src/test/org/apache/hadoop/hdfsproxy/TestProxyForwardServlet.java

@@ -0,0 +1,69 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfsproxy;
+
+import org.apache.cactus.ServletTestCase;
+import org.apache.cactus.WebRequest;
+import org.apache.cactus.WebResponse;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+import java.io.IOException;
+import javax.servlet.ServletException;
+
+/** Unit tests for ProxyUtil */
+public class TestProxyForwardServlet extends ServletTestCase {
+  public static final Log LOG = LogFactory.getLog(TestProxyForwardServlet.class);
+  
+ 
+  public void beginDoGet(WebRequest theRequest) {
+    theRequest.setURL("proxy-test:0", null, "/simple", null, null);
+  }
+  
+  public void testDoGet() throws IOException, ServletException {
+    ProxyForwardServlet servlet = new ProxyForwardServlet();
+    
+    servlet.init(config);
+    servlet.doGet(request, response);
+  }
+  
+  public void endDoGet(WebResponse theResponse)
+    throws IOException {
+    String expected = "<html><head/><body>A GET request</body></html>";
+    String result = theResponse.getText();
+
+    assertEquals(expected, result);
+  }
+  
+  
+  public void testForwardRequest() throws Exception  {
+    ProxyForwardServlet servlet = new ProxyForwardServlet();
+
+    servlet.forwardRequest(request, response, config.getServletContext(), "/simple");
+  }
+  
+  public void endForwardRequest(WebResponse theResponse) throws IOException  {
+    String expected = "<html><head/><body>A GET request</body></html>";
+    String result = theResponse.getText();
+    
+    assertEquals(expected, result);
+    
+  } 
+ 
+}

src/contrib/hdfsproxy/src/test/org/apache/hadoop/hdfsproxy/TestProxyUgiManager.java

@@ -0,0 +1,107 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfsproxy;
+
+import org.apache.hadoop.security.UnixUserGroupInformation;
+
+import junit.framework.TestCase;
+
+/** Unit tests for ProxyUgiManager */
+public class TestProxyUgiManager extends TestCase {
+
+  private static final UnixUserGroupInformation root1Ugi = new UnixUserGroupInformation(
+      "root", new String[] { "group1" });
+  private static final UnixUserGroupInformation root2Ugi = new UnixUserGroupInformation(
+      "root", new String[] { "group2" });
+  private static final long ugiLifetime = 1000L; // milliseconds
+
+  /** Test caching functionality */
+  public void testCache() throws Exception {
+    ProxyUgiManager.saveToCache(root1Ugi);
+    UnixUserGroupInformation ugi = ProxyUgiManager.getUgiForUser(root1Ugi
+        .getUserName());
+    assertEquals(root1Ugi, ugi);
+    ProxyUgiManager.saveToCache(root2Ugi);
+    ugi = ProxyUgiManager.getUgiForUser(root2Ugi.getUserName());
+    assertEquals(root2Ugi, ugi);
+  }
+
+  /** Test clearCache method */
+  public void testClearCache() throws Exception {
+    UnixUserGroupInformation ugi = ProxyUgiManager.getUgiForUser(root1Ugi
+        .getUserName());
+    if (root1Ugi.equals(ugi)) {
+      ProxyUgiManager.saveToCache(root2Ugi);
+      ugi = ProxyUgiManager.getUgiForUser(root2Ugi.getUserName());
+      assertEquals(root2Ugi, ugi);
+      ProxyUgiManager.clearCache();
+      ugi = ProxyUgiManager.getUgiForUser(root2Ugi.getUserName());
+      assertFalse(root2Ugi.equals(ugi));
+    } else {
+      ProxyUgiManager.saveToCache(root1Ugi);
+      ugi = ProxyUgiManager.getUgiForUser(root1Ugi.getUserName());
+      assertEquals(root1Ugi, ugi);
+      ProxyUgiManager.clearCache();
+      ugi = ProxyUgiManager.getUgiForUser(root1Ugi.getUserName());
+      assertFalse(root1Ugi.equals(ugi));
+    }
+  }
+
+  /** Test cache timeout */
+  public void testTimeOut() throws Exception {
+    String[] users = new String[] { "root", "nobody", "SYSTEM",
+        "Administrator", "Administrators", "Guest" };
+    String realUser = null;
+    UnixUserGroupInformation ugi = null;
+    ProxyUgiManager.clearCache();
+    for (String user : users) {
+      ugi = ProxyUgiManager.getUgiForUser(user);
+      if (ugi != null) {
+        realUser = user;
+        break;
+      }
+    }
+    if (realUser != null) {
+      ProxyUgiManager.setUgiLifetime(ugiLifetime);
+      ProxyUgiManager.clearCache();
+      UnixUserGroupInformation[] fakedUgis = generateUgi(ProxyUgiManager.CLEANUP_THRESHOLD);
+      for (int i = 0; i < ProxyUgiManager.CLEANUP_THRESHOLD; i++) {
+        ProxyUgiManager.saveToCache(fakedUgis[i]);
+      }
+      assertTrue(ProxyUgiManager.getCacheSize() == ProxyUgiManager.CLEANUP_THRESHOLD);
+      Thread.sleep(ugiLifetime + 1000L);
+      UnixUserGroupInformation newugi = ProxyUgiManager.getUgiForUser(realUser);
+      assertTrue(ProxyUgiManager.getCacheSize() == ProxyUgiManager.CLEANUP_THRESHOLD + 1);
+      assertEquals(newugi, ugi);
+      Thread.sleep(ugiLifetime + 1000L);
+      newugi = ProxyUgiManager.getUgiForUser(realUser);
+      assertTrue(ProxyUgiManager.getCacheSize() == 1);
+      assertEquals(newugi, ugi);
+    }
+  }
+
+  private static UnixUserGroupInformation[] generateUgi(int size) {
+    UnixUserGroupInformation[] ugis = new UnixUserGroupInformation[size];
+    for (int i = 0; i < size; i++) {
+      ugis[i] = new UnixUserGroupInformation("user" + i,
+          new String[] { "group" });
+    }
+    return ugis;
+  }
+}

src/contrib/hdfsproxy/src/test/org/apache/hadoop/hdfsproxy/TestProxyUtil.java

@@ -0,0 +1,49 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfsproxy;
+
+import junit.framework.TestCase;
+
+import org.apache.hadoop.conf.Configuration;
+
+/** Unit tests for ProxyUtil */
+public class TestProxyUtil extends TestCase {
+  
+  private static String TEST_PROXY_CONF_DIR = System.getProperty("test.proxy.conf.dir", "./conf");
+  private static String TEST_PROXY_HTTPS_PORT = System.getProperty("test.proxy.https.port", "8443");
+
+  public void testSendCommand() throws Exception {
+      
+    Configuration conf = new Configuration(false);  
+    conf.addResource("ssl-client.xml");
+    conf.addResource("hdfsproxy-default.xml");
+    String address = "localhost:" + TEST_PROXY_HTTPS_PORT;
+    conf.set("hdfsproxy.https.address", address);
+    String hostFname = TEST_PROXY_CONF_DIR + "/hdfsproxy-hosts";
+    conf.set("hdfsproxy.hosts", hostFname);    
+    
+    assertTrue(ProxyUtil.sendCommand(conf, "/test/reloadPermFiles"));
+    assertTrue(ProxyUtil.sendCommand(conf, "/test/clearUgiCache"));    
+    
+    conf.set("hdfsproxy.https.address", "localhost:0");
+    assertFalse(ProxyUtil.sendCommand(conf, "/test/reloadPermFiles"));
+    assertFalse(ProxyUtil.sendCommand(conf, "/test/reloadPermFiles"));
+  }
+ 
+}

src/contrib/hdfsproxy/src/test/resources/cactus-web.xml

@@ -0,0 +1,77 @@
+<?xml version="1.0" encoding="ISO-8859-1"?>
+<!DOCTYPE web-app
+    PUBLIC "-//Sun Microsystems, Inc.//DTD Web Application 2.3//EN"
+    "http://java.sun.com/dtd/web-app_2_3.dtd">
+<!--
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+-->
+
+<web-app>
+
+    <context-param>
+      <param-name>param</param-name>
+      <param-value>value used for testing</param-value>
+    </context-param>
+    
+    <servlet>
+        <servlet-name>ServletRedirector</servlet-name>
+        <servlet-class>org.apache.cactus.server.ServletTestRedirector</servlet-class>
+        <init-param>
+          <param-name>param1</param-name>
+          <param-value>value1 used for testing</param-value>
+        </init-param>
+    </servlet>
+    
+    <servlet>
+        <servlet-name>ServletRedirector_TestOverride</servlet-name>
+        <servlet-class>org.apache.cactus.server.ServletTestRedirector</servlet-class>
+        <init-param>
+          <param-name>param2</param-name>
+          <param-value>value2 used for testing</param-value>
+        </init-param>
+    </servlet>
+
+    <servlet>
+        <servlet-name>TestJsp</servlet-name>
+        <jsp-file>/test/test.jsp</jsp-file>
+    </servlet>
+
+    <servlet>
+        <servlet-name>JspRedirector</servlet-name>
+        <jsp-file>/jspRedirector.jsp</jsp-file>
+        <init-param>
+          <param-name>param1</param-name>
+          <param-value>value1 used for testing</param-value>
+        </init-param>
+    </servlet>
+		
+		<servlet>
+      <servlet-name>Simple</servlet-name>
+      <description> A Simple Servlet </description>
+      <servlet-class>org.apache.hadoop.hdfsproxy.SimpleServlet</servlet-class>
+    </servlet>
+    
+    <servlet-mapping>
+        <servlet-name>ServletRedirector_TestOverride</servlet-name>
+        <url-pattern>/ServletRedirectorOverride</url-pattern>
+    </servlet-mapping>
+    
+    <servlet-mapping>
+        <servlet-name>Simple</servlet-name>
+        <url-pattern>/simple/*</url-pattern>
+    </servlet-mapping>
+
+</web-app>

src/contrib/hdfsproxy/src/test/resources/proxy-config/hdfsproxy-default.xml

@@ -0,0 +1,74 @@
+<?xml version="1.0"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<!--
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+-->
+<!-- Put hdfsproxy specific properties in this file. -->
+
+<configuration>
+
+<property>
+  <name>hdfsproxy.https.address</name>
+  <value>0.0.0.0:8443</value>
+  <description>the SSL port that hdfsproxy listens on
+  </description>
+</property>
+
+<property>
+  <name>hdfsproxy.hosts</name>
+  <value>hdfsproxy-hosts</value>
+  <description>location of hdfsproxy-hosts file
+  </description>
+</property>
+
+<property>
+  <name>hdfsproxy.dfs.namenode.address</name>
+  <value>localhost:54321</value>
+  <description>namenode address of the HDFS cluster being proxied
+  </description>
+</property>
+
+<property>
+  <name>hdfsproxy.https.server.keystore.resource</name>
+  <value>ssl-server.xml</value>
+  <description>location of the resource from which ssl server keystore
+  information will be extracted
+  </description>
+</property>
+
+<property>
+  <name>hdfsproxy.user.permissions.file.location</name>
+  <value>user-permissions.xml</value>
+  <description>location of the user permissions file
+  </description>
+</property>
+
+<property>
+  <name>hdfsproxy.user.certs.file.location</name>
+  <value>user-certs.xml</value>
+  <description>location of the user certs file
+  </description>
+</property>
+
+<property>
+  <name>hdfsproxy.ugi.cache.ugi.lifetime</name>
+  <value>15</value>
+  <description> The lifetime (in minutes) of a cached ugi
+  </description>
+</property>
+
+</configuration>
+

src/contrib/hdfsproxy/src/test/resources/proxy-config/hdfsproxy-hosts

@@ -0,0 +1 @@
+localhost

src/contrib/hdfsproxy/src/test/resources/proxy-config/hdfsproxy-site.xml

@@ -0,0 +1,15 @@
+<?xml version="1.0"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+
+<!-- Put hdfsproxy specific properties in this file. -->
+
+<configuration>
+<property>
+  <name>proxy-test</name>
+  <value>/test</value>
+  <description>one hostname corresponds to one web application archive 
+  </description>
+</property>
+
+</configuration>
+

src/contrib/hdfsproxy/src/test/resources/proxy-config/log4j.properties

@@ -0,0 +1,76 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Define some default values that can be overridden by system properties
+hdfsproxy.root.logger=DEBUG,console
+hdfsproxy.log.dir=.
+hdfsproxy.log.file=hdfsproxy.log
+
+# Define the root logger to the system property "hdfsproxy.root.logger".
+log4j.rootLogger=${hdfsproxy.root.logger}
+
+# Logging Threshold
+log4j.threshhold=ALL
+
+#
+# Daily Rolling File Appender
+#
+
+log4j.appender.DRFA=org.apache.log4j.DailyRollingFileAppender
+log4j.appender.DRFA.File=${hdfsproxy.log.dir}/${hdfsproxy.log.file}
+
+# Rollver at midnight
+log4j.appender.DRFA.DatePattern=.yyyy-MM-dd
+
+# 30-day backup
+#log4j.appender.DRFA.MaxBackupIndex=30
+log4j.appender.DRFA.layout=org.apache.log4j.PatternLayout
+
+# Pattern format: Date LogLevel LoggerName LogMessage
+log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
+# Debugging Pattern format
+#log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n
+
+
+#
+# console
+# Add "console" to rootlogger above if you want to use this 
+#
+
+log4j.appender.console=org.apache.log4j.ConsoleAppender
+log4j.appender.console.target=System.err
+log4j.appender.console.layout=org.apache.log4j.PatternLayout
+log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n
+
+#
+# Rolling File Appender
+#
+
+#log4j.appender.RFA=org.apache.log4j.RollingFileAppender
+#log4j.appender.RFA.File=${hdfsproxy.log.dir}/${hdfsproxy.log.file}
+
+# Logfile size and and 30-day backups
+#log4j.appender.RFA.MaxFileSize=1MB
+#log4j.appender.RFA.MaxBackupIndex=30
+
+#log4j.appender.RFA.layout=org.apache.log4j.PatternLayout
+#log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} - %m%n
+#log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n
+
+# Custom Logging levels
+
+log4j.logger.org.apache.hadoop.hdfsproxy.HttpsProxy=DEBUG
+log4j.logger.org.apache.hadoop.hdfsproxy.ProxyFilter=DEBUG
+log4j.logger.org.apache.hadoop.hdfsproxy.HdfsProxy=DEBUG