ZOOKEEPER-3630: Autodetection of openssl during ZooKeeper C client build

**Thanks for ztzg for raising the issue and suggesting the solution!**

In this patch we enhance the way how the openssl library is found during C client build. I introduced  and documented a new build parameter for `configure` (`--with-openssl=...`), `cmake` (`-D WITH_OPENSSL=...`) and `mvn` (`-Dc-client-openssl=...`), so independent of the build tool, the build will work the same way:
- By default, the OpenSSL library will be autodetected. If the library is found, then the C-client will be compiled with SSL support, otherwise we get a warning message, but the build will continue without SSL support. The SSL related unit tests will be skipped as well.
- you can explicitly disable the SSL support (e.g. `-Dc-client-openssl=no`)
- or you can specify an alternative path to look for the openssl library (e.g. `-Dc-client-openssl=/path/to/openssl`)

I tested the patch
- using mvn on linux
- using make on linux
- using cmake on linux and on windows

In addition, I also added the openssl dependencies to the dev docker image and copied the OpenSSL license to the C client LICENSE file.

Author: Mate Szalay-Beko <szalay.beko.mate@gmail.com>

Reviewers: Enrico Olivelli <eolivelli@apache.org>, Norbert Kalmar <nkalmar@cloudera.com>, Damien Diederen

Closes #1159 from symat/ZOOKEEPER-3630

README_packaging.md

@@ -57,7 +57,13 @@ Optional parameters you might consider when using maven:
 -  `-Pfull-build`         -   activates the full-build profile, causing the C client to be built
 -  `-DskipTests`          -   this parameter will skip both java and C++ unit test execution during the build
 -  `-Pc-test-coverage`    -   activates the test coverage calculation during the execution of C client tests
-
+-  `-Dc-client-openssl`   -   specify ssl support and openssl library location. Default value: `yes`, resulting in 
+                              the autodetection of the openssl library. If the openssl library will not be detected, 
+                              then a warning will be shown and the C client will be compiled without SSL support.
+                              Use `-Dc-client-openssl=no` to explicitly disable SSL feature in C client. Or use 
+                              `-Dc-client-openssl=/path/to/openssl/` if you want to use a non-default / specific 
+                              openssl library location.
+    
 Please note: if you don't provide the `-Pfull-build` parameter, then the C client will not be built, the C client tests
 will not be executed and the previous C client builds will no be cleaned up (e.g. with simply using `mvn clean`).
 
@@ -66,4 +72,5 @@ The compiled C client can be found here:
 - `zookeeper-client/zookeeper-client-c/target/c/lib`                 - Native libraries
 - `zookeeper-client/zookeeper-client-c/target/c/include/zookeeper`   - Native library headers
 
-The same folders gets archived to the `zookeeper-assembly/target/apache-zookeeper-<version>-lib.tar.gz` file, assuming you activated the `full-build` maven profile.
+The same folders gets archived to the `zookeeper-assembly/target/apache-zookeeper-<version>-lib.tar.gz` file, assuming 
+you activated the `full-build` maven profile.

dev/docker/Dockerfile

@@ -20,4 +20,4 @@
 FROM maven:3.6.3-jdk-8
 
 RUN apt-get update
-RUN apt-get install -y g++ cmake autoconf libcppunit-dev libtool
+RUN apt-get install -y g++ cmake autoconf libcppunit-dev libtool openssl libssl-dev

pom.xml

@@ -297,6 +297,10 @@
     <dropwizard.version>3.2.5</dropwizard.version>
     <spotbugsannotations.version>3.1.9</spotbugsannotations.version>
     <checkstyle.version>8.17</checkstyle.version>
+
+    <!-- parameter to pass to C client build -->
+    <c-client-openssl>yes</c-client-openssl>
+
   </properties>
 
   <dependencyManagement>

zookeeper-assembly/src/main/assembly/lib-package.xml

@@ -50,10 +50,11 @@
     </fileSet>
     <fileSet>
       <!-- ZooKeeper license -->
-      <directory>${project.basedir}/..</directory>
+      <directory>${project.basedir}/../zookeeper-client/zookeeper-client-c</directory>
       <includes>
-        <include>LICENSE.txt</include>
+        <include>LICENSE</include>
       </includes>
+      <outputDirectory>/</outputDirectory>
       <fileMode>${rw.file.permission}</fileMode>
       <directoryMode>${rwx.file.permission}</directoryMode>
     </fileSet>

zookeeper-client/zookeeper-client-c/CMakeLists.txt

@@ -182,17 +182,20 @@ target_link_libraries(zookeeper PUBLIC
   $<$<PLATFORM_ID:Linux>:rt> # clock_gettime
   $<$<PLATFORM_ID:Windows>:ws2_32>) # Winsock 2.0
 
-option(WITH_OPENSSL "openssl directory" OFF)
-if(WITH_OPENSSL)
-  target_compile_definitions(zookeeper PUBLIC HAVE_OPENSSL_H)
-  include_directories(${WITH_OPENSSL}/include)
-  link_directories(${WITH_OPENSSL}/lib)
-  if(WIN32)
-    target_link_libraries(zookeeper PUBLIC ssleay32 libeay32)
+option(WITH_OPENSSL "turn ON/OFF SSL support, or define openssl library location (default: ON)" ON)
+message("-- using WITH_OPENSSL=${WITH_OPENSSL}")
+if(NOT WITH_OPENSSL STREQUAL "OFF")
+  if(NOT WITH_OPENSSL STREQUAL "ON")
+    set(OPENSSL_ROOT_DIR,${WITH_OPENSSL})
+  endif()
+  find_package(OpenSSL)
+  if(OPENSSL_FOUND)
+    target_compile_definitions(zookeeper PUBLIC HAVE_OPENSSL_H)
+    target_link_libraries(zookeeper PUBLIC OpenSSL::SSL OpenSSL::Crypto)
+    message("-- OpenSSL libraries found! will build with SSL support.")
   else()
-    target_link_libraries(zookeeper PUBLIC ssl crypto)
+    message("-- WARNING: unable to find OpenSSL libraries! will build without SSL support.")
   endif()
-
 endif()
 
 if(WANT_SYNCAPI AND NOT WIN32)

zookeeper-client/zookeeper-client-c/LICENSE

@@ -200,3 +200,136 @@
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.
+
+
+
+===========================================================================================
+===           The following part contains the dual OpenSSL and SSLeay license           ===
+===           for OpenSSL versions 1.1.1, 1.1.0, 1.0.2 and all prior releases           ===
+===                  (see https://www.openssl.org/source/license.html)                  ===
+===========================================================================================
+
+
+ LICENSE ISSUES
+  ==============
+
+  The OpenSSL toolkit stays under a double license, i.e. both the conditions of
+  the OpenSSL License and the original SSLeay license apply to the toolkit.
+  See below for the actual license texts.
+
+  OpenSSL License
+  ---------------
+
+/* ====================================================================
+ * Copyright (c) 1998-2019 The OpenSSL Project.  All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ *
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in
+ *    the documentation and/or other materials provided with the
+ *    distribution.
+ *
+ * 3. All advertising materials mentioning features or use of this
+ *    software must display the following acknowledgment:
+ *    "This product includes software developed by the OpenSSL Project
+ *    for use in the OpenSSL Toolkit. (http://www.openssl.org/)"
+ *
+ * 4. The names "OpenSSL Toolkit" and "OpenSSL Project" must not be used to
+ *    endorse or promote products derived from this software without
+ *    prior written permission. For written permission, please contact
+ *    openssl-core@openssl.org.
+ *
+ * 5. Products derived from this software may not be called "OpenSSL"
+ *    nor may "OpenSSL" appear in their names without prior written
+ *    permission of the OpenSSL Project.
+ *
+ * 6. Redistributions of any form whatsoever must retain the following
+ *    acknowledgment:
+ *    "This product includes software developed by the OpenSSL Project
+ *    for use in the OpenSSL Toolkit (http://www.openssl.org/)"
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE OpenSSL PROJECT ``AS IS'' AND ANY
+ * EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE OpenSSL PROJECT OR
+ * ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+ * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+ * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
+ * OF THE POSSIBILITY OF SUCH DAMAGE.
+ * ====================================================================
+ *
+ * This product includes cryptographic software written by Eric Young
+ * (eay@cryptsoft.com).  This product includes software written by Tim
+ * Hudson (tjh@cryptsoft.com).
+ *
+ */
+
+ Original SSLeay License
+ -----------------------
+
+/* Copyright (C) 1995-1998 Eric Young (eay@cryptsoft.com)
+ * All rights reserved.
+ *
+ * This package is an SSL implementation written
+ * by Eric Young (eay@cryptsoft.com).
+ * The implementation was written so as to conform with Netscapes SSL.
+ *
+ * This library is free for commercial and non-commercial use as long as
+ * the following conditions are aheared to.  The following conditions
+ * apply to all code found in this distribution, be it the RC4, RSA,
+ * lhash, DES, etc., code; not just the SSL code.  The SSL documentation
+ * included with this distribution is covered by the same copyright terms
+ * except that the holder is Tim Hudson (tjh@cryptsoft.com).
+ *
+ * Copyright remains Eric Young's, and as such any Copyright notices in
+ * the code are not to be removed.
+ * If this package is used in a product, Eric Young should be given attribution
+ * as the author of the parts of the library used.
+ * This can be in the form of a textual message at program startup or
+ * in documentation (online or textual) provided with the package.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ * 3. All advertising materials mentioning features or use of this software
+ *    must display the following acknowledgement:
+ *    "This product includes cryptographic software written by
+ *     Eric Young (eay@cryptsoft.com)"
+ *    The word 'cryptographic' can be left out if the rouines from the library
+ *    being used are not cryptographic related :-).
+ * 4. If you include any Windows specific code (or a derivative thereof) from
+ *    the apps directory (application code) you must include an acknowledgement:
+ *    "This product includes software written by Tim Hudson (tjh@cryptsoft.com)"
+ *
+ * THIS SOFTWARE IS PROVIDED BY ERIC YOUNG ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * The licence and distribution terms for any publically available version or
+ * derivative of this code cannot be changed.  i.e. this code cannot simply be
+ * copied and put under another distribution licence
+ * [including the GNU Public Licence.]
+ */

zookeeper-client/zookeeper-client-c/Makefile.am

@@ -9,7 +9,7 @@ if SOLARIS
 endif
 
 if WANT_OPENSSL
-  OPENSSL_CPPFLAGS = -DHAVE_OPENSSL_H -I$(OPENSSL_DIR)
+  OPENSSL_CPPFLAGS = -DHAVE_OPENSSL_H
   OPENSSL_LIB_LDFLAGS = -lssl -lcrypto
 endif
 

zookeeper-client/zookeeper-client-c/README

@@ -81,6 +81,8 @@ Follow steps 1 and 2 above, and then continue here.
    -DCMAKE_BUILD_TYPE  Debug by default, Release enables optimzation etc.
    -DWANT_SYNCAPI      ON by default, OFF disables the Sync API support
    -DWANT_CPPUNIT      ON except on Windows, OFF disables the tests
+   -DWITH_OPENSSL      ON by default, OFF disables the SSL support. You can also
+                       specify a custom path by -DWITH_OPENSSL=/path/to/openssl/
    -DBUILD_SHARED_LIBS not yet supported, only static libraries are built
    other CMake options see "cmake --help" for generic options, such as generator
 

zookeeper-client/zookeeper-client-c/configure.ac

@@ -38,25 +38,33 @@ else
    CHECK_CPPUNIT(1.10.2)
 fi
 
-AM_CONDITIONAL([WANT_OPENSSL],[test "x$with_openssl" != x])
-
-
+dnl OpenSSL
 AC_ARG_WITH(openssl,
-  AS_HELP_STRING([--without-openssl],
-                 [Do not use Openssl. Default: auto-detect]), [
-case "$with_openssl" in
-  yes|no)
-    : # Nothing special to do here
-    ;;
-  *)
-    if test ! -d "$withval" ; then
-      AC_MSG_ERROR([--with-openssl path does not point to a directory])
-    fi
-       OPENSSL_DIR="$withval"
-    AC_SUBST(OPENSSL_DIR)
-  esac
-])
-AH_TEMPLATE(USE_OPENSSL,[Openssl support is available])
+ [AC_HELP_STRING([--with-openssl[=DIR]], [build with openssl (autodetect openssl library by default) )])],
+ [], [with_openssl=yes])
+AC_MSG_NOTICE([configuring SSL using --with-openssl=$with_openssl])
+saved_CPPFLAGS="$CPPFLAGS"
+saved_LDFLAGS="$LDFLAGS"
+if test "x$with_openssl" != "xno" && test "x$with_openssl" != "xyes" ; then
+        CPPFLAGS="$CPPFLAGS -I$with_openssl/include"
+        LDFLAGS="$LDFLAGS -L$with_openssl/lib"
+fi
+have_openssl=no
+AC_CHECK_HEADER(openssl/ssl.h, [ AC_CHECK_LIB(ssl, SSL_CTX_new, [have_openssl=yes]) ])
+if test "x$with_openssl" != "xno" && test "x$with_openssl" != "xyes" && test "x$have_openssl" != "xyes"; then
+    CPPFLAGS="$saved_CPPFLAGS"
+    LDFLAGS="$saved_LDFLAGS"
+fi
+if test "x$with_openssl" != xno && test "x$have_openssl" = xno; then
+    AC_MSG_WARN([cannot build SSL support -- openssl not found])
+    with_openssl=no
+fi
+if test "x$with_openssl" != xno; then
+    AC_MSG_NOTICE([building with SSL support])
+else
+    AC_MSG_NOTICE([building without SSL support])
+fi
+AM_CONDITIONAL([WANT_OPENSSL],[test "x$with_openssl" != xno])
 
 if test "$CALLER" = "ANT" ; then
 CPPUNIT_CFLAGS="$CPPUNIT_CFLAGS -DZKSERVER_CMD=\"\\\"${base_dir}/zookeeper-client/zookeeper-client-c/tests/zkServer.sh\\\"\""

zookeeper-client/zookeeper-client-c/pom.xml

@@ -88,7 +88,7 @@
                 <CALLER>ANT</CALLER>
               </environmentVariables>
               <arguments>
-                <argument>--with-openssl=/usr/include/openssl/</argument>
+                <argument>--with-openssl=${c-client-openssl}</argument>
                 <argument>--prefix=${project.build.directory}/c</argument>
                 <argument>${c-test-coverage-arg}</argument>
               </arguments>

zookeeper-client/zookeeper-client-c/src/cli.c

@@ -836,8 +836,10 @@ int main(int argc, char **argv) {
                 "OPTIONAL ARGS:\n"
                 "-m, --myid <clientid file>     Path to the file contains the client ID\n"
                 "-c, --cmd <command>            Command to execute, e.g. ls|ls2|create|create2|od|...\n"
+#ifdef HAVE_OPENSSL_H
                 "-s, --ssl <ssl params>         Comma separated parameters to initiate SSL connection\n"
                 "                                 e.g.: server_cert.crt,client_cert.crt,client_priv_key.pem,passwd\n"
+#endif
                 "-r, --readonly                 Connect in read-only mode\n"
                 "-d, --debug                    Activate debug logs right from the beginning (you can also use the \n"
                 "                                 command 'verbose' later to activate debug logs in the cli shell)\n\n",

zookeeper-client/zookeeper-client-c/ssl/gencerts.sh

@@ -25,9 +25,13 @@
 # relative to the canonical path of this script.
 #
 
-# use local fully qualified domain name in the certificates, or fall back
-# to zookeeper.apache.org if no domain name is set or the `hostname` command fails
+
+# determining the domain name in the certificates:
+# - use the first commandline argument, if present
+# - if not, then use the fully qualified domain name
+# - if `hostname` command fails, fall back to zookeeper.apache.org
 FQDN=`hostname -f`
+FQDN=${1:-$FQDN}
 FQDN=${FQDN:-"zookeeper.apache.org"}
 
 # Generate the root key