ZOOKEEPER-3818: client SSL support for zkServer.sh status command

When the ZooKeeper cluster is started in client SSL-only mode (omitting the clientPort
from the zoo.cfg), then the current `zkServer.sh status` command fails to connect to
the server.

**This patch contains:**

- a fix for the zkServer.sh to fall-back to SSL connetion if no unsecure port is defined
- documenting the necessary system properties one needs to define in this case
- some formatting fixes in the `zookeeperTools.md` file to get proper code blocks generated

**Hints for testing:**

you can generate SSL certificate files e.g. by:
```
mkdir -p /tmp/ssl
mkdir -p /tmp/zkdata
cp ./zookeeper-client/zookeeper-client-c/ssl/gencerts.sh /tmp/ssl/
cd /tmp/ssl/
./gencerts.sh localhost
```

then you can change your zoo.cfg:
```
tickTime=3000
initLimit=10
syncLimit=5
dataDir=/tmp/zkdata

secureClientPort=22281
# clientPort=2181
serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
ssl.keyStore.location=/tmp/ssl/server.jks
ssl.keyStore.password=password
ssl.trustStore.location=/tmp/ssl/servertrust.jks
ssl.trustStore.password=password
```

then start ZooKeeper: `./bin/zkServer.sh start-foreground`

then you can run `zkServer.sh status` like:
```
CLIENT_JVMFLAGS="-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty -Dzookeeper.ssl.trustStore.location=/tmp/ssl/clienttrust.jks -Dzookeeper.ssl.trustStore.password=password -Dzookeeper.ssl.keyStore.location=/tmp/ssl/client.jks -Dzookeeper.ssl.keyStore.password=password -Dzookeeper.client.secure=true " ./bin/zkServer.sh status
```

Author: Mate Szalay-Beko <symat@apache.org>

Reviewers: Aishwarya Soni <aishwarya.vsoni@gmail.com>, Norbert Kalmar <nkalmar@apache.org>

Closes #1348 from symat/ZOOKEEPER-3818

bin/zkServer.sh

@@ -235,51 +235,70 @@ restart)
     ;;
 status)
     # -q is necessary on some versions of linux where nc returns too quickly, and no stat result is output
+    isSSL="false"
     clientPortAddress=`$GREP "^[[:space:]]*clientPortAddress[^[:alpha:]]" "$ZOOCFG" | sed -e 's/.*=//'`
     if ! [ $clientPortAddress ]
     then
-	clientPortAddress="localhost"
+	      clientPortAddress="localhost"
     fi
     clientPort=`$GREP "^[[:space:]]*clientPort[^[:alpha:]]" "$ZOOCFG" | sed -e 's/.*=//'`
     if ! [[ "$clientPort"  =~ ^[0-9]+$ ]]
     then
-       dataDir=`$GREP "^[[:space:]]*dataDir" "$ZOOCFG" | sed -e 's/.*=//'`
-       myid=`cat "$dataDir/myid"`
-       if ! [[ "$myid" =~ ^[0-9]+$ ]] ; then
-         echo "clientPort not found and myid could not be determined. Terminating."
-         exit 1
-       fi
-       clientPortAndAddress=`$GREP "^[[:space:]]*server.$myid=.*;.*" "$ZOOCFG" | sed -e 's/.*=//' | sed -e 's/.*;//'`
-       if [ ! "$clientPortAndAddress" ] ; then
-           echo "Client port not found in static config file. Looking in dynamic config file."
-           dynamicConfigFile=`$GREP "^[[:space:]]*dynamicConfigFile" "$ZOOCFG" | sed -e 's/.*=//'`
-           clientPortAndAddress=`$GREP "^[[:space:]]*server.$myid=.*;.*" "$dynamicConfigFile" | sed -e 's/.*=//' | sed -e 's/.*;//'`
-       fi
-       if [ ! "$clientPortAndAddress" ] ; then
-          echo "Client port not found. Terminating."
-          exit 1
-       fi
-       if [[ "$clientPortAndAddress" =~ ^.*:[0-9]+ ]] ; then
-          clientPortAddress=`echo "$clientPortAndAddress" | sed -e 's/:.*//'`
-       fi
-       clientPort=`echo "$clientPortAndAddress" | sed -e 's/.*://'`
-       if [ ! "$clientPort" ] ; then
-          echo "Client port not found. Terminating."
-          exit 1
-       fi
+      dataDir=`$GREP "^[[:space:]]*dataDir" "$ZOOCFG" | sed -e 's/.*=//'`
+      myid=`cat "$dataDir/myid" 2> /dev/null`
+      if ! [[ "$myid" =~ ^[0-9]+$ ]] ; then
+        echo "myid could not be determined, will not able to locate clientPort in the server configs."
+      else
+        clientPortAndAddress=`$GREP "^[[:space:]]*server.$myid=.*;.*" "$ZOOCFG" | sed -e 's/.*=//' | sed -e 's/.*;//'`
+        if [ ! "$clientPortAndAddress" ] ; then
+          echo "Client port not found in static config file. Looking in dynamic config file."
+          dynamicConfigFile=`$GREP "^[[:space:]]*dynamicConfigFile" "$ZOOCFG" | sed -e 's/.*=//'`
+          clientPortAndAddress=`$GREP "^[[:space:]]*server.$myid=.*;.*" "$dynamicConfigFile" | sed -e 's/.*=//' | sed -e 's/.*;//'`
+        fi
+        if [ ! "$clientPortAndAddress" ] ; then
+          echo "Client port not found in the server configs"
+        else
+          if [[ "$clientPortAndAddress" =~ ^.*:[0-9]+ ]] ; then
+            clientPortAddress=`echo "$clientPortAndAddress" | sed -e 's/:.*//'`
+          fi
+          clientPort=`echo "$clientPortAndAddress" | sed -e 's/.*://'`
+        fi
+      fi
     fi
-    echo "Client port found: $clientPort. Client address: $clientPortAddress."
+    if [ ! "$clientPort" ] ; then
+      echo "Client port not found. Looking for secureClientPort in the static config."
+      secureClientPort=`$GREP "^[[:space:]]*secureClientPort[^[:alpha:]]" "$ZOOCFG" | sed -e 's/.*=//'`
+      if [ "$secureClientPort" ] ; then
+        isSSL="true"
+        clientPort=$secureClientPort
+      else
+        echo "Unable to find either secure or unsecure client port in any configs. Terminating."
+        exit 1
+      fi
+    fi
+    echo "Client port found: $clientPort. Client address: $clientPortAddress. Client SSL: $isSSL."
     STAT=`"$JAVA" "-Dzookeeper.log.dir=${ZOO_LOG_DIR}" "-Dzookeeper.root.logger=${ZOO_LOG4J_PROP}" "-Dzookeeper.log.file=${ZOO_LOG_FILE}" \
-             -cp "$CLASSPATH" $JVMFLAGS org.apache.zookeeper.client.FourLetterWordMain \
-             $clientPortAddress $clientPort srvr 2> /dev/null    \
+          -cp "$CLASSPATH" $CLIENT_JVMFLAGS $JVMFLAGS org.apache.zookeeper.client.FourLetterWordMain \
+          $clientPortAddress $clientPort srvr $isSSL 2> /dev/null    \
           | $GREP Mode`
     if [ "x$STAT" = "x" ]
     then
-        echo "Error contacting service. It is probably not running."
-        exit 1
+      if [ "$isSSL" = "true" ] ; then
+        echo " "
+        echo "Note: We used secureClientPort ($secureClientPort) to establish connection, but we failed. The 'status'"
+        echo "  command establishes a client connection to the server to execute diagnostic commands. Please make sure you"
+        echo "  provided all the Client SSL connection related parameters in the CLIENT_JVMFLAGS environment variable! E.g.:"
+        echo "  CLIENT_JVMFLAGS=\"-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty"
+        echo "  -Dzookeeper.ssl.trustStore.location=/tmp/clienttrust.jks -Dzookeeper.ssl.trustStore.password=password"
+        echo "  -Dzookeeper.ssl.keyStore.location=/tmp/client.jks -Dzookeeper.ssl.keyStore.password=password"
+        echo "  -Dzookeeper.client.secure=true\" ./zkServer.sh status"
+        echo " "
+      fi
+      echo "Error contacting service. It is probably not running."
+      exit 1
     else
-        echo $STAT
-        exit 0
+      echo $STAT
+      exit 0
     fi
     ;;
 *)

zookeeper-docs/src/main/resources/markdown/zookeeperAdmin.md

@@ -480,9 +480,11 @@ these options.
 
 ### Monitoring
 
-The ZooKeeper service can be monitored in one of two
-primary ways; 1) the command port through the use of [4 letter words](#sc_zkCommands) and 2) [JMX](zookeeperJMX.html). See the appropriate section for
-your environment/requirements.
+The ZooKeeper service can be monitored in one of three primary ways:
+
+* the command port through the use of [4 letter words](#sc_zkCommands)
+* with [JMX](zookeeperJMX.html)
+* using the [`zkServer.sh status` command](zookeeperTools.html#zkServer)
 
 <a name="sc_logging"></a>
 

zookeeper-docs/src/main/resources/markdown/zookeeperTools.md

@@ -69,6 +69,14 @@ Apache ZooKeeper, version 3.6.0-SNAPSHOT 06/11/2019 05:39 GMT
 
 ```
 
+The `status` command establishes a client connection to the server to execute diagnostic commands. 
+When the ZooKeeper cluster is started in client SSL only mode (by omitting the clientPort
+from the zoo.cfg), then additional SSL related configuration has to be provided before using 
+the `./zkServer.sh status` command to find out if the ZooKeeper server is running. An example:
+
+    CLIENT_JVMFLAGS="-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty -Dzookeeper.ssl.trustStore.location=/tmp/clienttrust.jks -Dzookeeper.ssl.trustStore.password=password -Dzookeeper.ssl.keyStore.location=/tmp/client.jks -Dzookeeper.ssl.keyStore.password=password -Dzookeeper.client.secure=true" ./zkServer.sh status
+
+
 <a name="zkCli"></a>
 
 ### zkCli.sh