ZOOKEEPER-2940: Deal with maxbuffer as it relates to large requests from clients

https://issues.apache.org/jira/browse/ZOOKEEPER-2940

This PR covers the other part of Jute buffer monitoring which relates to client response sizes.

`jute.maxbuffer` is often set too small or the default (4MB) value is not enough to receive large responses from the server which causes the client unable to operate. e.g. `getChildren()` on a node which has lots of children or execution of a large multi.

These new metrics lets operators to monitor the size of generated responses to get an idea on how to properly set client's `jute.maxbuffer`.

The other part of monitoring which relates to proposal size has already been merged https://github.com/apache/zookeeper/pull/415

Author: Andor Molnar <andor@cloudera.com>

Reviewers: phunt@apache.org

Closes #466 from anmolnar/ZOOKEEPER-2940 and squashes the following commits:

29224d021 [Andor Molnar] ZOOKEEPER-2940. Test fixes
486d4961d [Andor Molnar] ZOOKEEPER-2940. Removed support of 4lw commands, because master is targeted
60916e09e [Andor Molnar] ZOOKEEPER-2940. Code review fixes
b39c4c815 [Andor Molnar] ZOOKEEPER-2940. Code review: variable names renamed to proposalStats
87e72ff31 [Andor Molnar] ZOOKEEPER-2940. Some code review cleanups
6f0adb124 [Andor Molnar] ZOOKEEPER-2940. Code review fixes: stats class renamed, added comments to unit test asserts, updated docs
bcb6d4ca5 [Andor Molnar] ZOOKEEPER-2940. Added new metrics to python files
77a5f0aa8 [Andor Molnar] ZOOKEEPER-2940. Fixed failing unit tests
2def61c30 [Andor Molnar] ZOOKEEPER-2940. Track size of client responses

Change-Id: I68df371097cc355b1f41b8dbb0168da4fa5dee43

docs/zookeeperAdmin.html

@@ -638,7 +638,9 @@ server.3=zoo3:2888:3888</pre>
           consists of a single line containing only the text of that machine's
           id. So <span class="codefrag filename">myid</span> of server 1 would contain the text
           "1" and nothing else. The id must be unique within the
-          ensemble and should have a value between 1 and 255.</p>
+          ensemble and should have a value between 1 and 255. <strong>IMPORTANT:</strong> if you
+          enable extended features such as TTL Nodes (see below) the id must be
+          between 1 and 254 due to internal limitations.</p>
         
 </li>
 
@@ -1345,6 +1347,37 @@ server.3=zoo3:2888:3888</pre>
               feature. Default is "true"</p>
 </dd>
 
+          
+<dt>
+<term>zookeeper.extendedTypesEnabled</term>
+</dt>
+<dd>
+<p>(Java system property only: <strong>zookeeper.extendedTypesEnabled</strong>)</p>
+<p>
+<strong>New in 3.5.4, 3.6.0:</strong> Define to "true" to enable
+              extended features such as the creation of <a href="zookeeperProgrammers.html#TTL+Nodes">TTL Nodes</a>.
+              They are disabled by default. IMPORTANT: when enabled server IDs must
+              be less than 255 due to internal limitations.
+              </p>
+</dd>
+
+          
+<dt>
+<term>zookeeper.emulate353TTLNodes</term>
+</dt>
+<dd>
+<p>(Java system property only: <strong>zookeeper.emulate353TTLNodes</strong>)</p>
+<p>
+<strong>New in 3.5.4, 3.6.0:</strong> Due to
+                <a href="https://issues.apache.org/jira/browse/ZOOKEEPER-2901">ZOOKEEPER-2901</a> TTL nodes
+                created in version 3.5.3 are not supported in 3.5.4/3.6.0. However, a workaround is provided via the
+                zookeeper.emulate353TTLNodes system property. If you used TTL nodes in ZooKeeper 3.5.3 and need to maintain
+                compatibility set <strong>zookeeper.emulate353TTLNodes</strong> to "true" in addition to
+                <strong>zookeeper.extendedTypesEnabled</strong>. NOTE: due to the bug, server IDs
+                must be 127 or less. Additionally, the maximum support TTL value is 1099511627775 which is smaller
+                than what was allowed in 3.5.3 (1152921504606846975)</p>
+</dd>
+
         
 </dl>
 <a name="sc_clusterOptions"></a>
@@ -2216,6 +2249,7 @@ server.3=zoo3:2888:3888</pre>
               zk_min_latency  0
               zk_packets_received 70
               zk_packets_sent 69
+              zk_num_alive_connections	1
               zk_outstanding_requests 0
               zk_server_state leader
               zk_znode_count   4
@@ -2227,6 +2261,9 @@ server.3=zoo3:2888:3888</pre>
               zk_pending_syncs    0               - only exposed by the Leader
               zk_open_file_descriptor_count 23    - only available on Unix platforms
               zk_max_file_descriptor_count 1024   - only available on Unix platforms
+              zk_last_proposal_size 23
+              zk_min_proposal_size 23
+              zk_max_proposal_size 64
               </pre>
 <p>The output is compatible with java properties format and the content 
               may change over time (new keys added). Your scripts should expect changes.</p>

docs/zookeeperProgrammers.html

@@ -596,6 +596,10 @@ document.write("Last Published: " + document.lastModified);
           you can optionally set a TTL in milliseconds for the znode. If the znode
           is not modified within the TTL and has no children it will become a candidate
           to be deleted by the server at some point in the future.</p>
+<p>Note: TTL Nodes must be enabled via System property as
+        they are disabled by default. See the <a href="zookeeperAdmin.html#sc_configuration">Administrator's
+        Guide</a> for details. If you attempt to create TTL Nodes without the proper System property set the server
+        will throw <em>KeeperException.UnimplementedException</em>.</p>
 <a name="sc_timeInZk"></a>
 <h3 class="h4">Time in ZooKeeper</h3>
 <p>ZooKeeper tracks time multiple ways:</p>

src/contrib/monitoring/check_zookeeper.py

@@ -293,7 +293,14 @@ class ZooKeeperServer(object):
                 result['zk_zxid_epoch']   = int(m.group(1), 16) >>32 # high 32 bits
                 continue
 
-        return result 
+            m = re.match('Proposal sizes last/min/max: (\d+)/(\d+)/(\d+)', line)
+            if m is not None:
+                result['zk_last_proposal_size'] = int(m.group(1))
+                result['zk_min_proposal_size'] = int(m.group(2))
+                result['zk_max_proposal_size'] = int(m.group(3))
+                continue
+
+        return result
 
     def _parse_line(self, line):
         try:

src/contrib/monitoring/ganglia/zookeeper.pyconf

@@ -46,5 +46,8 @@ collection_group {
   metric { name = "zk_followers" }
   metric { name = "zk_synced_followers" }
   metric { name = "zk_pending_syncs" }
+  metric { name = "zk_last_proposal_size" }
+  metric { name = "zk_min_proposal_size" }
+  metric { name = "zk_max_proposal_size" }
 }
 

src/contrib/monitoring/ganglia/zookeeper_ganglia.py

@@ -214,7 +214,10 @@ def metric_init(params=None):
         'zk_max_file_descriptor_count': {'units': 'descriptors'},
         'zk_followers': {'units': 'nodes'},
         'zk_synced_followers': {'units': 'nodes'},
-        'zk_pending_syncs': {'units': 'syncs'}
+        'zk_pending_syncs': {'units': 'syncs'},
+        'zk_last_proposal_size': {'units': 'bytes'},
+        'zk_min_proposal_size': {'units': 'bytes'},
+        'zk_max_proposal_size': {'units': 'bytes'}
     }
     metric_handler.descriptors = {}
     for name, updates in metrics.iteritems():

src/java/main/org/apache/zookeeper/ClientCnxnSocket.java

@@ -115,7 +115,7 @@ abstract class ClientCnxnSocket {
         this.lastHeard = now;
     }
 
-    protected void readLength() throws IOException {
+    void readLength() throws IOException {
         int len = incomingBuffer.getInt();
         if (len < 0 || len >= packetLen) {
             throw new IOException("Packet len " + len + " is out of range!");

src/java/main/org/apache/zookeeper/server/NIOServerCnxn.java

@@ -678,8 +678,6 @@ public class NIOServerCnxn extends ServerCnxn {
         }
     }
 
-    private final static byte fourBytes[] = new byte[4];
-
     /*
      * (non-Javadoc)
      *
@@ -689,23 +687,7 @@ public class NIOServerCnxn extends ServerCnxn {
     @Override
     public void sendResponse(ReplyHeader h, Record r, String tag) {
         try {
-            ByteArrayOutputStream baos = new ByteArrayOutputStream();
-            // Make space for length
-            BinaryOutputArchive bos = BinaryOutputArchive.getArchive(baos);
-            try {
-                baos.write(fourBytes);
-                bos.writeRecord(h, "header");
-                if (r != null) {
-                    bos.writeRecord(r, tag);
-                }
-                baos.close();
-            } catch (IOException e) {
-                LOG.error("Error serializing response");
-            }
-            byte b[] = baos.toByteArray();
-            ByteBuffer bb = ByteBuffer.wrap(b);
-            bb.putInt(b.length - 4).rewind();
-            sendBuffer(bb);
+            super.sendResponse(h, r, tag);
             if (h.getXid() > 0) {
                 // check throttling
                 if (outstandingRequests.decrementAndGet() < 1 ||

src/java/main/org/apache/zookeeper/server/NettyServerCnxn.java

@@ -160,7 +160,6 @@ public class NettyServerCnxn extends ServerCnxn {
         }
     }
 
-    private static final byte[] fourBytes = new byte[4];
     static class ResumeMessageEvent implements MessageEvent {
         Channel channel;
         ResumeMessageEvent(Channel channel) {
@@ -182,23 +181,7 @@ public class NettyServerCnxn extends ServerCnxn {
         if (closingChannel || !channel.isOpen()) {
             return;
         }
-        ByteArrayOutputStream baos = new ByteArrayOutputStream();
-        // Make space for length
-        BinaryOutputArchive bos = BinaryOutputArchive.getArchive(baos);
-        try {
-            baos.write(fourBytes);
-            bos.writeRecord(h, "header");
-            if (r != null) {
-                bos.writeRecord(r, tag);
-            }
-            baos.close();
-        } catch (IOException e) {
-            LOG.error("Error serializing response");
-        }
-        byte b[] = baos.toByteArray();
-        ByteBuffer bb = ByteBuffer.wrap(b);
-        bb.putInt(b.length - 4).rewind();
-        sendBuffer(bb);
+        super.sendResponse(h, r, tag);
         if (h.getXid() > 0) {
             // zks cannot be null otherwise we would not have gotten here!
             if (!zkServer.shouldThrottle(outstandingCount.decrementAndGet())) {

src/java/main/org/apache/zookeeper/server/ServerCnxn.java

@@ -18,6 +18,7 @@
 
 package org.apache.zookeeper.server;
 
+import java.io.ByteArrayOutputStream;
 import java.io.IOException;
 import java.io.PrintWriter;
 import java.io.StringWriter;
@@ -34,6 +35,7 @@ import java.util.Set;
 import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.atomic.AtomicLong;
 
+import org.apache.jute.BinaryOutputArchive;
 import org.apache.jute.Record;
 import org.apache.zookeeper.WatchedEvent;
 import org.apache.zookeeper.Watcher;
@@ -55,6 +57,8 @@ public abstract class ServerCnxn implements Stats, Watcher {
     
     private Set<Id> authInfo = Collections.newSetFromMap(new ConcurrentHashMap<Id, Boolean>());
 
+    private static final byte[] fourBytes = new byte[4];
+
     /**
      * If the client is of old version, we don't send r-o mode info to it.
      * The reason is that if we would, old C client doesn't read it, which
@@ -66,8 +70,26 @@ public abstract class ServerCnxn implements Stats, Watcher {
 
     abstract void close();
 
-    public abstract void sendResponse(ReplyHeader h, Record r, String tag)
-        throws IOException;
+    public void sendResponse(ReplyHeader h, Record r, String tag) throws IOException {
+        ByteArrayOutputStream baos = new ByteArrayOutputStream();
+        // Make space for length
+        BinaryOutputArchive bos = BinaryOutputArchive.getArchive(baos);
+        try {
+            baos.write(fourBytes);
+            bos.writeRecord(h, "header");
+            if (r != null) {
+                bos.writeRecord(r, tag);
+            }
+            baos.close();
+        } catch (IOException e) {
+            LOG.error("Error serializing response");
+        }
+        byte b[] = baos.toByteArray();
+        serverStats().updateClientResponseSize(b.length - 4);
+        ByteBuffer bb = ByteBuffer.wrap(b);
+        bb.putInt(b.length - 4).rewind();
+        sendBuffer(bb);
+    }
 
     /* notify the client the session is closing and close/cleanup socket */
     abstract void sendCloseSession();
@@ -133,12 +155,12 @@ public abstract class ServerCnxn implements Stats, Watcher {
         incrPacketsSent();
         ServerStats serverStats = serverStats();
         if (serverStats != null) {
-            serverStats().incrementPacketsSent();
+            serverStats.incrementPacketsSent();
         }
     }
 
     protected abstract ServerStats serverStats();
-    
+
     protected final Date established = new Date();
 
     protected final AtomicLong packetsReceived = new AtomicLong();

src/java/main/org/apache/zookeeper/server/ServerStats.java

@@ -21,6 +21,9 @@ package org.apache.zookeeper.server;
 
 
 import org.apache.zookeeper.common.Time;
+import org.apache.zookeeper.server.quorum.BufferStats;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 import java.util.concurrent.atomic.AtomicLong;
 
@@ -28,6 +31,8 @@ import java.util.concurrent.atomic.AtomicLong;
  * Basic Server Statistics
  */
 public class ServerStats {
+    private static final Logger LOG = LoggerFactory.getLogger(ServerStats.class);
+
     private long packetsSent;
     private long packetsReceived;
     private long maxLatency;
@@ -36,6 +41,8 @@ public class ServerStats {
     private long count = 0;
     private AtomicLong fsyncThresholdExceedCount = new AtomicLong(0);
 
+    private final BufferStats clientResponseStats = new BufferStats();
+
     private final Provider provider;
 
     public interface Provider {
@@ -167,6 +174,14 @@ public class ServerStats {
     synchronized public void reset() {
         resetLatency();
         resetRequestCounters();
+        clientResponseStats.reset();
     }
 
+    public void updateClientResponseSize(int size) {
+        clientResponseStats.setLastBufferSize(size);
+    }
+
+    public BufferStats getClientResponseStats() {
+        return clientResponseStats;
+    }
 }

src/java/main/org/apache/zookeeper/server/ZooKeeperServerBean.java

@@ -182,4 +182,19 @@ public class ZooKeeperServerBean implements ZooKeeperServerMXBean, ZKMBeanInfo {
     public int getJuteMaxBufferSize() {
         return BinaryInputArchive.maxBuffer;
     }
+
+    @Override
+    public int getLastClientResponseSize() {
+        return zks.serverStats().getClientResponseStats().getLastBufferSize();
+    }
+
+    @Override
+    public int getMinClientResponseSize() {
+        return zks.serverStats().getClientResponseStats().getMinBufferSize();
+    }
+
+    @Override
+    public int getMaxClientResponseSize() {
+        return zks.serverStats().getClientResponseStats().getMaxBufferSize();
+    }
 }

src/java/main/org/apache/zookeeper/server/ZooKeeperServerMXBean.java

@@ -143,4 +143,19 @@ public interface ZooKeeperServerMXBean {
      * @return Returns the value of the following config setting: jute.maxbuffer
      */
     public int getJuteMaxBufferSize();
+
+    /**
+     * @return size of latest generated client response
+     */
+    public int getLastClientResponseSize();
+
+    /**
+     * @return size of smallest generated client response
+     */
+    public int getMinClientResponseSize();
+
+    /**
+     * @return size of largest generated client response
+     */
+    public int getMaxClientResponseSize();
 }

src/java/main/org/apache/zookeeper/server/admin/Commands.java

@@ -330,12 +330,20 @@ public class Commands {
             response.put("open_file_descriptor_count", osMbean.getOpenFileDescriptorCount());
             response.put("max_file_descriptor_count", osMbean.getMaxFileDescriptorCount());
 
+            response.put("last_client_response_size", stats.getClientResponseStats().getLastBufferSize());
+            response.put("max_client_response_size", stats.getClientResponseStats().getMaxBufferSize());
+            response.put("min_client_response_size", stats.getClientResponseStats().getMinBufferSize());
+
             if (zkServer instanceof LeaderZooKeeperServer) {
                 Leader leader = ((LeaderZooKeeperServer) zkServer).getLeader();
 
                 response.put("followers", leader.getLearners().size());
                 response.put("synced_followers", leader.getForwardingFollowers().size());
                 response.put("pending_syncs", leader.getNumPendingSyncs());
+
+                response.put("last_proposal_size", leader.getProposalStats().getLastBufferSize());
+                response.put("max_proposal_size", leader.getProposalStats().getMaxBufferSize());
+                response.put("min_proposal_size", leader.getProposalStats().getMinBufferSize());
             }
 
             return response;
@@ -415,9 +423,13 @@ public class Commands {
             response.put("version", Version.getFullVersion());
             response.put("read_only", zkServer instanceof ReadOnlyZooKeeperServer);
             response.put("server_stats", zkServer.serverStats());
+            response.put("client_response", zkServer.serverStats().getClientResponseStats());
+            if (zkServer instanceof LeaderZooKeeperServer) {
+                Leader leader = ((LeaderZooKeeperServer)zkServer).getLeader();
+                response.put("proposal_stats", leader.getProposalStats());
+            }
             response.put("node_count", zkServer.getZKDatabase().getNodeCount());
             return response;
-
         }
     }
 

src/java/main/org/apache/zookeeper/server/command/MonitorCommand.java

@@ -75,9 +75,9 @@ public class MonitorCommand extends AbstractFourLetterCommand {
             print("synced_followers", leader.getForwardingFollowers().size());
             print("pending_syncs", leader.getNumPendingSyncs());
 
-            print("last_proposal_size", leader.getProposalStats().getLastProposalSize());
-            print("max_proposal_size", leader.getProposalStats().getMaxProposalSize());
-            print("min_proposal_size", leader.getProposalStats().getMinProposalSize());
+            print("last_proposal_size", leader.getProposalStats().getLastBufferSize());
+            print("max_proposal_size", leader.getProposalStats().getMaxBufferSize());
+            print("min_proposal_size", leader.getProposalStats().getMinBufferSize());
         }
     }
 

src/java/main/org/apache/zookeeper/server/command/StatCommand.java

@@ -25,7 +25,7 @@ import org.apache.zookeeper.server.ServerCnxn;
 import org.apache.zookeeper.server.ServerStats;
 import org.apache.zookeeper.server.quorum.Leader;
 import org.apache.zookeeper.server.quorum.LeaderZooKeeperServer;
-import org.apache.zookeeper.server.quorum.ProposalStats;
+import org.apache.zookeeper.server.quorum.BufferStats;
 import org.apache.zookeeper.server.quorum.ReadOnlyZooKeeperServer;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -64,7 +64,7 @@ public class StatCommand extends AbstractFourLetterCommand {
             pw.println(zkServer.getZKDatabase().getNodeCount());
             if (serverStats.getServerState().equals("leader")) {
                 Leader leader = ((LeaderZooKeeperServer)zkServer).getLeader();
-                ProposalStats proposalStats = leader.getProposalStats();
+                BufferStats proposalStats = leader.getProposalStats();
                 pw.printf("Proposal sizes last/min/max: %s%n", proposalStats.toString());
             }
         }

src/java/main/org/apache/zookeeper/server/quorum/BufferStats.java

@@ -0,0 +1,89 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server.quorum;
+
+/**
+ * Provides live statistics about Jute buffer usage in term of proposal and client request size.
+ */
+public class BufferStats {
+    public static final int INIT_VALUE = -1;
+
+    /**
+     * Size of the last buffer usage.
+     */
+    private int lastBufferSize = INIT_VALUE;
+
+    /**
+     * Size of the smallest buffer usage.
+     */
+    private int minBufferSize = INIT_VALUE;
+
+    /**
+     * Size of the largest buffer usage.
+     */
+    private int maxBufferSize = INIT_VALUE;
+
+    /**
+     * Size of the last buffer usage.
+     */
+    public synchronized int getLastBufferSize() {
+        return lastBufferSize;
+    }
+
+    /**
+     * Updates statistics by setting the last buffer usage size.
+     */
+    public synchronized void setLastBufferSize(int value) {
+        lastBufferSize = value;
+        if (minBufferSize == INIT_VALUE || value < minBufferSize) {
+            minBufferSize = value;
+        }
+        if (value > maxBufferSize) {
+            maxBufferSize = value;
+        }
+    }
+
+    /**
+     * Size of the smallest buffer usage.
+     */
+    public synchronized int getMinBufferSize() {
+        return minBufferSize;
+    }
+
+    /**
+     * Size of the largest buffer usage.
+     */
+    public synchronized int getMaxBufferSize() {
+        return maxBufferSize;
+    }
+
+    /**
+     * Reset statistics.
+     */
+    public synchronized void reset() {
+        lastBufferSize = INIT_VALUE;
+        minBufferSize = INIT_VALUE;
+        maxBufferSize = INIT_VALUE;
+    }
+
+    @Override
+    public synchronized String toString() {
+        return String.format("%d/%d/%d", lastBufferSize, minBufferSize, maxBufferSize);
+    }
+}

src/java/main/org/apache/zookeeper/server/quorum/Leader.java

@@ -18,7 +18,6 @@
 
 package org.apache.zookeeper.server.quorum;
 
-import java.io.ByteArrayOutputStream;
 import java.io.BufferedInputStream;
 import java.io.IOException;
 import java.net.BindException;
@@ -42,7 +41,6 @@ import java.util.concurrent.ConcurrentMap;
 
 import javax.security.sasl.SaslException;
 
-import org.apache.jute.BinaryOutputArchive;
 import org.apache.zookeeper.ZooDefs.OpCode;
 import org.apache.zookeeper.common.Time;
 import org.apache.zookeeper.server.FinalRequestProcessor;
@@ -107,9 +105,9 @@ public class Leader {
     private final HashSet<LearnerHandler> learners =
         new HashSet<LearnerHandler>();
 
-    private final ProposalStats proposalStats;
+    private final BufferStats proposalStats;
 
-    public ProposalStats getProposalStats() {
+    public BufferStats getProposalStats() {
         return proposalStats;
     }
 
@@ -230,7 +228,7 @@ public class Leader {
 
     Leader(QuorumPeer self,LeaderZooKeeperServer zk) throws IOException {
         this.self = self;
-        this.proposalStats = new ProposalStats();
+        this.proposalStats = new BufferStats();
         try {
             if (self.getQuorumListenOnAllIPs()) {
                 ss = new ServerSocket(self.getQuorumAddress().getPort());
@@ -1061,7 +1059,7 @@ public class Leader {
         }
 
         byte[] data = SerializeUtils.serializeRequest(request);
-        proposalStats.setLastProposalSize(data.length);
+        proposalStats.setLastBufferSize(data.length);
         QuorumPacket pp = new QuorumPacket(Leader.PROPOSAL, request.zxid, data, null);
 
         Proposal p = new Proposal();

src/java/main/org/apache/zookeeper/server/quorum/LeaderBean.java

@@ -55,17 +55,17 @@ public class LeaderBean extends ZooKeeperServerBean implements LeaderMXBean {
 
     @Override
     public int getLastProposalSize() {
-        return leader.getProposalStats().getLastProposalSize();
+        return leader.getProposalStats().getLastBufferSize();
     }
 
     @Override
     public int getMinProposalSize() {
-        return leader.getProposalStats().getMinProposalSize();
+        return leader.getProposalStats().getMinBufferSize();
     }
 
     @Override
     public int getMaxProposalSize() {
-        return leader.getProposalStats().getMaxProposalSize();
+        return leader.getProposalStats().getMaxBufferSize();
     }
 
     @Override

src/java/main/org/apache/zookeeper/server/quorum/ProposalStats.java

@@ -1,71 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.zookeeper.server.quorum;
-
-/**
- * Provides live statistics about a running Leader.
- */
-public class ProposalStats {
-    /**
-     * Size of the last generated proposal. This should fit into server's jute.maxbuffer setting.
-     */
-    private int lastProposalSize = -1;
-
-    /**
-     * Size of the smallest proposal which has been generated since the server was started.
-     */
-    private int minProposalSize = -1;
-
-    /**
-     * Size of the largest proposal which has been generated since the server was started.
-     */
-    private int maxProposalSize = -1;
-
-    public synchronized int getLastProposalSize() {
-        return lastProposalSize;
-    }
-
-    synchronized void setLastProposalSize(int value) {
-        lastProposalSize = value;
-        if (minProposalSize == -1 || value < minProposalSize) {
-            minProposalSize = value;
-        }
-        if (value > maxProposalSize) {
-            maxProposalSize = value;
-        }
-    }
-
-    public synchronized int getMinProposalSize() {
-        return minProposalSize;
-    }
-
-    public synchronized int getMaxProposalSize() {
-        return maxProposalSize;
-    }
-
-    public synchronized void reset() {
-        lastProposalSize = -1;
-        minProposalSize = -1;
-        maxProposalSize = -1;
-    }
-
-    public synchronized String toString() {
-        return String.format("%d/%d/%d", lastProposalSize, minProposalSize, maxProposalSize);
-    }
-}

src/java/test/org/apache/zookeeper/server/NIOServerCnxnTest.java

@@ -24,12 +24,19 @@ import org.apache.zookeeper.CreateMode;
 import org.apache.zookeeper.KeeperException;
 import org.apache.zookeeper.ZooDefs.Ids;
 import org.apache.zookeeper.ZooKeeper;
+import org.apache.zookeeper.server.quorum.BufferStats;
 import org.apache.zookeeper.test.ClientBase;
 import org.junit.Assert;
 import org.junit.Test;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.greaterThan;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertThat;
+
 public class NIOServerCnxnTest extends ClientBase {
     private static final Logger LOG = LoggerFactory
                         .getLogger(NIOServerCnxnTest.class);
@@ -47,7 +54,7 @@ public class NIOServerCnxnTest extends ClientBase {
             // make sure zkclient works
             zk.create(path, "test".getBytes(), Ids.OPEN_ACL_UNSAFE,
                     CreateMode.PERSISTENT);
-            Assert.assertNotNull("Didn't create znode:" + path,
+            assertNotNull("Didn't create znode:" + path,
                     zk.exists(path, false));
             // Defaults ServerCnxnFactory would be instantiated with
             // NIOServerCnxnFactory
@@ -68,5 +75,21 @@ public class NIOServerCnxnTest extends ClientBase {
         } finally {
             zk.close();
         }
+
+    }
+
+    @Test
+    public void testClientResponseStatsUpdate() throws IOException, InterruptedException, KeeperException {
+        try (ZooKeeper zk = createClient()) {
+            BufferStats clientResponseStats = serverFactory.getZooKeeperServer().serverStats().getClientResponseStats();
+            assertThat("Last client response size should be initialized with INIT_VALUE",
+                    clientResponseStats.getLastBufferSize(), equalTo(BufferStats.INIT_VALUE));
+
+            zk.create("/a", "test".getBytes(), Ids.OPEN_ACL_UNSAFE,
+                    CreateMode.PERSISTENT);
+
+            assertThat("Last client response size should be greater then zero after client request was performed",
+                    clientResponseStats.getLastBufferSize(), greaterThan(0));
+        }
     }
 }

src/java/test/org/apache/zookeeper/server/NettyServerCnxnTest.java

@@ -20,14 +20,22 @@ package org.apache.zookeeper.server;
 
 
 import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.KeeperException;
 import org.apache.zookeeper.ZooDefs.Ids;
 import org.apache.zookeeper.ZooKeeper;
+import org.apache.zookeeper.server.quorum.BufferStats;
 import org.apache.zookeeper.test.ClientBase;
 import org.junit.Assert;
 import org.junit.Test;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+import java.io.IOException;
+
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.greaterThan;
+import static org.junit.Assert.assertThat;
+
 /**
  * Test verifies the behavior of NettyServerCnxn which represents a connection
  * from a client to the server.
@@ -84,4 +92,19 @@ public class NettyServerCnxnTest extends ClientBase {
             zk.close();
         }
     }
+
+    @Test
+    public void testClientResponseStatsUpdate() throws IOException, InterruptedException, KeeperException {
+        try (ZooKeeper zk = createClient()) {
+            BufferStats clientResponseStats = serverFactory.getZooKeeperServer().serverStats().getClientResponseStats();
+            assertThat("Last client response size should be initialized with INIT_VALUE",
+                    clientResponseStats.getLastBufferSize(), equalTo(BufferStats.INIT_VALUE));
+
+            zk.create("/a", "test".getBytes(), Ids.OPEN_ACL_UNSAFE,
+                    CreateMode.PERSISTENT);
+
+            assertThat("Last client response size should be greater than 0 after client request was performed",
+                    clientResponseStats.getLastBufferSize(), greaterThan(0));
+        }
+    }
 }

src/java/test/org/apache/zookeeper/server/admin/CommandsTest.java

@@ -23,11 +23,13 @@ import static org.junit.Assert.assertNull;
 import static org.junit.Assert.assertTrue;
 
 import java.io.IOException;
+import java.nio.Buffer;
 import java.util.HashMap;
 import java.util.Map;
 
 import org.apache.zookeeper.server.ServerStats;
 import org.apache.zookeeper.server.ZooKeeperServer;
+import org.apache.zookeeper.server.quorum.BufferStats;
 import org.apache.zookeeper.test.ClientBase;
 import org.junit.Test;
 
@@ -173,7 +175,10 @@ public class CommandsTest extends ClientBase {
                     new Field("ephemerals_count", Integer.class),
                     new Field("approximate_data_size", Long.class),
                     new Field("open_file_descriptor_count", Long.class),
-                    new Field("max_file_descriptor_count", Long.class));
+                    new Field("max_file_descriptor_count", Long.class),
+                    new Field("last_client_response_size", Integer.class),
+                    new Field("max_client_response_size", Integer.class),
+                    new Field("min_client_response_size", Integer.class));
     }
 
     @Test
@@ -187,7 +192,8 @@ public class CommandsTest extends ClientBase {
                 new Field("version", String.class),
                 new Field("read_only", Boolean.class),
                 new Field("server_stats", ServerStats.class),
-                new Field("node_count", Integer.class));
+                new Field("node_count", Integer.class),
+                new Field("client_response", BufferStats.class));
     }
 
     @Test
@@ -205,7 +211,8 @@ public class CommandsTest extends ClientBase {
                     new Field("read_only", Boolean.class),
                     new Field("server_stats", ServerStats.class),
                     new Field("node_count", Integer.class),
-                    new Field("connections", Iterable.class));
+                    new Field("connections", Iterable.class),
+                    new Field("client_response", BufferStats.class));
     }
 
     @Test

src/java/test/org/apache/zookeeper/server/quorum/BufferStatsTest.java

@@ -0,0 +1,58 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server.quorum;
+
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+
+public class BufferStatsTest {
+    @Test
+    public void testSetProposalSizeSetMinMax() {
+        BufferStats stats = new BufferStats();
+        assertEquals(-1, stats.getLastBufferSize());
+        assertEquals(-1, stats.getMinBufferSize());
+        assertEquals(-1, stats.getMaxBufferSize());
+        stats.setLastBufferSize(10);
+        assertEquals(10, stats.getLastBufferSize());
+        assertEquals(10, stats.getMinBufferSize());
+        assertEquals(10, stats.getMaxBufferSize());
+        stats.setLastBufferSize(20);
+        assertEquals(20, stats.getLastBufferSize());
+        assertEquals(10, stats.getMinBufferSize());
+        assertEquals(20, stats.getMaxBufferSize());
+        stats.setLastBufferSize(5);
+        assertEquals(5, stats.getLastBufferSize());
+        assertEquals(5, stats.getMinBufferSize());
+        assertEquals(20, stats.getMaxBufferSize());
+    }
+
+    @Test
+    public void testReset() {
+        BufferStats stats = new BufferStats();
+        stats.setLastBufferSize(10);
+        assertEquals(10, stats.getLastBufferSize());
+        assertEquals(10, stats.getMinBufferSize());
+        assertEquals(10, stats.getMaxBufferSize());
+        stats.reset();
+        assertEquals(-1, stats.getLastBufferSize());
+        assertEquals(-1, stats.getMinBufferSize());
+        assertEquals(-1, stats.getMaxBufferSize());
+    }
+}

src/java/test/org/apache/zookeeper/server/quorum/ProposalStatsTest.java

@@ -1,58 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.zookeeper.server.quorum;
-
-import org.junit.Test;
-
-import static org.junit.Assert.assertEquals;
-
-public class ProposalStatsTest {
-    @Test
-    public void testSetProposalSizeSetMinMax() {
-        ProposalStats stats = new ProposalStats();
-        assertEquals(-1, stats.getLastProposalSize());
-        assertEquals(-1, stats.getMinProposalSize());
-        assertEquals(-1, stats.getMaxProposalSize());
-        stats.setLastProposalSize(10);
-        assertEquals(10, stats.getLastProposalSize());
-        assertEquals(10, stats.getMinProposalSize());
-        assertEquals(10, stats.getMaxProposalSize());
-        stats.setLastProposalSize(20);
-        assertEquals(20, stats.getLastProposalSize());
-        assertEquals(10, stats.getMinProposalSize());
-        assertEquals(20, stats.getMaxProposalSize());
-        stats.setLastProposalSize(5);
-        assertEquals(5, stats.getLastProposalSize());
-        assertEquals(5, stats.getMinProposalSize());
-        assertEquals(20, stats.getMaxProposalSize());
-    }
-
-    @Test
-    public void testReset() {
-        ProposalStats stats = new ProposalStats();
-        stats.setLastProposalSize(10);
-        assertEquals(10, stats.getLastProposalSize());
-        assertEquals(10, stats.getMinProposalSize());
-        assertEquals(10, stats.getMaxProposalSize());
-        stats.reset();
-        assertEquals(-1, stats.getLastProposalSize());
-        assertEquals(-1, stats.getMinProposalSize());
-        assertEquals(-1, stats.getMaxProposalSize());
-    }
-}

src/java/test/org/apache/zookeeper/server/quorum/StatCommandTest.java

@@ -27,7 +27,6 @@ import org.apache.zookeeper.server.command.StatCommand;
 import org.junit.Before;
 import org.junit.Test;
 
-import java.io.IOException;
 import java.io.PrintWriter;
 import java.io.StringWriter;
 import java.util.ArrayList;
@@ -44,7 +43,7 @@ public class StatCommandTest {
     private ServerStats.Provider providerMock;
 
     @Before
-    public void setUp() throws IOException {
+    public void setUp() {
         outputWriter = new StringWriter();
         ServerCnxn serverCnxnMock = mock(ServerCnxn.class);
 
@@ -55,7 +54,7 @@ public class StatCommandTest {
         ZKDatabase zkDatabaseMock = mock(ZKDatabase.class);
         when(zks.getZKDatabase()).thenReturn(zkDatabaseMock);
         Leader leaderMock = mock(Leader.class);
-        when(leaderMock.getProposalStats()).thenReturn(new ProposalStats());
+        when(leaderMock.getProposalStats()).thenReturn(new BufferStats());
         when(zks.getLeader()).thenReturn(leaderMock);
 
         ServerCnxnFactory serverCnxnFactory = mock(ServerCnxnFactory.class);

src/java/test/org/apache/zookeeper/server/quorum/StatResetCommandTest.java

@@ -96,8 +96,8 @@ public class StatResetCommandTest {
 
         when(serverStats.getServerState()).thenReturn("leader");
 
-        ProposalStats proposalStats = mock(ProposalStats.class);
-        when(leader.getProposalStats()).thenReturn(proposalStats);
+        BufferStats bufferStats = mock(BufferStats.class);
+        when(leader.getProposalStats()).thenReturn(bufferStats);
 
         // Act
         statResetCommand.commandRun();
@@ -106,6 +106,6 @@ public class StatResetCommandTest {
         String output = outputWriter.toString();
         assertEquals("Server stats reset.\n", output);
         verify(serverStats, times(1)).reset();
-        verify(proposalStats, times(1)).reset();
+        verify(bufferStats, times(1)).reset();
     }
 }

zookeeper-docs/src/documentation/content/xdocs/zookeeperAdmin.xml

@@ -1933,6 +1933,9 @@ server.3=zoo3:2888:3888</programlisting>
               zk_pending_syncs    0               - only exposed by the Leader
               zk_open_file_descriptor_count 23    - only available on Unix platforms
               zk_max_file_descriptor_count 1024   - only available on Unix platforms
+              zk_last_proposal_size 23
+              zk_min_proposal_size 23
+              zk_max_proposal_size 64
               </programlisting>
 
               <para>The output is compatible with java properties format and the content