ZOOKEEPER-3601: introduce the fault injection framework: Byteman for ZooKeeper

- **Byteman** is a powerful tool to inject faults during runtime, especially for distributed system
- In the future, we can also introduce it into our unit test infra, just as hadoop, cassandra did. [PR-123](https://github.com/apache/zookeeper/pull/123) is a good starting.
- more details in the [ZOOKEEPER-3601](https://issues.apache.org/jira/browse/ZOOKEEPER-3601)

Author: maoling <maoling199210191@sina.com>

Reviewers: Andor Molnar <anmolnar@apache.org>

Closes #1135 from maoling/ZOOKEEPER-3601

zookeeper-docs/src/main/resources/markdown/zookeeperTools.md

@@ -30,6 +30,8 @@ limitations under the License.
     * [zk-smoketest](#zk-smoketest)
     
 * [Testing](#Testing)
+    * [Fault Injection Framework](#fault-injection)
+        * [Byteman](#Byteman)
     * [Jepsen Test](#jepsen-test)
     
 <a name="Scripts"></a>
@@ -526,6 +528,87 @@ existing installations. More details are [here](https://github.com/phunt/zk-smok
 
 ## Testing
 
+<a name="fault-injection"></a>
+
+### Fault Injection Framework
+
+<a name="Byteman"></a>
+
+#### Byteman
+
+- **Byteman** is a tool which makes it easy to trace, monitor and test the behaviour of Java application and JDK runtime code.
+It injects Java code into your application methods or into Java runtime methods without the need for you to recompile, repackage or even redeploy your application.
+Injection can be performed at JVM startup or after startup while the application is still running.
+- Visit the official [website](https://byteman.jboss.org/) to download the latest release
+- A brief tutorial can be found [here](https://developer.jboss.org/wiki/ABytemanTutorial)
+
+    ```bash
+    Preparations:
+    # attach the byteman to 3 zk servers during runtime
+    # 55001,55002,55003 is byteman binding port; 714,740,758 is the zk server pid
+    ./bminstall.sh -b -Dorg.jboss.byteman.transform.all -Dorg.jboss.byteman.verbose -p 55001 714
+    ./bminstall.sh -b -Dorg.jboss.byteman.transform.all -Dorg.jboss.byteman.verbose -p 55002 740
+    ./bminstall.sh -b -Dorg.jboss.byteman.transform.all -Dorg.jboss.byteman.verbose -p 55003 758
+
+    # load the fault injection script
+    ./bmsubmit.sh -p 55002 -l my_zk_fault_injection.btm
+    # unload the fault injection script
+    ./bmsubmit.sh -p 55002 -u my_zk_fault_injectionr.btm
+    ```
+
+Look at the below examples to customize your byteman fault injection script
+
+Example 1: This script makes leader's zxid roll over, to force re-election.
+
+```bash
+cat zk_leader_zxid_roll_over.btm
+
+RULE trace zk_leader_zxid_roll_over
+CLASS org.apache.zookeeper.server.quorum.Leader
+METHOD propose
+IF true
+DO
+  traceln("*** Leader zxid has rolled over, forcing re-election ***");
+  $1.zxid = 4294967295L
+ENDRULE
+```
+
+Example 2: This script makes the leader drop the ping packet to a specific follower.
+The leader will close the **LearnerHandler** with that follower, and the follower will enter the state:LOOKING
+then re-enter the quorum with the state:FOLLOWING
+
+```bash
+cat zk_leader_drop_ping_packet.btm
+
+RULE trace zk_leader_drop_ping_packet
+CLASS org.apache.zookeeper.server.quorum.LearnerHandler
+METHOD ping
+AT ENTRY
+IF $0.sid == 2
+DO
+  traceln("*** Leader drops ping packet to sid: 2 ***");
+  return;
+ENDRULE
+```
+
+Example 3: This script makes one follower drop ACK packet which has no big effect in the broadcast phrase, since after receiving
+the majority of ACKs from the followers, the leader can commit that proposal
+
+```bash
+cat zk_leader_drop_ping_packet.btm
+
+RULE trace zk.follower_drop_ack_packet
+CLASS org.apache.zookeeper.server.quorum.SendAckRequestProcessor
+METHOD processRequest
+AT ENTRY
+IF true
+DO
+  traceln("*** Follower drops ACK packet ***");
+  return;
+ENDRULE
+```
+
+
 <a name="jepsen-test"></a>
 
 ### Jepsen Test