HADOOP-12951. Improve documentation on KMS ACLs and delegation tokens. Contributed by Xiao Chen.

hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java

@@ -40,85 +40,26 @@ import java.text.SimpleDateFormat;
 import java.util.*;
 
 /**
- * <p>The {@link AuthenticationFilter} enables protecting web application
+ * The {@link AuthenticationFilter} enables protecting web application
  * resources with different (pluggable)
  * authentication mechanisms and signer secret providers.
- * </p>
  * <p>
- * Out of the box it provides 2 authentication mechanisms: Pseudo and Kerberos SPNEGO.
- * </p>
  * Additional authentication mechanisms are supported via the {@link AuthenticationHandler} interface.
  * <p>
  * This filter delegates to the configured authentication handler for authentication and once it obtains an
  * {@link AuthenticationToken} from it, sets a signed HTTP cookie with the token. For client requests
  * that provide the signed HTTP cookie, it verifies the validity of the cookie, extracts the user information
  * and lets the request proceed to the target resource.
- * </p>
- * The supported configuration properties are:
- * <ul>
- * <li>config.prefix: indicates the prefix to be used by all other configuration properties, the default value
- * is no prefix. See below for details on how/why this prefix is used.</li>
- * <li>[#PREFIX#.]type: simple|kerberos|#CLASS#, 'simple' is short for the
- * {@link PseudoAuthenticationHandler}, 'kerberos' is short for {@link KerberosAuthenticationHandler}, otherwise
- * the full class name of the {@link AuthenticationHandler} must be specified.</li>
- * <li>[#PREFIX#.]signature.secret.file: when signer.secret.provider is set to
- * "file" or not specified, this is the location of file including the secret
- *  used to sign the HTTP cookie.</li>
- * <li>[#PREFIX#.]token.validity: time -in seconds- that the generated token is
- * valid before a new authentication is triggered, default value is
- * <code>3600</code> seconds. This is also used for the rollover interval for
- * the "random" and "zookeeper" SignerSecretProviders.</li>
- * <li>[#PREFIX#.]cookie.domain: domain to use for the HTTP cookie that stores the authentication token.</li>
- * <li>[#PREFIX#.]cookie.path: path to use for the HTTP cookie that stores the authentication token.</li>
- * </ul>
  * <p>
  * The rest of the configuration properties are specific to the {@link AuthenticationHandler} implementation and the
  * {@link AuthenticationFilter} will take all the properties that start with the prefix #PREFIX#, it will remove
  * the prefix from it and it will pass them to the the authentication handler for initialization. Properties that do
  * not start with the prefix will not be passed to the authentication handler initialization.
- * </p>
  * <p>
- * Out of the box it provides 3 signer secret provider implementations:
- * "file", "random" and "zookeeper"
- * </p>
- * Additional signer secret providers are supported via the
- * {@link SignerSecretProvider} class.
- * <p>
- * For the HTTP cookies mentioned above, the SignerSecretProvider is used to
- * determine the secret to use for signing the cookies. Different
- * implementations can have different behaviors. The "file" implementation
- * loads the secret from a specified file. The "random" implementation uses a
- * randomly generated secret that rolls over at the interval specified by the
- * [#PREFIX#.]token.validity mentioned above.  The "zookeeper" implementation
- * is like the "random" one, except that it synchronizes the random secret
- * and rollovers between multiple servers; it's meant for HA services.
- * </p>
- * The relevant configuration properties are:
- * <ul>
- * <li>signer.secret.provider: indicates the name of the SignerSecretProvider
- * class to use. Possible values are: "file", "random", "zookeeper", or a
- * classname. If not specified, the "file" implementation will be used with
- * [#PREFIX#.]signature.secret.file; and if that's not specified, the "random"
- * implementation will be used.</li>
- * <li>[#PREFIX#.]signature.secret.file: When the "file" implementation is
- * specified, this content of this file is used as the secret.</li>
- * <li>[#PREFIX#.]token.validity: When the "random" or "zookeeper"
- * implementations are specified, this value is used as the rollover
- * interval.</li>
- * </ul>
+ * Details of the configurations are listed on <a href="../../../../../../../Configuration.html">Configuration Page</a>
  * <p>
  * The "zookeeper" implementation has additional configuration properties that
  * must be specified; see {@link ZKSignerSecretProvider} for details.
- * </p>
- * For subclasses of AuthenticationFilter that want additional control over the
- * SignerSecretProvider, they can use the following attribute set in the
- * ServletContext:
- * <ul>
- * <li>signer.secret.provider.object: A SignerSecretProvider implementation can
- * be passed here that will be used instead of the signer.secret.provider
- * configuration property. Note that the class should already be
- * initialized.</li>
- * </ul>
  */
 
 @InterfaceAudience.Private

hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.java

@@ -57,34 +57,7 @@ import org.slf4j.LoggerFactory;
  * {@link org.apache.hadoop.security.authentication.server.AuthenticationFilter}
  * for more details.
  * <p>
- * The supported configuration properties are:
- * <ul>
- * <li>signer.secret.provider.zookeeper.connection.string: indicates the
- * ZooKeeper connection string to connect with.</li>
- * <li>signer.secret.provider.zookeeper.path: indicates the ZooKeeper path
- * to use for storing and retrieving the secrets.  All ZKSignerSecretProviders
- * that need to coordinate should point to the same path.</li>
- * <li>signer.secret.provider.zookeeper.auth.type: indicates the auth type to
- * use.  Supported values are "none" and "sasl".  The default value is "none"
- * </li>
- * <li>signer.secret.provider.zookeeper.kerberos.keytab: set this to the path
- * with the Kerberos keytab file.  This is only required if using Kerberos.</li>
- * <li>signer.secret.provider.zookeeper.kerberos.principal: set this to the
- * Kerberos principal to use.  This only required if using Kerberos.</li>
- * <li>signer.secret.provider.zookeeper.disconnect.on.close: when set to "true",
- * ZKSignerSecretProvider will close the ZooKeeper connection on shutdown.  The
- * default is "true". Only set this to "false" if a custom Curator client is
- * being provided and the disconnection is being handled elsewhere.</li>
- * </ul>
- *
- * The following attribute in the ServletContext can also be set if desired:
- * <ul>
- * <li>signer.secret.provider.zookeeper.curator.client: A CuratorFramework
- * client object can be passed here. If given, the "zookeeper" implementation
- * will use this Curator client instead of creating its own, which is useful if
- * you already have a Curator client or want more control over its
- * configuration.</li>
- * </ul>
+ * Details of the configurations are listed on <a href="../../../../../../../Configuration.html">Configuration Page</a>
  */
 @InterfaceStability.Unstable
 @InterfaceAudience.Private

hadoop-common-project/hadoop-auth/src/site/markdown/Configuration.md

@@ -34,12 +34,11 @@ Hadoop Auth uses SLF4J-API for logging. Auth Maven POM dependencies define the S
 *   `[PREFIX.]type`: the authentication type keyword (`simple` or \
     `kerberos`) or a Authentication handler implementation.
 
-*   `[PREFIX.]signature.secret`: When `signer.secret.provider` is set to
-    `string` or not specified, this is the value for the secret used to sign
-    the HTTP cookie.
+*   `[PREFIX.]signature.secret.file`: When `signer.secret.provider` is set to
+    `file`, this is the location of file including the secret used to sign the HTTP cookie.
 
 *   `[PREFIX.]token.validity`: The validity -in seconds- of the generated
-    authentication token. The default value is `3600` seconds. This is also
+    authentication token. The default value is `36000` seconds. This is also
     used for the rollover interval when `signer.secret.provider` is set to
     `random` or `zookeeper`.
 
@@ -50,10 +49,11 @@ Hadoop Auth uses SLF4J-API for logging. Auth Maven POM dependencies define the S
     authentication token.
 
 *   `signer.secret.provider`: indicates the name of the SignerSecretProvider
-    class to use. Possible values are: `string`, `random`,
-    `zookeeper`, or a classname. If not specified, the `string`
+    class to use. Possible values are: `file`, `random`,
+    `zookeeper`, or a classname. If not specified, the `file`
     implementation will be used; and failing that, the `random`
-    implementation will be used.
+    implementation will be used. If "file" is to be used, one need to specify
+    `signature.secret.file` and point to the secret file.
 
 ### Kerberos Configuration
 
@@ -232,24 +232,25 @@ The SignerSecretProvider is used to provide more advanced behaviors for the secr
 These are the relevant configuration properties:
 
 *   `signer.secret.provider`: indicates the name of the
-    SignerSecretProvider class to use. Possible values are: "string",
-    "random", "zookeeper", or a classname. If not specified, the "string"
+    SignerSecretProvider class to use. Possible values are: "file",
+    "random", "zookeeper", or a classname. If not specified, the "file"
     implementation will be used; and failing that, the "random" implementation
-    will be used.
+    will be used. If "file" is to be used, one need to specify `signature.secret.file`
+    and point to the secret file.
 
-*   `[PREFIX.]signature.secret`: When `signer.secret.provider` is set
-    to `string` or not specified, this is the value for the secret used to
+*   `[PREFIX.]signature.secret.file`: When `signer.secret.provider` is set
+    to `file` or not specified, this is the value for the secret used to
     sign the HTTP cookie.
 
 *   `[PREFIX.]token.validity`: The validity -in seconds- of the generated
-    authentication token. The default value is `3600` seconds. This is
+    authentication token. The default value is `36000` seconds. This is
     also used for the rollover interval when `signer.secret.provider` is
     set to `random` or `zookeeper`.
 
 The following configuration properties are specific to the `zookeeper` implementation:
 
 *   `signer.secret.provider.zookeeper.connection.string`: Indicates the
-    ZooKeeper connection string to connect with.
+    ZooKeeper connection string to connect with. The default value is `localhost:2181`
 
 *   `signer.secret.provider.zookeeper.path`: Indicates the ZooKeeper path
     to use for storing and retrieving the secrets. All servers
@@ -266,6 +267,17 @@ The following configuration properties are specific to the `zookeeper` implement
 *   `signer.secret.provider.zookeeper.kerberos.principal`: Set this to the
     Kerberos principal to use. This only required if using Kerberos.
 
+*   `signer.secret.provider.zookeeper.disconnect.on.shutdown`: Whether to close the
+    ZooKeeper connection when the provider is shutdown. The default value is `true`.
+    Only set this to `false` if a custom Curator client is being provided and
+    the disconnection is being handled elsewhere.
+
+The following attribute in the ServletContext can also be set if desired:
+*   `signer.secret.provider.zookeeper.curator.client`: A CuratorFramework client
+    object can be passed here. If given, the "zookeeper" implementation will use
+    this Curator client instead of creating its own, which is useful if you already
+    have a Curator client or want more control over its configuration.
+
 **Example**:
 
 ```xml
@@ -276,11 +288,11 @@ The following configuration properties are specific to the `zookeeper` implement
             <!-- AuthenticationHandler configs not shown -->
             <init-param>
                 <param-name>signer.secret.provider</param-name>
-                <param-value>string</param-value>
+                <param-value>file</param-value>
             </init-param>
             <init-param>
-                <param-name>signature.secret</param-name>
-                <param-value>my_secret</param-value>
+                <param-name>signature.secret.file</param-name>
+                <param-value>/myapp/secret_file</param-value>
             </init-param>
         </filter>
 
@@ -334,10 +346,6 @@ The following configuration properties are specific to the `zookeeper` implement
                 <param-name>signer.secret.provider.zookeeper.path</param-name>
                 <param-value>/myapp/secrets</param-value>
             </init-param>
-            <init-param>
-                <param-name>signer.secret.provider.zookeeper.use.kerberos.acls</param-name>
-                <param-value>true</param-value>
-            </init-param>
             <init-param>
                 <param-name>signer.secret.provider.zookeeper.kerberos.keytab</param-name>
                 <param-value>/tmp/auth.keytab</param-value>

hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm

@@ -284,7 +284,15 @@ The answer to "What is your first and last name?" (i.e. "CN") must be the hostna
 
 NOTE: You need to restart the KMS for the configuration changes to take effect.
 
-$H4 KMS Access Control
+$H4 ACLs (Access Control Lists)
+
+KMS supports ACLs (Access Control Lists) for fine-grained permission control.
+
+Two levels of ACLs exist in KMS: KMS ACLs and Key ACLs. KMS ACLs control access at KMS operation level, and precede Key ACLs. In particular, only if permission is granted at KMS ACLs level, shall the permission check against Key ACLs be performed.
+
+The configuration and usage of KMS ACLs and Key ACLs are described in the sections below.
+
+$H5 KMS ACLs
 
 KMS ACLs configuration are defined in the KMS `etc/hadoop/kms-acls.xml` configuration file. This file is hot-reloaded when it changes.
 
@@ -452,7 +460,7 @@ A user accessing KMS is first checked for inclusion in the Access Control List f
 </configuration>
 ```
 
-$H4 Key Access Control
+$H5 Key ACLs
 
 KMS supports access control for all non-read operations at the Key level. All Key Access operations are classified as :
 
@@ -466,9 +474,9 @@ These can be defined in the KMS `etc/hadoop/kms-acls.xml` as follows
 
 For all keys for which a key access has not been explicitly configured, It is possible to configure a default key access control for a subset of the operation types.
 
-It is also possible to configure a "whitelist" key ACL for a subset of the operation types. The whitelist key ACL is a whitelist in addition to the explicit or default per-key ACL. That is, if no per-key ACL is explicitly set, a user will be granted access if they are present in the default per-key ACL or the whitelist key ACL. If a per-key ACL is explicitly set, a user will be granted access if they are present in the per-key ACL or the whitelist key ACL.
+It is also possible to configure a "whitelist" key ACL for a subset of the operation types. The whitelist key ACL grants access to the key, in addition to the explicit or default per-key ACL. That is, if no per-key ACL is explicitly set, a user will be granted access if they are present in the default per-key ACL or the whitelist key ACL. If a per-key ACL is explicitly set, a user will be granted access if they are present in the per-key ACL or the whitelist key ACL.
 
-If no ACL is configured for a specific key AND no default ACL is configured AND no root key ACL is configured for the requested operation, then access will be DENIED.
+If no ACL is configured for a specific key AND no default ACL is configured AND no whitelist key ACL is configured for the requested operation, then access will be DENIED.
 
 **NOTE:** The default and whitelist key ACL does not support `ALL` operation qualifier.
 
@@ -575,7 +583,11 @@ If no ACL is configured for a specific key AND no default ACL is configured AND
 
 $H3 KMS Delegation Token Configuration
 
-KMS delegation token secret manager can be configured with the following properties:
+KMS supports delegation tokens to authenticate to the key providers from processes without Kerberos credentials.
+
+KMS delegation token authentication extends the default Hadoop authentication. See [Hadoop Auth](../hadoop-auth/index.html) page for more details.
+
+Additionally, KMS delegation token secret manager can be configured with the following properties:
 
 ```xml
   <property>
@@ -590,7 +602,7 @@ KMS delegation token secret manager can be configured with the following propert
     <name>hadoop.kms.authentication.delegation-token.max-lifetime.sec</name>
     <value>604800</value>
     <description>
-      Maximum lifetime of a delagation token, in seconds. Default value 7 days.
+      Maximum lifetime of a delegation token, in seconds. Default value 7 days.
     </description>
   </property>
 
@@ -598,7 +610,7 @@ KMS delegation token secret manager can be configured with the following propert
     <name>hadoop.kms.authentication.delegation-token.renew-interval.sec</name>
     <value>86400</value>
     <description>
-      Renewal interval of a delagation token, in seconds. Default value 1 day.
+      Renewal interval of a delegation token, in seconds. Default value 1 day.
     </description>
   </property>
 
@@ -640,7 +652,7 @@ $H4 HTTP Authentication Signature
 
 KMS uses Hadoop Authentication for HTTP authentication. Hadoop Authentication issues a signed HTTP Cookie once the client has authenticated successfully. This HTTP Cookie has an expiration time, after which it will trigger a new authentication sequence. This is done to avoid triggering the authentication on every HTTP request of a client.
 
-A KMS instance must verify the HTTP Cookie signatures signed by other KMS instances. To do this all KMS instances must share the signing secret.
+A KMS instance must verify the HTTP Cookie signatures signed by other KMS instances. To do this, all KMS instances must share the signing secret. Please see [SignerSecretProvider Configuration](../hadoop-auth/Configuration.html#SignerSecretProvider_Configuration) for detailed description and configuration examples. Note that KMS configurations need to be prefixed with `hadoop.kms.authentication`, as shown in the example below.
 
 This secret sharing can be done using a Zookeeper service which is configured in KMS with the following properties in the `kms-site.xml`:
 
@@ -650,8 +662,9 @@ This secret sharing can be done using a Zookeeper service which is configured in
     <value>zookeeper</value>
     <description>
       Indicates how the secret to sign the authentication cookies will be
-      stored. Options are 'random' (default), 'string' and 'zookeeper'.
+      stored. Options are 'random' (default), 'file' and 'zookeeper'.
       If using a setup with multiple KMS instances, 'zookeeper' should be used.
+      If using file, signature.secret.file should be configured and point to the secret file.
     </description>
   </property>
   <property>
@@ -659,7 +672,7 @@ This secret sharing can be done using a Zookeeper service which is configured in
     <value>/hadoop-kms/hadoop-auth-signature-secret</value>
     <description>
       The Zookeeper ZNode path where the KMS instances will store and retrieve
-      the secret from.
+      the secret from. All KMS instances that need to coordinate should point to the same path.
     </description>
   </property>
   <property>
@@ -696,7 +709,11 @@ This secret sharing can be done using a Zookeeper service which is configured in
 
 $H4 Delegation Tokens
 
-TBD
+Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation tokens too.
+
+Under HA, A KMS instance must verify the delegation token given by another KMS instance, by checking the shared secret used to sign the delegation token. To do this, all KMS instances must be able to retrieve the shared secret from ZooKeeper.
+
+Please see the examples given in HTTP Authentication section to configure ZooKeeper for secret sharing.
 
 $H3 KMS HTTP REST API