HDFS-13414. Ozone: Update existing Ozone documentation according to the recent changes. Contributed by Elek, Marton.

hadoop-ozone/ozone-manager/src/main/site/markdown/OzoneCommandShell.md

@@ -25,10 +25,10 @@ The Ozone commands take the following format.
 <name> -root`
 
 The *port* specified in command should match the port mentioned in the config
-property `dfs.datanode.http.address`. This property can be set in `hdfs-site.xml`.
-The default value for the port is `9864` and is used in below commands.
+property `hdds.rest.http-address`. This property can be set in `ozone-site.xml`.
+The default value for the port is `9880` and is used in below commands.
 
-The *--root* option is a command line short cut that allows *ozone oz*
+The *-root* option is a command line short cut that allows *ozone oz*
 commands to be run as the user that started the cluster. This is useful to
 indicate that you want the commands to be run as some admin user. The only
 reason for this option is that it makes the life of a lazy developer more
@@ -44,37 +44,37 @@ ozone cluster.
 
 Volumes can be created only by Admins. Here is an example of creating a volume.
 
-* `ozone oz -createVolume http://localhost:9864/hive -user bilbo -quota
+* `ozone oz -createVolume http://localhost:9880/hive -user bilbo -quota
 100TB -root`
 
 The above command creates a volume called `hive` owned by user `bilbo`. The
-`--root` option allows the command to be executed as user `hdfs` which is an
+`-root` option allows the command to be executed as user `hdfs` which is an
 admin in the cluster.
 
 ### Update Volume
 
 Updates information like ownership and quota on an existing volume.
 
-* `ozone oz  -updateVolume  http://localhost:9864/hive -quota 500TB -root`
+* `ozone oz  -updateVolume  http://localhost:9880/hive -quota 500TB -root`
 
 The above command changes the volume quota of hive from 100TB to 500TB.
 
 ### Delete Volume
 Deletes a Volume if it is empty.
 
-* `ozone oz -deleteVolume http://localhost:9864/hive -root`
+* `ozone oz -deleteVolume http://localhost:9880/hive -root`
 
 
 ### Info Volume
 Info volume command allows the owner or the administrator of the cluster to read meta-data about a specific volume.
 
-* `ozone oz -infoVolume http://localhost:9864/hive -root`
+* `ozone oz -infoVolume http://localhost:9880/hive -root`
 
 ### List Volumes
 
 List volume command can be used by administrator to list volumes of any user. It can also be used by a user to list volumes owned by him.
 
-* `ozone oz -listVolume http://localhost:9864/ -user bilbo -root`
+* `ozone oz -listVolume http://localhost:9880/ -user bilbo -root`
 
 The above command lists all volumes owned by user bilbo.
 
@@ -89,7 +89,7 @@ Following examples assume that these commands are run by the owner of the volume
 
 Create bucket call allows the owner of a volume to create a bucket.
 
-* `ozone oz -createBucket http://localhost:9864/hive/january`
+* `ozone oz -createBucket http://localhost:9880/hive/january`
 
 This call creates a bucket called `january` in the volume called `hive`. If
 the volume does not exist, then this call will fail.
@@ -98,23 +98,23 @@ the volume does not exist, then this call will fail.
 ### Update Bucket
 Updates bucket meta-data, like ACLs.
 
-* `ozone oz -updateBucket http://localhost:9864/hive/january  -addAcl
+* `ozone oz -updateBucket http://localhost:9880/hive/january  -addAcl
 user:spark:rw`
 
 ### Delete Bucket
 Deletes a bucket if it is empty.
 
-* `ozone oz -deleteBucket http://localhost:9864/hive/january`
+* `ozone oz -deleteBucket http://localhost:9880/hive/january`
 
 ### Info Bucket
 Returns information about a given bucket.
 
-* `ozone oz -infoBucket http://localhost:9864/hive/january`
+* `ozone oz -infoBucket http://localhost:9880/hive/january`
 
 ### List Buckets
 List buckets on a given volume.
 
-* `ozone oz -listBucket http://localhost:9864/hive`
+* `ozone oz -listBucket http://localhost:9880/hive`
 
 Ozone Key Commands
 ------------------
@@ -125,26 +125,26 @@ Ozone key commands allows users to put, delete and get keys from ozone buckets.
 Creates or overwrites a key in ozone store, -file points to the file you want
 to upload.
 
-* `ozone oz -putKey  http://localhost:9864/hive/january/processed.orc  -file
+* `ozone oz -putKey  http://localhost:9880/hive/january/processed.orc  -file
 processed.orc`
 
 ### Get Key
 Downloads a file from the ozone bucket.
 
-* `ozone oz -getKey  http://localhost:9864/hive/january/processed.orc  -file
+* `ozone oz -getKey  http://localhost:9880/hive/january/processed.orc  -file
   processed.orc.copy`
 
 ### Delete Key
 Deletes a key  from the ozone store.
 
-* `ozone oz -deleteKey http://localhost:9864/hive/january/processed.orc`
+* `ozone oz -deleteKey http://localhost:9880/hive/january/processed.orc`
 
 ### Info Key
 Reads  key metadata from the ozone store.
 
-* `ozone oz -infoKey http://localhost:9864/hive/january/processed.orc`
+* `ozone oz -infoKey http://localhost:9880/hive/january/processed.orc`
 
 ### List Keys
 List all keys in an ozone bucket.
 
-* `ozone oz -listKey  http://localhost:9864/hive/january`
+* `ozone oz -listKey  http://localhost:9880/hive/january`

hadoop-ozone/ozone-manager/src/main/site/markdown/OzoneGettingStarted.md.vm

@@ -22,16 +22,17 @@ API for accessing the store.
 
 Getting Started
 ---------------
-Ozone is a work in progress and  currently lives in its own branch. To
+Ozone is a work in progress and currently lives in the hadoop source tree.
+The subprojects (ozone/hdds) are part of the hadoop source tree but by default
+not compiled and not part of the official releases. To
 use it, you have to build a package by yourself and deploy a cluster.
 
 ### Building Ozone
 
 To build Ozone, please checkout the hadoop sources from github. Then
-checkout the ozone branch, HDFS-7240 and build it.
+checkout the trunk branch and build it.
 
-- `git checkout HDFS-7240`
-- `mvn clean package -DskipTests=true -Dmaven.javadoc.skip=true -Pdist -Phdsl -Dtar -DskipShade`
+`mvn clean package -DskipTests=true -Dmaven.javadoc.skip=true -Pdist -Phdds -Dtar -DskipShade`
 
 skipShade is just to make compilation faster and not really required.
 
@@ -66,13 +67,12 @@ Now let us run some work load against ozone, to do that we will run freon.
 This will log into the datanode and run bash.
 
  - `docker-compose exec datanode bash`
- - `cd /opt/hadoop/bin`
 
-Now you can run the oz command shell or freon, the ozone load generator.
+Now you can run the `ozone` command shell or freon, the ozone load generator.
 
 This is the command to run freon.
 
- - `./oz freon -mode offline -validateWrites -numOfVolumes 1 -numOfBuckets 10 -numOfKeys 100`
+ - `ozone freon -mode offline -validateWrites -numOfVolumes 1 -numOfBuckets 10 -numOfKeys 100`
 
 You can checkout the KSM UI to see the requests information.
 
@@ -122,9 +122,9 @@ place  and not mingled with HDFS settings.
    </property>
 ```
 
-* _*ozone.scm.names*_ Ozone is build on top of container framework (See Ozone
- Architecture TODO). Storage container manager(SCM) is a distributed block
- service which is used by ozone and other storage services.
+* _*ozone.scm.names*_ Ozone is build on top of container framework. Storage
+ container manager(SCM) is a distributed block service which is used by ozone
+ and other storage services.
  This property allows datanodes to discover where SCM is, so that
  datanodes can send heartbeat to SCM. SCM is designed to be highly available
  and datanodes assume there are multiple instances of SCM which form a highly
@@ -175,6 +175,19 @@ Ozone File System.
     </property>
 ```
 
+* _*dfs.datanode.plugin*_ Datanode service plugins: the container manager part
+ of ozone is running inside the datanode as a service plugin. To activate ozone
+ you should define the service plugin implementation class. **Important**
+ It should be added to the **hdfs-site.xml** as the plugin should be activated
+ as part of the normal HDFS Datanode bootstrap.
+
+```
+    <property>
+       <name>dfs.datanode.plugins</name>
+       <value>org.apache.hadoop.ozone.HddsDatanodeService</value>
+    </property>
+```
+
 Here is a quick summary of settings needed by Ozone.
 
 | Setting                        | Value                        | Comment |
@@ -230,6 +243,16 @@ Here is a quick summary of settings needed by Ozone.
     </configuration>
 ```
 
+And don't forget to enable the datanode component with adding the
+following configuration to the hdfs-site.xml:
+
+```
+    <property>
+       <name>dfs.datanode.plugins</name>
+       <value>org.apache.hadoop.ozone.HddsDatanodeService</value>
+    </property>
+```
+
 ### Starting Ozone
 
 Ozone is designed to run concurrently with HDFS. The simplest way to [start
@@ -246,19 +269,19 @@ is running, please verify it is fully functional by running some commands like
 
  The first time you bring up Ozone, SCM must be initialized.
 
-   - `./hdfs scm -init`
+   - `./ozone scm -init`
 
  Start SCM.
 
-   - `./hdfs --daemon start scm`
+   - `./ozone --daemon start scm`
 
  Once SCM gets started, KSM must be initialized.
 
-   - `./hdfs ksm -createObjectStore`
+   - `./ozone ksm -createObjectStore`
 
  Start KSM.
 
-   - `./hdfs --daemon start ksm`
+   - `./ozone --daemon start ksm`
 
 if you would like to start HDFS and Ozone together, you can do that by running
  a single command.
@@ -316,5 +339,9 @@ On the SCM/KSM side, you will be able to see
   - `hadoop-hdfs-ksm-hostname.log`
   - `hadoop-hdfs-scm-hostname.log`
 
-Please file any issues you see under [Object store in HDFS (HDFS-7240)](https://issues.apache.org/jira/browse/HDFS-7240)
-as this is still a work in progress.
+Please file any issues you see under the related issues:
+
+ - [Object store in HDFS: HDFS-7240](https://issues.apache.org/jira/browse/HDFS-7240)
+ - [Ozone File System: HDFS-13074](https://issues.apache.org/jira/browse/HDFS-13074)
+ - [Building HDFS on top of new storage layer (HDDS): HDFS-10419](https://issues.apache.org/jira/browse/HDFS-10419)
+

hadoop-ozone/ozone-manager/src/main/site/markdown/OzoneRest.md

@@ -84,7 +84,7 @@ Query Parameter:
 
 Sample HTTP POST request:
 
-    curl -i -X POST -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE root" "http://localhost:9864/volume-to-create"
+    curl -i -X POST -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE root" "http://localhost:9880/volume-to-create"
 
 this request creates a volume as user *bilbo*, the authorization field is set to *OZONE root* because this call requires administration privilege. The client receives a response with zero content length.
 
@@ -112,7 +112,7 @@ Query Parameter:
 
 Sample HTTP PUT request:
 
-    curl -X PUT -H "Authorization:OZONE root" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "x-ozone-version: v1" -H "x-ozone-user: john"  http://localhost:9864/volume-to-update
+    curl -X PUT -H "Authorization:OZONE root" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "x-ozone-version: v1" -H "x-ozone-user: john"  http://localhost:9880/volume-to-update
 
 this request modifies the owner of */volume-to-update* to *john*.
 
@@ -126,7 +126,7 @@ Schema:
 
 Sample HTTP DELETE request:
 
-    curl -i -X DELETE -H "Authorization:OZONE root" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "x-ozone-version: v1" -H "x-ozone-user: bilbo"  http://localhost:9864/volume-to-delete
+    curl -i -X DELETE -H "Authorization:OZONE root" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "x-ozone-version: v1" -H "x-ozone-user: bilbo"  http://localhost:9880/volume-to-delete
 
 this request deletes an empty volume */volume-to-delete*. The client receives a zero length content.
 
@@ -154,7 +154,7 @@ Query Parameter:
 
 Sample HTTP GET request:
 
-    curl -i -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" "http://localhost:9864/volume-of-bilbo?info=volume"
+    curl -i -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" "http://localhost:9880/volume-of-bilbo?info=volume"
 
 this request gets the info of volume */volume-of-bilbo*, the client receives a response with a JSON object of volume info.
 
@@ -193,7 +193,7 @@ Query Parameter:
 
 Sample HTTP GET request:
 
-    curl -i -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" "http://localhost:9864/?max-keys=100&prefix=Jan"
+    curl -i -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" "http://localhost:9880/?max-keys=100&prefix=Jan"
 
 this request gets all volumes owned by *bilbo* and each volume's name contains prefix *Jan*, the result at most contains *100* entries. The client receives a list of SON objects, each of them describes the info of a volume.
 
@@ -239,7 +239,7 @@ Additional HTTP Headers:
 
 Sample HTTP POST request:
 
-    curl -i -X POST -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" http://localhost:9864/volume-of-bilbo/bucket-0
+    curl -i -X POST -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" http://localhost:9880/volume-of-bilbo/bucket-0
 
 this request creates a bucket *bucket-0* under volume *volume-of-bilbo*.
 
@@ -268,7 +268,7 @@ Additional HTTP Headers:
 
 Sample HTTP PUT request:
 
-    curl -i -X PUT -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" -H "x-ozone-acl: ADD user:peregrin:rw" http://localhost:9864/volume-of-bilbo/bucket-to-update
+    curl -i -X PUT -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" -H "x-ozone-acl: ADD user:peregrin:rw" http://localhost:9880/volume-of-bilbo/bucket-to-update
 
 this request adds an ACL policy specified by HTTP header *x-ozone-acl* to bucket */volume-of-bilbo/bucket-to-update*, the ACL field *ADD user:peregrin:rw* gives add additional read/write permission to user *peregrin* to this bucket.
 
@@ -290,7 +290,7 @@ Schema:
 
 Sample HTTP DELETE request:
 
-    curl -i -X DELETE -H "Authorization:OZONE root" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "x-ozone-version: v1" -H "x-ozone-user:bilbo" "http://localhost:9864/volume-of-bilbo/bucket-0"
+    curl -i -X DELETE -H "Authorization:OZONE root" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "x-ozone-version: v1" -H "x-ozone-user:bilbo" "http://localhost:9880/volume-of-bilbo/bucket-0"
 
 this request deletes bucket */volume-of-bilbo/bucket-0*. The client receives a zero length content response.
 
@@ -319,7 +319,7 @@ Query Parameters:
 
 Sample HTTP GET request:
 
-    curl -i -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" "http://localhost:9864/volume-of-bilbo/bucket-0?info=bucket"
+    curl -i -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" "http://localhost:9880/volume-of-bilbo/bucket-0?info=bucket"
 
 this request gets the info of bucket */volume-of-bilbo/bucket-0*. The client receives a response of JSON object contains bucket info.
 
@@ -358,7 +358,7 @@ Query Parameters:
 
 Sample HTTP GET request:
 
-    curl -i -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" "http://localhost:9864/volume-of-bilbo?max-keys=10"
+    curl -i -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" "http://localhost:9880/volume-of-bilbo?max-keys=10"
 
 this request lists all the buckets under volume *volume-of-bilbo*, and the result at most contains 10 entries. The client receives response of a array of JSON objects, each of them represents for a bucket info.
 
@@ -404,7 +404,7 @@ Additional HTTP headers:
 
 Sample PUT HTTP request:
 
-    curl -X PUT -T /path/to/localfile -H "Authorization:OZONE" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "x-ozone-version: v1" -H "x-ozone-user:bilbo" "http://localhost:9864/volume-of-bilbo/bucket-0/file-0"
+    curl -X PUT -T /path/to/localfile -H "Authorization:OZONE" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "x-ozone-version: v1" -H "x-ozone-user:bilbo" "http://localhost:9880/volume-of-bilbo/bucket-0/file-0"
 
 this request uploads a local file */path/to/localfile* specified by option *-T* to ozone as user *bilbo*, mapped to ozone key */volume-of-bilbo/bucket-0/file-0*. The client receives a zero length content response.
 
@@ -418,7 +418,7 @@ Schema:
 
 Sample HTTP GET request:
 
-    curl -i -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" "http://localhost:9864/volume-of-bilbo/bucket-0/file-0"
+    curl -i -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" "http://localhost:9880/volume-of-bilbo/bucket-0/file-0"
 
 this request reads the content of key */volume-of-bilbo/bucket-0/file-0*. If the content of the file is plain text, it can be directly dumped onto stdout.
 
@@ -434,7 +434,7 @@ this request reads the content of key */volume-of-bilbo/bucket-0/file-0*. If the
 
 if the file is not plain text, specify *-O* option in curl command and the file *file-0* will be downloaded into current working directory, file name will be same as the key. A sample request like following:
 
-    curl -O -i -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" "http://localhost:9864/volume-of-bilbo/bucket-0/file-1"
+    curl -O -i -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" "http://localhost:9880/volume-of-bilbo/bucket-0/file-1"
 
 response looks like following:
 
@@ -452,7 +452,7 @@ Schema:
 
 Sample HTTP DELETE request:
 
-    curl -i -X DELETE -H "Authorization:OZONE root" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "x-ozone-version: v1" -H "x-ozone-user:bilbo" "http://localhost:9864/volume-of-bilbo/bucket-0/file-0"
+    curl -i -X DELETE -H "Authorization:OZONE root" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "x-ozone-version: v1" -H "x-ozone-user:bilbo" "http://localhost:9880/volume-of-bilbo/bucket-0/file-0"
 
 this request deletes key */volume-of-bilbo/bucket-0/file-0*. The client receives a zero length content result:
 
@@ -480,7 +480,7 @@ Query Parameter:
 
 Sample HTTP DELETE request:
 
-    curl -i -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" "http://localhost:9864/volume-of-bilbo/buket-0/file-0?info=key"
+    curl -i -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" "http://localhost:9880/volume-of-bilbo/buket-0/file-0?info=key"
 
 this request returns information of the key */volume-of-bilbo/bucket-0/file-0*. The client receives a JSON object listed attributes of the key.
 
@@ -519,7 +519,7 @@ Query Parameters:
 
 Sample HTTP GET request:
 
-    curl -i -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" "http:/localhost:9864/volume-of-bilbo/bucket-0/?max-keys=100&prefix=file"
+    curl -i -H "x-ozone-user: bilbo" -H "x-ozone-version: v1" -H "Date: Mon, 26 Jun 2017 04:23:30 GMT" -H "Authorization:OZONE" "http:/localhost:9880/volume-of-bilbo/bucket-0/?max-keys=100&prefix=file"
 
 this request list keys under bucket */volume-of-bilbo/bucket-0*, the listing result is filtered by prefix *file*. The client receives an array of JSON objects, each of them represents the info of a matched key.