HADOOP-19015. Increase fs.s3a.connection.maximum to 500 to minimize risk of Timeout waiting for connection from pool. (#6372)

HADOOP-19015.  Increase fs.s3a.connection.maximum to 500 to minimize the risk of Timeout waiting for connection from the pool

Contributed By: Mukund Thakur

hadoop-common-project/hadoop-common/src/main/resources/core-default.xml

@@ -1530,7 +1530,7 @@
 
 <property>
   <name>fs.s3a.connection.maximum</name>
-  <value>200</value>
+  <value>500</value>
   <description>Controls the maximum number of simultaneous connections to S3.
     This must be bigger than the value of fs.s3a.threads.max so as to stop
     threads being blocked waiting for new HTTPS connections.
@@ -1538,6 +1538,7 @@
   </description>
 </property>
 
+
 <property>
   <name>fs.s3a.connection.ssl.enabled</name>
   <value>true</value>

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java

@@ -173,7 +173,7 @@ public final class Constants {
    * Future releases are likely to increase this value.
    * Keep in sync with the value in {@code core-default.xml}
    */
-  public static final int DEFAULT_MAXIMUM_CONNECTIONS = 200;
+  public static final int DEFAULT_MAXIMUM_CONNECTIONS = 500;
 
   /**
    * Configuration option to configure expiration time of

hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md

@@ -226,7 +226,7 @@ The choice is yours. Generally recovery is better, but sometimes fail-fast is mo
 
 | Property                                | Default | V2  | Meaning                                               |
 |-----------------------------------------|---------|:----|-------------------------------------------------------|
-| `fs.s3a.connection.maximum`             | `200`   |     | Connection pool size                                  |
+| `fs.s3a.connection.maximum`             | `500`   |     | Connection pool size                                  |
 | `fs.s3a.connection.keepalive`           | `false` | `*` | Use TCP keepalive on open channels                    |
 | `fs.s3a.connection.acquisition.timeout` | `60s`   | `*` | Timeout for waiting for a connection from the pool.   |
 | `fs.s3a.connection.establish.timeout`   | `30s`   |     | Time to establish the TCP/TLS connection              |