瀏覽代碼

HADOOP-17454. [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0 (#2593)

Also fixes HADOOP-16995. ITestS3AConfiguration proxy tests failures when bucket probes == 0
The improvement should include the fix, ebcause the test would fail by default otherwise.

Change-Id: I9a7e4b5e6d4391ebba096c15e84461c038a2ec59
Gabor Bota 4 年之前
父節點
當前提交
42eb9ff68e

+ 1 - 1
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java

@@ -499,7 +499,7 @@ public final class Constants {
    * will be validated using {@code S3AFileSystem.verifyBucketExistsV2()}.
    * Value: {@value}
    */
-  public static final int S3A_BUCKET_PROBE_DEFAULT = 2;
+  public static final int S3A_BUCKET_PROBE_DEFAULT = 0;
 
   /**
    * How long a directory listing in the MS is considered as authoritative.

+ 2 - 2
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md

@@ -1039,9 +1039,9 @@ options are covered in [Testing](./testing.md).
 
 <property>
   <name>fs.s3a.bucket.probe</name>
-  <value>2</value>
+  <value>0</value>
   <description>
-     The value can be 0, 1 or 2 (default).
+     The value can be 0 (default), 1 or 2.
      When set to 0, bucket existence checks won't be done
      during initialization thus making it faster.
      Though it should be noted that when the bucket is not available in S3,

+ 1 - 0
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AConfiguration.java

@@ -159,6 +159,7 @@ public class ITestS3AConfiguration {
     return intercept(clazz,
         () -> {
           fs = S3ATestUtils.createTestFileSystem(conf);
+          fs.listFiles(new Path("/"), false);
           return "expected failure creating FS " + text + " got " + fs;
         });
   }