Pārlūkot izejas kodu

HADOOP-18246. Reduce lower limit on fs.s3a.prefetch.block.size to 1 byte. (#5120)

The minimum value of fs.s3a.prefetch.block.size is now 1

Contributed by Ankit Saurabh
Ankit Saurabh 2 gadi atpakaļ
vecāks
revīzija
f439801484

+ 1 - 2
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java

@@ -543,8 +543,7 @@ public class S3AFileSystem extends FileSystem implements StreamCapabilities,
           DEFAULT_MULTIPART_UPLOAD_ENABLED);
       this.prefetchEnabled = conf.getBoolean(PREFETCH_ENABLED_KEY, PREFETCH_ENABLED_DEFAULT);
       long prefetchBlockSizeLong =
-          longBytesOption(conf, PREFETCH_BLOCK_SIZE_KEY, PREFETCH_BLOCK_DEFAULT_SIZE,
-              PREFETCH_BLOCK_DEFAULT_SIZE);
+          longBytesOption(conf, PREFETCH_BLOCK_SIZE_KEY, PREFETCH_BLOCK_DEFAULT_SIZE, 1);
       if (prefetchBlockSizeLong > (long) Integer.MAX_VALUE) {
         throw new IOException("S3A prefetch block size exceeds int limit");
       }

+ 1 - 0
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md

@@ -1107,6 +1107,7 @@ options are covered in [Testing](./testing.md).
   <value>8MB</value>
   <description>
       The size of a single prefetched block of data.
+      Decreasing this will increase the number of prefetches required, and may negatively impact performance.
   </description>
 </property>
 

+ 4 - 0
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/prefetching.md

@@ -43,6 +43,10 @@ Multiple blocks may be read in parallel.
 |`fs.s3a.prefetch.block.size`    |Size of a block    |`8M`    |
 |`fs.s3a.prefetch.block.count`    |Number of blocks to prefetch    |`8`    |
 
+The default size of a block is 8MB, and the minimum allowed block size is 1 byte.
+Decreasing block size will increase the number of blocks to be read for a file.
+A smaller block size may negatively impact performance as the number of prefetches required will increase.
+
 ### Key Components
 
 `S3PrefetchingInputStream` - When prefetching is enabled, S3AFileSystem will return an instance of