[HADOOP-17139] Re-enable optimized copyFromLocal implementation in S3AFileSystem

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.3.0, 3.2.1
• Fix Version/s: 3.3.2
• Component/s: fs/s3
• Labels: pull-request-available
• Assignee: Bogdan Stolojan
• Reporter: Sahil Takiar
• Created: 17/Jul/20 19:29
• Updated: 31/Mar/22 04:19
• Resolved: 02/Aug/21 15:12

Description

It looks like HADOOP-15932 disabled the optimized copyFromLocal implementation in S3A for correctness reasons.  innerCopyFromLocalFile should be fixed and re-enabled. The current implementation uses FileSystem.copyFromLocal which will open an input stream from the local fs and an output stream to the destination fs, and then call IOUtils.copyBytes. With default configs, this will cause S3A to read the file into memory, write it back to a file on the local fs, and then when the file is closed, upload it to S3.

The optimized version of copyFromLocal in innerCopyFromLocalFile, directly creates a PutObjectRequest request with the local file as the input.

Issue Links

• causes: SPARK-38706 Use URI in FallbackStorage.copy
• is related to: SPARK-38652 uploadFileUri should preserve file scheme
• relates to: IMPALA-9828 Add support for spilling to Remote Storage
• relates to: HADOOP-17936 TestLocalFSCopyFromLocal.testDestinationFileIsToParentDirectory failure after reverting HADOOP-16878
• supercedes: HADOOP-15932 Oozie unable to create sharelib in s3a filesystem
• links to: GitHub Pull Request #3101