[HADOOP-16840] AliyunOSS: getFileStatus throws FileNotFoundException in versioning bucket

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.10.0, 2.9.2, 3.0.3, 3.2.1, 3.1.3
• Fix Version/s: 3.0.4, 3.3.0, 2.9.3, 3.1.4, 3.2.2, 2.10.1
• Component/s: fs/oss
• Assignee: wujinhu
• Reporter: wujinhu
• Created: 05/Feb/20 02:13
• Updated: 15/Apr/20 09:05
• Resolved: 09/Mar/20 05:07

Description

When hadoop lists object in versioning bucket with many delete marker in it, OSS will return

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult>
  <Name>select-us-east-1</Name>
  <Prefix>test/hadoop/file/</Prefix>
  <Marker></Marker>
  <MaxKeys>100</MaxKeys>
  <Delimiter>/</Delimiter>
  <IsTruncated>true</IsTruncated>
  <NextMarker>test/hadoop/file/sub2</NextMarker>
</ListBucketResult>

It sets IsTruncated to true and without ObjectSummaries or CommonPrefixes, and will throw FileNotFoundException

// code placeholder
java.io.FileNotFoundException: oss://select-us-east-1/test/hadoop/file: No such file or directory!java.io.FileNotFoundException: oss://select-us-east-1/test/hadoop/file: No such file or directory!
 at org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.getFileStatus(AliyunOSSFileSystem.java:281) at org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract.testGetFileStatusInVersioningBucket

 

// code placeholder
ObjectListing listing = store.listObjects(key, 1, null, false);
if (CollectionUtils.isNotEmpty(listing.getObjectSummaries()) ||
    CollectionUtils.isNotEmpty(listing.getCommonPrefixes())) {
  return new OSSFileStatus(0, true, 1, 0, 0, qualifiedPath, username);
} else {
  throw new FileNotFoundException(path + ": No such file or directory!");
}

 In this case, we should call listObjects until IsTruncated is false or ObjectSummaries is not empty or CommonPrefixes is not empty.