[HADOOP-16433] S3Guard: Filter expired entries and tombstones when listing with MetadataStore#listChildren

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Component/s: fs/s3
• Target Version/s: 3.3.0
• Assignee: Gabor Bota
• Reporter: Gabor Bota
• Created: 16/Jul/19 16:40
• Updated: 24/Jul/19 18:32
• Resolved: 24/Jul/19 17:12

Description

Currently, we don't filter out entries in listChildren implementations.

This can cause bugs and inconsistencies, so this should be fixed.
It can lead to a status where we can't recover from the following:
guarded and raw (OOB op) clients are doing ops to S3

Guarded: touch /AAAA
Guarded: touch /ZZZZ
Guarded: rm /AAAA {{-> tombstone in MS}}
RAW: touch /AAAA/file.ext {{-> file is hidden with a tombstone}}
Guarded: ls / {{-> only ZZZZ will show up in the listing. }}

After we change the following code

          final List<PathMetadata> metas = new ArrayList<>();
          for (Item item : items) {
            DDBPathMetadata meta = itemToPathMetadata(item, username);
            metas.add(meta);
          }

to

            // handle expiry - only add not expired entries to listing.
            if (meta.getLastUpdated() == 0 ||
                !meta.isExpired(ttlTimeProvider.getMetadataTtl(),
                ttlTimeProvider.getNow())) {
              metas.add(meta);
            }

we will filter out expired entries from the listing, so we can recover form these kind of OOB ops.

Note: we have to handle the lastUpdated == 0 case, where the lastUpdated field is not filled in!
Note: this can only be fixed cleanly after HADOOP-16383 is fixed because we need to have the TTLtimeProvider in MS to handle this internally.

Issue Links

• Dependent: HADOOP-16383 Pass ITtlTimeProvider instance in initialize method in MetadataStore interface
• links to: GitHub Pull Request #1134