[HADOOP-15400] Improve S3Guard documentation on Authoritative Mode implementation

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.1
• Fix Version/s: 3.2.0
• Component/s: fs/s3
• Target Version/s: 3.2.0
• Assignee: Gabor Bota
• Reporter: Aaron Fabbri
• Created: 19/Apr/18 21:15
• Updated: 08/Aug/18 04:52
• Resolved: 08/Aug/18 02:14

Description

Part of the design of S3Guard is support for skipping the call to S3 listObjects and serving directory listings out of the MetadataStore under certain circumstances.  This feature is called "authoritative" mode.  I've talked to many people about this feature and it seems to be universally confusing.

I suggest we improve / add a section to the s3guard.md site docs elaborating on what Authoritative Mode is.

It is not treating the MetadataStore (e.g. dynamodb) as the source of truth in general.

It is the ability to short-circuit S3 list objects and serve listings from the MetadataStore in some circumstances: 

For S3A to skip S3's list objects on some path, and serve it directly from the MetadataStore, the following things must all be true:

1. The MetadataStore implementation persists the bit DirListingMetadata.isAuthorititative set when calling MetadataStore#put(DirListingMetadata)
2. The S3A client is configured to allow metadatastore to be authoritative source of a directory listing (fs.s3a.metadatastore.authoritative=true).
3. The MetadataStore has a full listing for path stored in it.  This only happens if the FS client (s3a) explicitly has stored a full directory listing with DirListingMetadata.isAuthorititative=true before the said listing request happens.

Note that #1 only currently happens in LocalMetadataStore. Adding support to DynamoDBMetadataStore is covered in HADOOP-14154.

Also, the multiple uses of the word "authoritative" are confusing. Two meanings are used:
1. In the FS client configuration fs.s3a.metadatastore.authoritative

• Behavior of S3A code (not MetadataStore)
• "S3A is allowed to skip S3.list() when it has full listing from MetadataStore"

2. MetadataStore
When storing a dir listing, can set a bit isAuthoritative
1 : "full contents of directory"
0 : "may not be full listing"

Note that a MetadataStore MAY persist this bit. (not MUST).

We should probably rename the DirListingMetadata.isAuthorititative to .fullListing or at least put a comment where it is used to clarify its meaning.

Issue Links

• relates to: HADOOP-14154 Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)