[HADOOP-13985] s3guard: add a version marker to every table

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: HADOOP-13345
• Fix Version/s: HADOOP-13345
• Component/s: fs/s3
• Target Version/s: HADOOP-13345
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 13/Jan/17 12:57
• Updated: 27/Jan/17 09:43
• Resolved: 27/Jan/17 09:43

Description

This is something else we need before any preview: a way to identify a table version, so that if future versions change the table structure:

• older clients can recognise that it's a newer format, and fail
• the future version can identify that it's an older format, and fail until some fsck-upgrade operation has taken place

I think something like a row on a path which is impossible in a real filesystem, such as "../VERSION" would allow a version marker to go in; the length field could be abused for the version number.

This field would be something that'd be checked in init(), so be the simple test for table existence we need for faster init

Issue Links

• incorporates: HADOOP-13959 S3guard: replace dynamo.describe() call in init with more efficient query
• is depended upon by: HADOOP-13998 Merge initial S3guard release into trunk
• is related to: HADOOP-13959 S3guard: replace dynamo.describe() call in init with more efficient query