[GOBBLIN-397] Create a new dataset version selection policy for filtering dataset versions that have "hidden" paths

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.11.0, 0.12.0, 0.13.0
• Fix Version/s: 0.12.0
• Component/s: distcp-ng
• Assignee: Sudarshan Vasudevan
• Reporter: Sudarshan Vasudevan
• Created: 31/Jan/18 05:24
• Updated: 21/Mar/18 11:43
• Resolved: 31/Jan/18 22:01

Description

We need the ability to filter out dataset versions that represent "hidden" paths i.e. paths which contain directories that start with "." or "_" (e.g. /data/logs/_temp/2018/01/01).