[GOBBLIN-381] Add ability to filter hidden directories for ConfigBasedDatasets

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.11.0, 0.12.0, 0.13.0
• Fix Version/s: 0.12.0
• Component/s: distcp-ng
• Assignee: Sudarshan Vasudevan
• Reporter: Sudarshan Vasudevan
• Created: 22/Jan/18 23:18
• Updated: 21/Mar/18 11:44
• Resolved: 21/Mar/18 09:11

Description

This is related to GOBBLIN-368. ConfigBasedDatasets do not have the ability to apply path filters to directories. Similar to RecursiveCopyableDataset, ConfigBasedDatasets should filter files and/or directories based on the following configuration keys:

gobblin.copy.applyFilterToDirectories

gobblin.dataset.path.filter.class