[APEXMALHAR-2237] Dynamic partitioning support for FSInputModule

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.6.0
• Assignee: Yogi Devendra
• Reporter: Yogi Devendra
• Created: 14/Sep/16 10:05
• Updated: 02/Dec/16 21:11
• Resolved: 24/Oct/16 11:50

Description

Currently, FSInputModule exposes readersCount property which is used for static partitioning of BlockReader.

To support dynamic partitioning, following additional properties will be exposed.
1. minReaders = minimum number of readers for dynamic partitioning.
2. maxReaders = maximum number of readers for dynamic partitioning.
3. repartitionCheckInterval = time in seconds for checking if repartition is required.

minReaders, maxReaders will set respective property on BlockReader.
repartitionCheckInterval will set intervalMillis property on BlockReader after converting from seconds to milliseconds.

Issue Links

• links to: GitHub Pull Request #428