Major Description
The SparkDynamicPartitionPruner first loads the contents of each partition pruning file into memory, and then prunes all the partitions from the MapWork. This can cause increased memory pressure on the HoS Remote Driver because it requires loading all the partition metadata into memory. It would be more efficient if pruning of partitions was done while scanning the files, so that all the partition metadata doesn't need to be buffered in memory.