Description
Currently the KuduBackup job uses 1 scanner and therefore 1 Spark task per Kudu partition. When KUDU-2670 is complete, we should consider and test having more than one scanner per partition and instead configuring a target data size for each scanner. That should result in faster and more reliable/predictable backup jobs regardless of partition count.
It may however make restoring more difficult because it could cause compactions. Restore side testing and improvements may also be required.
Improvements to the estimation for key range sizes may also need to be done, so this should be well tested.
Attachments
Issue Links
- depends upon
-
KUDU-2670 Splitting more tasks for spark job, and add more concurrent for scan operation
-
- Open
-