Description
in production envrinment, we use kudu-spark to read kudu table, but even we don't use the
HdrHistogramAccumulator, the HdrHistogramAccumulator stored in an array is stiil so big,
totoal of them are almost 2 MB, so that when the number of kudu-spark task(for read kudu data and shuffle) is more than 900, the spark job failed, and the follwing error occured,
Job aborted due to stage failure: Total size of serialized results of 1413 tasks (3.0 GB) is bigger than spark.driver.maxResultSize (3.0 GB)
Attachments
Attachments
Issue Links
- is duplicated by
-
KUDU-3054 Init kudu.write_duration accumulator lazily
- Resolved