Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-3054

Init kudu.write_duration accumulator lazily

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 1.9.0
    • Fix Version/s: NA
    • Component/s: spark
    • Labels:
      None

      Description

      Currently, we encountered a issue in kudu-spark that will causing spark sql query failure:

      ```

      Job aborted due to stage failure: Total size of serialized results of 942 tasks (2.0 GB) is bigger than spark.driver.maxResultSize (2.0 GB)

      ```

      After carefully debug, we find out that it's the kudu.write_duration accumulators causing single spark task larger than 2M, thus all tasks size of the stage will bigger than the limit.

      However, this stage is just reading kudu table and do shuffle exchange, no writing any kudu tables.

      So I think should init this accumulator lazily in KuduContext to avoid such issues.

       

        Attachments

        1. read_kudu_and_shuffle.png
          150 kB
          liupengcheng
        2. durationhisto.png
          83 kB
          liupengcheng
        3. durationHisto_large.png
          209 kB
          liupengcheng

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                liupengcheng liupengcheng
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h