Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-2138

The KMeans algorithm in the MLlib can lead to the Serialized Task size become bigger and bigger

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Not A Problem
    • 0.9.0, 0.9.1
    • None
    • MLlib

    Description

      When the algorithm running at certain stage, when running the reduceBykey() function, It can lead to Executor Lost and Task lost, after several times. the application exit.

      When this error occurred, the size of serialized task is bigger than 10MB, and the size become larger as the iteration increase.

      the data generation file: https://gist.github.com/djvulee/7e3b2c9eb33ff0037622

      the running code: https://gist.github.com/djvulee/6bf00e60885215e3bfd5

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-1112 When spark.akka.frameSize > 10, task results bigger than 10MiB block execution

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-3424 KMeans Plus Plus is too slow

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              mengxr Xiangrui Meng
              DjvuLee DjvuLee
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: