Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-1006

MLlib ALS gets stack overflow with too many iterations

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Duplicate
    • None
    • None
    • MLlib
    • None

    Description

      The tipping point seems to be around 50. We should fix this by checkpointing the RDDs every 10-20 iterations to break the lineage chain, but checkpointing currently requires HDFS installed, which not all users will have.

      We might also be able to fix DAGScheduler to not be recursive.

      Attachments

        Issue Links

          Is contained by

          New Feature - A new feature of the product, which has yet to be developed. SPARK-5955 Add checkpointInterval to ALS

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. SPARK-958 When iteration in ALS increases to 10 running in local mode, spark throws out error of StackOverflowError

          • Major - Major loss of function.
          • Resolved

          Question - A formal question. Initially added for the Legal JIRA. SPARK-3370 The simple test error

          • Major - Major loss of function.
          • Closed
          is related to

          New Feature - A new feature of the product, which has yet to be developed. SPARK-1855 Provide memory-and-local-disk RDD checkpointing

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              matei Matei Alexandru Zaharia
              Votes:
              3 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: