Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-1201

Do not materialize partitions whenever possible in BlockManager

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • None
    • None
    • Block Manager, Spark Core
    • None

    Description

      This is a slightly more complex version of SPARK-942 where we try to avoid unrolling iterators in other situations where it is possible. SPARK-942 focused on the case where the DISK_ONLY storage level was used. There are other cases though, such as if data is stored serialized and in memory and but there is not enough memory left to store the RDD.

      Attachments

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-1777 Pass "cached" blocks directly to disk if memory is not large enough

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-942 Do not materialize partitions when DISK_ONLY storage level is used

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              andrewor14 Andrew Or
              pwendell Patrick Wendell
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: