Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6235

Address various 2G limits

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Umbrella
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.4.0
    • Component/s: Shuffle, Spark Core
    • Labels:
      None

      Description

      An umbrella ticket to track the various 2G limit we have in Spark, due to the use of byte arrays and ByteBuffers.

        Attachments

        Issue Links

        is duplicated by

        Bug - A problem which impairs or prevents the functions of the product. SPARK-22622 OutOfMemory thrown by Closure Serializer without proper failure propagation

        • Critical - Crashes, loss of data, severe memory leak.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. SPARK-2755 TorrentBroadcast cannot broadcast very large objects

        • Major - Major loss of function.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. SPARK-1391 BlockManager cannot transfer blocks larger than 2G in size

        • Major - Major loss of function.
        • Closed
        is related to

        Bug - A problem which impairs or prevents the functions of the product. SPARK-22352 task failures with java.lang.IllegalArgumentException: Size exceeds Integer.MAX_VALUE error

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. SPARK-22062 BlockManager does not account for memory consumed by remote fetches

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved
        links to

        Pull request #14647 [Github] Pull Request #14647 (witgo)

        Pull request #14977 [Github] Pull Request #14977 (witgo)

        Pull request #14995 [Github] Pull Request #14995 (witgo)

        Web Link Design doc

        There are no Sub-Tasks for this issue.

          Activity
          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

            • Assignee:
              Unassigned
              Reporter:
              rxin Reynold Xin

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment