Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6235

Address various 2G limits

          Details

          • Type: Umbrella
          • Status: Open
          • Priority: Major
          • Resolution: Unresolved
          • Affects Version/s: None
          • Fix Version/s: None
          • Component/s: Shuffle, Spark Core
          • Labels:
            None

            Description

            An umbrella ticket to track the various 2G limit we have in Spark, due to the use of byte arrays and ByteBuffers.

              Attachments

                Issue Links

                is duplicated by

                Bug - A problem which impairs or prevents the functions of the product. SPARK-22622 OutOfMemory thrown by Closure Serializer without proper failure propagation

                • Critical - Crashes, loss of data, severe memory leak.
                • Resolved

                Improvement - An improvement or enhancement to an existing feature or task. SPARK-2755 TorrentBroadcast cannot broadcast very large objects

                • Major - Major loss of function.
                • Resolved

                Improvement - An improvement or enhancement to an existing feature or task. SPARK-1391 BlockManager cannot transfer blocks larger than 2G in size

                • Major - Major loss of function.
                • Closed
                is related to

                Bug - A problem which impairs or prevents the functions of the product. SPARK-22352 task failures with java.lang.IllegalArgumentException: Size exceeds Integer.MAX_VALUE error

                • Major - Major loss of function.
                • Resolved

                Bug - A problem which impairs or prevents the functions of the product. SPARK-22062 BlockManager does not account for memory consumed by remote fetches

                • Minor - Minor loss of function, or other problem where easy workaround is present.
                • Resolved
                links to

                Pull request #14647 [Github] Pull Request #14647 (witgo)

                Pull request #14977 [Github] Pull Request #14977 (witgo)

                Pull request #14995 [Github] Pull Request #14995 (witgo)

                Web Link Design doc

                  Activity

                    People

                    • Assignee:
                      Unassigned
                      Reporter:
                      rxin Reynold Xin
                    • Votes:
                      58 Vote for this issue
                      Watchers:
                      120 Start watching this issue

                      Dates

                      • Created:
                        Updated: