Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.0.0, 1.0.1
    • Fix Version/s: 1.1.0
    • Component/s: PySpark
    • Labels:
    • Target Version/s:

      Description

      For huge reduce tasks, user will got out of memory exception when all the data can not fit in memory.

      It should put some of the data into disks and then merge them together, just like what we do in Scala.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                davies Davies Liu
                Reporter:
                davies Davies Liu
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 72h
                  72h
                  Remaining:
                  Remaining Estimate - 72h
                  72h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified