Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-4410

Support for external sort

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2.0
    • Component/s: SQL
    • Labels:
      None
    • Target Version/s:

      Description

      When any given key is of too high cardinality the current sorting code can tip over (since it loads a whole partition into memory). I propose we add optional support for using sparks built in external sorting mechanism. It can be off by default, but if we determine this code path does not regress performance we can turn it on by default in the future.

        Attachments

          Activity

            People

            • Assignee:
              marmbrus Michael Armbrust
              Reporter:
              marmbrus Michael Armbrust
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: