Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-16923 Hive-on-Spark DPP Improvements
  3. HIVE-17959

SparkPartitionPruningSinkOperator buffers all writes in memory

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Spark
    • None

    Description

      SparkPartitionPruningSinkOperator buffers everything in memory and only flushes to the output file with the operator is closed. This can cause memory issues.

      Attachments

        Activity

          People

            janulatha Janaki Lahorani
            stakiar Sahil Takiar
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: