Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-23883

Streaming does not flush the side file

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Streaming, Transactions
    • Labels:
      None

      Description

      When a streaming write commits a mid-batch write with connection.commitTransaction() then it tries to flush the sideFile with OrcInputFormat.SHIMS.hflush(flushLengths). This uses FSOutputSummer.flush, which does not flush the buffer data to the disk so the actual data is not written.

      Had to remove the check from the end of the streaming tests in TestCrudCompactorOnTez.java

            CompactorTestUtilities.checkAcidVersion(fs.listFiles(new Path(table.getSd().getLocation()), true), fs,
                conf.getBoolVar(HiveConf.ConfVars.HIVE_WRITE_ACID_VERSION_FILE),
                new String[] { AcidUtils.DELTA_PREFIX });
      

      These checks verifies the _flush_length files, and they would fail otherwise.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              pvary Peter Vary
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: