Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-8498

Insert into table misses some rows when vectorization is enabled

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.14.0, 0.13.1
    • Fix Version/s: 0.14.0
    • Component/s: Vectorization
    • Labels:

      Description

      Following is a small reproducible case for the issue

      create table orc1
      stored as orc
      tblproperties("orc.compress"="ZLIB")
      as
      select rn
      from
      (
      select cast(1 as int) as rn from src limit 1
      union all
      select cast(100 as int) as rn from src limit 1
      union all
      select cast(10000 as int) as rn from src limit 1
      ) t;

      create table orc_rn1 (rn int);
      create table orc_rn2 (rn int);
      create table orc_rn3 (rn int);

      // These inserts should produce 3 rows but only 1 row is produced
      from orc1 a
      insert overwrite table orc_rn1 select a.* where a.rn < 100
      insert overwrite table orc_rn2 select a.* where a.rn >= 100 and a.rn < 1000
      insert overwrite table orc_rn3 select a.* where a.rn >= 1000;

      select * from orc_rn1
      union all
      select * from orc_rn2
      union all
      select * from orc_rn3;

      The expected output of the query is
      1
      100
      10000

      But with vectorization enabled we get
      1

      1. HIVE-8498.4.patch
        16 kB
        Jitendra Nath Pandey
      2. HIVE-8498.3.patch
        16 kB
        Jitendra Nath Pandey
      3. HIVE-8498.02.patch
        34 kB
        Matt McCline
      4. HIVE-8498.01.patch
        34 kB
        Matt McCline

        Issue Links

          Activity

          Hide
          thejas Thejas M Nair added a comment -

          This has been fixed in 0.14 release. Please open new jira if you see any issues.

          Show
          thejas Thejas M Nair added a comment - This has been fixed in 0.14 release. Please open new jira if you see any issues.
          Hide
          hagleitn Gunther Hagleitner added a comment -

          Committed to trunk and branch. Thanks Jitendra Nath Pandey!

          Show
          hagleitn Gunther Hagleitner added a comment - Committed to trunk and branch. Thanks Jitendra Nath Pandey !
          Hide
          jnp Jitendra Nath Pandey added a comment -

          None of the test failures are related to this patch because they don't execute this code path.

          Show
          jnp Jitendra Nath Pandey added a comment - None of the test failures are related to this patch because they don't execute this code path.
          Hide
          hagleitn Gunther Hagleitner added a comment -

          +1 for .14

          Show
          hagleitn Gunther Hagleitner added a comment - +1 for .14
          Hide
          gopalv Gopal V added a comment -

          LGTM, adding to Prasanth's +1.

          Show
          gopalv Gopal V added a comment - LGTM, adding to Prasanth's +1.
          Hide
          hiveqa Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12678429/HIVE-8498.4.patch

          ERROR: -1 due to 4 failed/errored test(s), 6609 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
          org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection
          org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchCommit_Json
          org.apache.hive.minikdc.TestJdbcWithMiniKdc.testNegativeTokenAuth
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1578/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1578/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1578/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 4 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12678429 - PreCommit-HIVE-TRUNK-Build

          Show
          hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12678429/HIVE-8498.4.patch ERROR: -1 due to 4 failed/errored test(s), 6609 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchCommit_Json org.apache.hive.minikdc.TestJdbcWithMiniKdc.testNegativeTokenAuth Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1578/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1578/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1578/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed This message is automatically generated. ATTACHMENT ID: 12678429 - PreCommit-HIVE-TRUNK-Build
          Hide
          jnp Jitendra Nath Pandey added a comment -

          Updated patch addresses Gopal's comments, and also fixes the test failures.

          Show
          jnp Jitendra Nath Pandey added a comment - Updated patch addresses Gopal's comments, and also fixes the test failures.
          Hide
          jnp Jitendra Nath Pandey added a comment -

          The local copy of the vector needs to hold onto the selectedInUse as well.

          Good catch I will update the patch.

          Show
          jnp Jitendra Nath Pandey added a comment - The local copy of the vector needs to hold onto the selectedInUse as well. Good catch I will update the patch.
          Hide
          gopalv Gopal V added a comment -

          The local copy of the vector needs to hold onto the selectedInUse as well.

          Show
          gopalv Gopal V added a comment - The local copy of the vector needs to hold onto the selectedInUse as well.
          Hide
          hiveqa Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12678293/HIVE-8498.3.patch

          ERROR: -1 due to 5 failed/errored test(s), 6609 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_shufflejoin
          org.apache.hadoop.hive.ql.exec.vector.TestVectorFilterOperator.testBasicFilterOperator
          org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection
          org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchEmptyCommit
          org.apache.hive.minikdc.TestJdbcWithMiniKdc.testNegativeTokenAuth
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1565/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1565/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1565/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 5 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12678293 - PreCommit-HIVE-TRUNK-Build

          Show
          hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12678293/HIVE-8498.3.patch ERROR: -1 due to 5 failed/errored test(s), 6609 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_shufflejoin org.apache.hadoop.hive.ql.exec.vector.TestVectorFilterOperator.testBasicFilterOperator org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchEmptyCommit org.apache.hive.minikdc.TestJdbcWithMiniKdc.testNegativeTokenAuth Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1565/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1565/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1565/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed This message is automatically generated. ATTACHMENT ID: 12678293 - PreCommit-HIVE-TRUNK-Build
          Hide
          mmccline Matt McCline added a comment -

          Similar theme to the VectorSelectOperator restoring the projection arrays afterward.

          +1 (non-binding).

          Show
          mmccline Matt McCline added a comment - Similar theme to the VectorSelectOperator restoring the projection arrays afterward. +1 (non-binding).
          Hide
          jnp Jitendra Nath Pandey added a comment -

          The attached patch modifies the VectorFilterOperator to restore the selected vector before returning. This takes care of restoring the rowbatch after in-place filtering in the filter operator. A test is also included.

          Show
          jnp Jitendra Nath Pandey added a comment - The attached patch modifies the VectorFilterOperator to restore the selected vector before returning. This takes care of restoring the rowbatch after in-place filtering in the filter operator. A test is also included.
          Hide
          jnp Jitendra Nath Pandey added a comment -

          It might be ok to re-use the vectorization context on the branches of tablescan:
          1) Same rowbatch is passed to each branch as table scan is emitting same row-schema to each branch. The temporary columns are re-used across the branches. If same rowbatch can be made to work, there shouldn't be a reason to create a new context.
          2) A branch may have an operator that changes the rowbatch and might need a different vectorization context. However, this case is already handled using VectorizationContextRegion.

          This issue is being caused because we are not preserving the rowbatch to pass it to other children of table scan. The in-place filtering in the selected vector makes the rowbatch unusable for the other branches.
          I will post a patch with this fix shortly.

          Show
          jnp Jitendra Nath Pandey added a comment - It might be ok to re-use the vectorization context on the branches of tablescan: 1) Same rowbatch is passed to each branch as table scan is emitting same row-schema to each branch. The temporary columns are re-used across the branches. If same rowbatch can be made to work, there shouldn't be a reason to create a new context. 2) A branch may have an operator that changes the rowbatch and might need a different vectorization context. However, this case is already handled using VectorizationContextRegion. This issue is being caused because we are not preserving the rowbatch to pass it to other children of table scan. The in-place filtering in the selected vector makes the rowbatch unusable for the other branches. I will post a patch with this fix shortly.
          Hide
          hagleitn Gunther Hagleitner added a comment -

          Spoke to jitendra. The problem seems to be that the the vector context stuff is keyed off of a table scan. Moving that to table scan + index of child should fix the problem.

          Show
          hagleitn Gunther Hagleitner added a comment - Spoke to jitendra. The problem seems to be that the the vector context stuff is keyed off of a table scan. Moving that to table scan + index of child should fix the problem.
          Hide
          hagleitn Gunther Hagleitner added a comment -

          Tagging afaik only comes into play only for demux/mux. It might be easier to fix the multi insert case, especially since I know the event broadcast is already working (and you would disable this). The plan for this multi-insert query should be something like:

          ts -> fil[1] -> fs[1]
          -> fil[2] -> fs[2]
          -> fil[3] -> fs[3]

          The problem might be as simple as making sure the TS fowards to all it's children.
          It might, however, also be a case of the vectorization code not converting operators correctly.

          If it's simple, the best approach might be to put a fix for the multi-insert case, and disable correlation optimizer (tagging) when vectorization is on.

          Jitendra Nath Pandey do you have any insights?

          Show
          hagleitn Gunther Hagleitner added a comment - Tagging afaik only comes into play only for demux/mux. It might be easier to fix the multi insert case, especially since I know the event broadcast is already working (and you would disable this). The plan for this multi-insert query should be something like: ts -> fil [1] -> fs [1] -> fil [2] -> fs [2] -> fil [3] -> fs [3] The problem might be as simple as making sure the TS fowards to all it's children. It might, however, also be a case of the vectorization code not converting operators correctly. If it's simple, the best approach might be to put a fix for the multi-insert case, and disable correlation optimizer (tagging) when vectorization is on. Jitendra Nath Pandey do you have any insights?
          Hide
          mmccline Matt McCline added a comment -

          Jitendra Jitendra Nath Pandey told me a while ago the vectorization logic doesn't support / wasn't architected for tagging/multiple children. Part of this may be due to do with the shadow VectorizationContext data structures that track which columns of vectorized row batches for each vectorized operator.

          This JIRA is about multi insert queries basic functionality not working – only rows from first inset being processed. I don't know if the solution is difficult or not.

          Show
          mmccline Matt McCline added a comment - Jitendra Jitendra Nath Pandey told me a while ago the vectorization logic doesn't support / wasn't architected for tagging/multiple children. Part of this may be due to do with the shadow VectorizationContext data structures that track which columns of vectorized row batches for each vectorized operator. This JIRA is about multi insert queries basic functionality not working – only rows from first inset being processed. I don't know if the solution is difficult or not.
          Hide
          hagleitn Gunther Hagleitner added a comment -

          When I did the vectorized dynamic pruning stuff there was no problem with vectorization. It seems that the multi-child case is at least partially working. Do we know why the multi insert case is failing? The fix might not be that difficult. Is it?

          I can see how correlation optimizer might be more tricky. That one produces diamond shapes in the plan as far as I remember.

          Show
          hagleitn Gunther Hagleitner added a comment - When I did the vectorized dynamic pruning stuff there was no problem with vectorization. It seems that the multi-child case is at least partially working. Do we know why the multi insert case is failing? The fix might not be that difficult. Is it? I can see how correlation optimizer might be more tricky. That one produces diamond shapes in the plan as far as I remember.
          Hide
          hagleitn Gunther Hagleitner added a comment -

          Multi children happens in:

          • Multi insert queries
          • Dynamic partition pruning
          • Correlation optimizer

          For the first two it would really be better to have vectorization work. Can we fix the actual issues here instead of disabling this stuff wholesale?

          Show
          hagleitn Gunther Hagleitner added a comment - Multi children happens in: Multi insert queries Dynamic partition pruning Correlation optimizer For the first two it would really be better to have vectorization work. Can we fix the actual issues here instead of disabling this stuff wholesale?
          Hide
          hiveqa Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12675700/HIVE-8498.02.patch

          ERROR: -1 due to 2 failed/errored test(s), 6566 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_in_db
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1339/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1339/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1339/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 2 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12675700

          • PreCommit-HIVE-TRUNK-Build
          Show
          hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12675700/HIVE-8498.02.patch ERROR: -1 due to 2 failed/errored test(s), 6566 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_in_db org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_dynamic_partition_pruning Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1339/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1339/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1339/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12675700 PreCommit-HIVE-TRUNK-Build
          Hide
          mmccline Matt McCline added a comment -

          The failure of vectorized_dynamic_partition_pruning.q could be related to the changes, but it runs successfully on my laptop.

          Rebased patch and resubmitting.

          Show
          mmccline Matt McCline added a comment - The failure of vectorized_dynamic_partition_pruning.q could be related to the changes, but it runs successfully on my laptop. Rebased patch and resubmitting.
          Hide
          hiveqa Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12675603/HIVE-8498.01.patch

          ERROR: -1 due to 2 failed/errored test(s), 6566 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_in_db
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1331/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1331/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1331/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 2 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12675603

          • PreCommit-HIVE-TRUNK-Build
          Show
          hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12675603/HIVE-8498.01.patch ERROR: -1 due to 2 failed/errored test(s), 6566 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_in_db org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_dynamic_partition_pruning Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1331/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1331/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1331/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12675603 PreCommit-HIVE-TRUNK-Build
          Hide
          mmccline Matt McCline added a comment -

          The follow up JIRA to vectorize operators with multiple children is HIVE-8513.

          Show
          mmccline Matt McCline added a comment - The follow up JIRA to vectorize operators with multiple children is HIVE-8513 .
          Hide
          prasanth_j Prasanth Jayachandran added a comment -

          LGTM, +1. Pending unit tests.

          This means that correlation operators Mux/Demux will also not work right? Can you create a follow up jira for supporting multiple children and link it to this jira?

          Show
          prasanth_j Prasanth Jayachandran added a comment - LGTM, +1. Pending unit tests. This means that correlation operators Mux/Demux will also not work right? Can you create a follow up jira for supporting multiple children and link it to this jira?
          Hide
          mmccline Matt McCline added a comment -

          Fix turns off multiple children operators for now.

          Show
          mmccline Matt McCline added a comment - Fix turns off multiple children operators for now.

            People

            • Assignee:
              jnp Jitendra Nath Pandey
              Reporter:
              prasanth_j Prasanth Jayachandran
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development