Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
exposed by: HIVE-18359
in case of vectorization, the summary row object was left as is (presumed null earlier); which may cause it to be inconsistent isNull conditions in .VectorHashKeyWrapperBatch
issue happens only if:
- vectorizable groupby
- groupping set contains empty
- non-trivial empty; mapper is run
- groupping key is select ; with a type which is backed by a bytea; ex:string
set hive.vectorized.execution.enabled=true; create table tx2 (a integer,b integer,c integer,d double,u string,bi binary) stored as orc; insert into tx2 values (1,2,3,1.1,'x','b'), (3,2,3,1.1,'y','b'); select sum(a), u, bi, 'asd', grouping(bi), 'NULL,1' as expected from tx2 where a=2 group by a,u,bi grouping sets ( u, (), bi);
causes:
Caused by: java.lang.NullPointerException
at java.lang.System.arraycopy(Native Method)
at org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173)
at org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapperBatch.assignRowColumn(VectorHashKeyWrapperBatch.java:1065)
at org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.writeSingleRow(VectorGroupByOperator.java:1134)
at org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.access$800(VectorGroupByOperator.java:74)
at org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeReduceMergePartial.close(VectorGroupByOperator.java:862)
at org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.closeOp(VectorGroupByOperator.java:1176)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:705)
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:383)
... 16 more
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1515531021543_0001_12_01 [Reducer 2] killed/failed due to:OWN_TASK_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
2018-01-09T12:50:30,611 DEBUG [01fdcefd-40b0-45a6-8e5b-b1cd14241088 main] ql.Driver: Shutting down query
Attachments
Attachments
- HIVE-18413.01.patch
- 4 kB
- Zoltan Haindrich
- HIVE-18413.02.patch
- 7 kB
- Zoltan Haindrich
- HIVE-18413.03.patch
- 10 kB
- Zoltan Haindrich
Issue Links
- is caused by
-
HIVE-17617 Rollup of an empty resultset should contain the grouping of the empty grouping set
- Closed
Activity
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
Prechecks | |||
0 | findbugs | 0m 0s | Findbugs executables are not available. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
master Compile Tests | |||
+1 | mvninstall | 6m 28s | master passed |
+1 | compile | 0m 55s | master passed |
+1 | checkstyle | 0m 31s | master passed |
+1 | javadoc | 0m 48s | master passed |
Patch Compile Tests | |||
+1 | mvninstall | 1m 13s | the patch passed |
+1 | compile | 0m 52s | the patch passed |
+1 | javac | 0m 52s | the patch passed |
+1 | checkstyle | 0m 32s | ql: The patch generated 0 new + 122 unchanged - 5 fixed = 122 total (was 127) |
-1 | whitespace | 0m 0s | The patch 5 line(s) with tabs. |
+1 | javadoc | 0m 52s | the patch passed |
Other Tests | |||
+1 | asflicense | 0m 12s | The patch does not generate ASF License warnings. |
12m 39s |
Subsystem | Report/Notes |
---|---|
Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
Build tool | maven |
Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
git revision | master / 5042398 |
Default Java | 1.8.0_111 |
whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-8529/yetus/whitespace-tabs.txt |
modules | C: ql U: ql |
Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-8529/yetus.txt |
Powered by | Apache Yetus http://yetus.apache.org |
This message was automatically generated.
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12905343/HIVE-18413.02.patch
SUCCESS: +1 due to 1 test(s) being added or modified.
ERROR: -1 due to 21 failed/errored test(s), 11550 tests executed
Failed tests:
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_rollup_empty] (batchId=55) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook] (batchId=12) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_ppd_decimal] (batchId=9) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2] (batchId=151) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2] (batchId=156) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=164) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] (batchId=168) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast] (batchId=159) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=159) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part] (batchId=93) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1] (batchId=93) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_publisher_error_1] (batchId=93) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] (batchId=120) org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation (batchId=213) org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253) org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints (batchId=225) org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231) org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231) org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8529/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8529/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8529/
Messages:
Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 21 tests failed
This message is automatically generated.
ATTACHMENT ID: 12905343 - PreCommit-HIVE-Build
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
Prechecks | |||
0 | findbugs | 0m 0s | Findbugs executables are not available. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
master Compile Tests | |||
+1 | mvninstall | 6m 51s | master passed |
+1 | compile | 0m 59s | master passed |
+1 | checkstyle | 0m 33s | master passed |
+1 | javadoc | 0m 51s | master passed |
Patch Compile Tests | |||
+1 | mvninstall | 1m 21s | the patch passed |
+1 | compile | 1m 0s | the patch passed |
+1 | javac | 1m 0s | the patch passed |
+1 | checkstyle | 0m 35s | ql: The patch generated 0 new + 122 unchanged - 5 fixed = 122 total (was 127) |
-1 | whitespace | 0m 0s | The patch 5 line(s) with tabs. |
+1 | javadoc | 0m 58s | the patch passed |
Other Tests | |||
+1 | asflicense | 0m 12s | The patch does not generate ASF License warnings. |
13m 36s |
Subsystem | Report/Notes |
---|---|
Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
Build tool | maven |
Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
git revision | master / d69601c |
Default Java | 1.8.0_111 |
whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-8539/yetus/whitespace-tabs.txt |
modules | C: ql U: ql |
Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-8539/yetus.txt |
Powered by | Apache Yetus http://yetus.apache.org |
This message was automatically generated.
#3)
- add missed update of TestCliDriver version of groupby_rollup_empty
- other failures can't be reproduced
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12905343/HIVE-18413.02.patch
SUCCESS: +1 due to 1 test(s) being added or modified.
ERROR: -1 due to 18 failed/errored test(s), 11551 tests executed
Failed tests:
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_rollup_empty] (batchId=55) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook] (batchId=12) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2] (batchId=151) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2] (batchId=156) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=164) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] (batchId=168) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast] (batchId=159) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin] (batchId=164) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=159) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part] (batchId=93) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] (batchId=120) org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation (batchId=213) org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253) org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints (batchId=225) org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231) org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231) org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8539/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8539/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8539/
Messages:
Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 18 tests failed
This message is automatically generated.
ATTACHMENT ID: 12905343 - PreCommit-HIVE-Build
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
Prechecks | |||
0 | findbugs | 0m 1s | Findbugs executables are not available. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
master Compile Tests | |||
+1 | mvninstall | 7m 11s | master passed |
+1 | compile | 1m 5s | master passed |
+1 | checkstyle | 0m 35s | master passed |
+1 | javadoc | 0m 52s | master passed |
Patch Compile Tests | |||
+1 | mvninstall | 1m 17s | the patch passed |
+1 | compile | 1m 0s | the patch passed |
+1 | javac | 1m 0s | the patch passed |
+1 | checkstyle | 0m 34s | ql: The patch generated 0 new + 122 unchanged - 5 fixed = 122 total (was 127) |
-1 | whitespace | 0m 0s | The patch 5 line(s) with tabs. |
+1 | javadoc | 0m 59s | the patch passed |
Other Tests | |||
+1 | asflicense | 0m 12s | The patch does not generate ASF License warnings. |
14m 5s |
Subsystem | Report/Notes |
---|---|
Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
Build tool | maven |
Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
git revision | master / 022a95e |
Default Java | 1.8.0_111 |
whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-8556/yetus/whitespace-tabs.txt |
modules | C: ql U: ql |
Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-8556/yetus.txt |
Powered by | Apache Yetus http://yetus.apache.org |
This message was automatically generated.
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12905407/HIVE-18413.03.patch
SUCCESS: +1 due to 1 test(s) being added or modified.
ERROR: -1 due to 19 failed/errored test(s), 11551 tests executed
Failed tests:
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynpart_sort_opt_bucketing] (batchId=85) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook] (batchId=12) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2] (batchId=151) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2] (batchId=156) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=164) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] (batchId=168) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast] (batchId=159) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=159) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part] (batchId=93) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1] (batchId=93) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] (batchId=120) org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation (batchId=213) org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253) org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints (batchId=225) org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231) org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231) org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8556/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8556/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8556/
Messages:
Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 19 tests failed
This message is automatically generated.
ATTACHMENT ID: 12905407 - PreCommit-HIVE-Build
kgyrtkirk Does this also fix the empty grouping sets issue that was discussed in HIVE-18359?
This jira is resolved and released with Hive 3.0 If you find an issue with it, please create a new jira.
#1)