Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-9743

Incorrect result set for vectorized left outer join

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.14.0
    • Fix Version/s: 1.2.0
    • Component/s: SQL
    • Labels:
      None

      Description

      This query is supposed to return 3 rows and will when run without Tez but returns 2 rows when run with Tez.

      select tjoin1.rnum, tjoin1.c1, tjoin1.c2, tjoin2.c2 as c2j2 from tjoin1 left outer join tjoin2 on ( tjoin1.c1 = tjoin2.c1 and tjoin1.c2 > 15 )

      tjoin1.rnum tjoin1.c1 tjoin1.c2 c2j2
      1 20 25 <null>
      2 <null> 50 <null>

      instead of

      tjoin1.rnum tjoin1.c1 tjoin1.c2 c2j2
      0 10 15 <null>
      1 20 25 <null>
      2 <null> 50 <null>

      create table if not exists TJOIN1 (RNUM int , C1 int, C2 int)
      STORED AS orc ;

      0|10|15
      1|20|25
      2|\N|50

      create table if not exists TJOIN2 (RNUM int , C1 int, C2 char(2))
      ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n'
      STORED AS TEXTFILE ;

      0|10|BB
      1|15|DD
      2|\N|EE
      3|10|FF

      1. HIVE-9743.01.patch
        23 kB
        Matt McCline
      2. HIVE-9743.02.patch
        27 kB
        Matt McCline
      3. HIVE-9743.03.patch
        37 kB
        Matt McCline
      4. HIVE-9743.04.patch
        74 kB
        Matt McCline
      5. HIVE-9743.05.patch
        73 kB
        Matt McCline
      6. HIVE-9743.06.patch
        1.22 MB
        Matt McCline
      7. HIVE-9743.08.patch
        1.19 MB
        Matt McCline
      8. HIVE-9743.09.patch
        1.15 MB
        Matt McCline
      9. HIVE-9743.091.patch
        1.15 MB
        Matt McCline

        Activity

        Hide
        sushanth Sushanth Sowmyan added a comment -

        This issue has been fixed and released as part of the 1.2.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

        Show
        sushanth Sushanth Sowmyan added a comment - This issue has been fixed and released as part of the 1.2.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.
        Hide
        vikram.dixit Vikram Dixit K added a comment -

        Thanks Matt and Jason.

        Show
        vikram.dixit Vikram Dixit K added a comment - Thanks Matt and Jason.
        Hide
        jdere Jason Dere added a comment -

        Committed to branch-1.2/master

        Show
        jdere Jason Dere added a comment - Committed to branch-1.2/master
        Hide
        mmccline Matt McCline added a comment -

        None of the test failures are related to my changes.

        Show
        mmccline Matt McCline added a comment - None of the test failures are related to my changes.
        Hide
        vikram.dixit Vikram Dixit K added a comment -

        +1

        Show
        vikram.dixit Vikram Dixit K added a comment - +1
        Hide
        hiveqa Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12730767/HIVE-9743.091.patch

        ERROR: -1 due to 24 failed/errored test(s), 8904 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_parts
        org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
        org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static
        org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_unencrypted_tbl
        org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys
        org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_load_data_to_encrypted_tables
        org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_select_read_only_encrypted_tbl
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_disallow_transform
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_droppartition
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_sba_drop_table
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_alterpart_loc
        org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
        org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase
        org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
        org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable
        org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropView
        org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
        org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
        org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure
        org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
        org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
        org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccess
        org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.TestSQLStdHiveAccessControllerHS2.testConfigProcessing
        org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.TestSQLStdHiveAccessControllerHS2.testConfigProcessingCustomSetWhitelistAppend
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3785/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3785/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3785/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 24 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12730767 - PreCommit-HIVE-TRUNK-Build

        Show
        hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12730767/HIVE-9743.091.patch ERROR: -1 due to 24 failed/errored test(s), 8904 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_parts org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_unencrypted_tbl org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_load_data_to_encrypted_tables org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_select_read_only_encrypted_tbl org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_disallow_transform org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_droppartition org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_sba_drop_table org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_alterpart_loc org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropView org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccess org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.TestSQLStdHiveAccessControllerHS2.testConfigProcessing org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.TestSQLStdHiveAccessControllerHS2.testConfigProcessingCustomSetWhitelistAppend Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3785/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3785/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3785/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 24 tests failed This message is automatically generated. ATTACHMENT ID: 12730767 - PreCommit-HIVE-TRUNK-Build
        Hide
        mmccline Matt McCline added a comment -

        Forgot to delete unneeded vector_leftsemi_join_orig.q file.

        Show
        mmccline Matt McCline added a comment - Forgot to delete unneeded vector_leftsemi_join_orig.q file.
        Hide
        hiveqa Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12730688/HIVE-9743.09.patch

        ERROR: -1 due to 27 failed/errored test(s), 8905 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_parts
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables_compact
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_leftsemi_mapjoin_orig
        org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
        org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static
        org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_unencrypted_tbl
        org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys
        org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_load_data_to_encrypted_tables
        org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_select_read_only_encrypted_tbl
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_disallow_transform
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_droppartition
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_sba_drop_table
        org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_alterpart_loc
        org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
        org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase
        org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
        org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable
        org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropView
        org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
        org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
        org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure
        org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
        org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
        org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccess
        org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.TestSQLStdHiveAccessControllerHS2.testConfigProcessing
        org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.TestSQLStdHiveAccessControllerHS2.testConfigProcessingCustomSetWhitelistAppend
        org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchEmptyCommit
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3750/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3750/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3750/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 27 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12730688 - PreCommit-HIVE-TRUNK-Build

        Show
        hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12730688/HIVE-9743.09.patch ERROR: -1 due to 27 failed/errored test(s), 8905 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_parts org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables_compact org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_leftsemi_mapjoin_orig org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_unencrypted_tbl org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_load_data_to_encrypted_tables org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_select_read_only_encrypted_tbl org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_disallow_transform org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_droppartition org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_sba_drop_table org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_alterpart_loc org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropView org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccess org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.TestSQLStdHiveAccessControllerHS2.testConfigProcessing org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.TestSQLStdHiveAccessControllerHS2.testConfigProcessingCustomSetWhitelistAppend org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchEmptyCommit Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3750/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3750/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3750/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 27 tests failed This message is automatically generated. ATTACHMENT ID: 12730688 - PreCommit-HIVE-TRUNK-Build
        Hide
        mmccline Matt McCline added a comment -

        Vikram Dixit K Ok, SMB removed. I think this one is good to go as soon as the Apache tests pass.

        Show
        mmccline Matt McCline added a comment - Vikram Dixit K Ok, SMB removed. I think this one is good to go as soon as the Apache tests pass.
        Hide
        mmccline Matt McCline added a comment -

        Given lack of time, I think I'll pull the SMB changes since the regular map join case repro is very clear.

        Show
        mmccline Matt McCline added a comment - Given lack of time, I think I'll pull the SMB changes since the regular map join case repro is very clear.
        Hide
        vikram.dixit Vikram Dixit K added a comment -

        That seems to be because with SMB there seems to be full delegation to the base class. I am not sure if we need the SMB changes at all.

        Show
        vikram.dixit Vikram Dixit K added a comment - That seems to be because with SMB there seems to be full delegation to the base class. I am not sure if we need the SMB changes at all.
        Hide
        mmccline Matt McCline added a comment -

        Vikram Dixit K I removed the annotations and the MR vector_left_outer_join3.q.out and fiddled with environment variables so that it now has "Sorted Merge Bucket Map Join Operator" operators; Tez has "Merge Join Operator" as you said.

        The original LEFT OUTER JOIN problem does not repro with vector_left_outer_join3.q though.

        Show
        mmccline Matt McCline added a comment - Vikram Dixit K I removed the annotations and the MR vector_left_outer_join3.q.out and fiddled with environment variables so that it now has "Sorted Merge Bucket Map Join Operator" operators; Tez has "Merge Join Operator" as you said. The original LEFT OUTER JOIN problem does not repro with vector_left_outer_join3.q though.
        Hide
        vikram.dixit Vikram Dixit K added a comment - - edited

        Matt McCline I checked the output of the test and it looks like queries have hints added to them and the explain outputs have gone missing? In tez, the equivalent for the Sort Merge Bucket Map Join Operator would be the Merge Join Operator in the MapOperator. Can you remove hints and enable auto conversion to smb join so that I can verify the plans. Take a look at tez_smb_1.q for e.g.

        Show
        vikram.dixit Vikram Dixit K added a comment - - edited Matt McCline I checked the output of the test and it looks like queries have hints added to them and the explain outputs have gone missing? In tez, the equivalent for the Sort Merge Bucket Map Join Operator would be the Merge Join Operator in the MapOperator. Can you remove hints and enable auto conversion to smb join so that I can verify the plans. Take a look at tez_smb_1.q for e.g.
        Hide
        mmccline Matt McCline added a comment -

        Vikram Dixit K thank you for noticing SMB not showing up in the vector_left_outer_join3.q output.

        I fixed that Q file so that MR now shows "Sorted Merge Bucket Map Join Operator" in the explain output.
        However, I notice the Tez output does not. But also vectorized_bucketmapjoin1.q uses a Merge Join.

        And, vector_left_outer_join3.q doesn't show the same problem as non-bucketed... So, I am wondering if there is a problem on the SMB side. I thought there was...

        Show
        mmccline Matt McCline added a comment - Vikram Dixit K thank you for noticing SMB not showing up in the vector_left_outer_join3.q output. I fixed that Q file so that MR now shows "Sorted Merge Bucket Map Join Operator" in the explain output. However, I notice the Tez output does not. But also vectorized_bucketmapjoin1.q uses a Merge Join. And, vector_left_outer_join3.q doesn't show the same problem as non-bucketed... So, I am wondering if there is a problem on the SMB side. I thought there was...
        Hide
        vikram.dixit Vikram Dixit K added a comment -

        The patch looks good to me. Although, we could do with at least one SMB join test. From looking at the tests, I could not find any that triggers the Vector SMB code path.

        Show
        vikram.dixit Vikram Dixit K added a comment - The patch looks good to me. Although, we could do with at least one SMB join test. From looking at the tests, I could not find any that triggers the Vector SMB code path.
        Hide
        hiveqa Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12730138/HIVE-9743.06.patch

        ERROR: -1 due to 5 failed/errored test(s), 8889 tests executed
        Failed tests:

        org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.TestSQLStdHiveAccessControllerHS2.testConfigProcessing
        org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.TestSQLStdHiveAccessControllerHS2.testConfigProcessingCustomSetWhitelistAppend
        org.apache.hive.jdbc.TestJdbcWithLocalClusterSpark.testTempTable
        org.apache.hive.jdbc.TestSSL.testSSLConnectionWithProperty
        org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3724/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3724/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3724/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 5 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12730138 - PreCommit-HIVE-TRUNK-Build

        Show
        hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12730138/HIVE-9743.06.patch ERROR: -1 due to 5 failed/errored test(s), 8889 tests executed Failed tests: org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.TestSQLStdHiveAccessControllerHS2.testConfigProcessing org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.TestSQLStdHiveAccessControllerHS2.testConfigProcessingCustomSetWhitelistAppend org.apache.hive.jdbc.TestJdbcWithLocalClusterSpark.testTempTable org.apache.hive.jdbc.TestSSL.testSSLConnectionWithProperty org.apache.hive.jdbc.TestSSL.testSSLFetchHttp Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3724/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3724/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3724/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed This message is automatically generated. ATTACHMENT ID: 12730138 - PreCommit-HIVE-TRUNK-Build
        Hide
        sushanth Sushanth Sowmyan added a comment -

        Thanks for the update, Matt! Will await Vikram & tests, then.

        Show
        sushanth Sushanth Sowmyan added a comment - Thanks for the update, Matt! Will await Vikram & tests, then.
        Hide
        mmccline Matt McCline added a comment -

        (I got it back up to speed and ready this weekend)

        Show
        mmccline Matt McCline added a comment - (I got it back up to speed and ready this weekend)
        Hide
        mmccline Matt McCline added a comment -

        This patch was intended to go in February and fell between the cracks because it went into another branch.

        Vikram Dixit K it is relevant because although the new native vector map join is on by default, it can be turned off and there are cases like N-way where we haven't gone native yet.

        Since Hybrid Grace was added, the change got a little more complicated (I added a common base class).

        Sushanth Sowmyan Yes, it needs to go into 1.2.

        Show
        mmccline Matt McCline added a comment - This patch was intended to go in February and fell between the cracks because it went into another branch. Vikram Dixit K it is relevant because although the new native vector map join is on by default, it can be turned off and there are cases like N-way where we haven't gone native yet. Since Hybrid Grace was added, the change got a little more complicated (I added a common base class). Sushanth Sowmyan Yes, it needs to go into 1.2.
        Hide
        vikram.dixit Vikram Dixit K added a comment -

        Matt McCline Is the patch uploaded today an updated version of the earlier +1'd patch? I think this one slipped through the cracks. Also, is it relevant with all the new vectorized map join code that went in recently?

        Thanks
        Vikram.

        Show
        vikram.dixit Vikram Dixit K added a comment - Matt McCline Is the patch uploaded today an updated version of the earlier +1'd patch? I think this one slipped through the cracks. Also, is it relevant with all the new vectorized map join code that went in recently? Thanks Vikram.
        Hide
        sushanth Sushanth Sowmyan added a comment -

        Hi, This JIRA has been requested for inclusion in branch-1.2 and has not seen activity for some time after marking as being ready-ish to commit. Vikram Dixit K/Matt McCline, what is the state of this jira?

        Show
        sushanth Sushanth Sowmyan added a comment - Hi, This JIRA has been requested for inclusion in branch-1.2 and has not seen activity for some time after marking as being ready-ish to commit. Vikram Dixit K / Matt McCline , what is the state of this jira?
        Hide
        hagleitn Gunther Hagleitner added a comment -

        Vikram Dixit K ready to commit?

        Show
        hagleitn Gunther Hagleitner added a comment - Vikram Dixit K ready to commit?
        Hide
        hiveqa Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12701442/HIVE-9743.05.patch

        ERROR: -1 due to 1 failed/errored test(s), 7580 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2905/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2905/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2905/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 1 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12701442 - PreCommit-HIVE-TRUNK-Build

        Show
        hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12701442/HIVE-9743.05.patch ERROR: -1 due to 1 failed/errored test(s), 7580 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2905/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2905/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2905/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12701442 - PreCommit-HIVE-TRUNK-Build
        Hide
        mmccline Matt McCline added a comment -

        Use different class for Outer Join and Filtered.

        Show
        mmccline Matt McCline added a comment - Use different class for Outer Join and Filtered.
        Hide
        vikram.dixit Vikram Dixit K added a comment -

        +1 LGTM will commit it shortly.

        Show
        vikram.dixit Vikram Dixit K added a comment - +1 LGTM will commit it shortly.
        Hide
        hiveqa Hive QA added a comment -

        Overall: +1 all checks pass

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12701202/HIVE-9743.04.patch

        SUCCESS: +1 7572 tests passed

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2892/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2892/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2892/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        

        This message is automatically generated.

        ATTACHMENT ID: 12701202 - PreCommit-HIVE-TRUNK-Build

        Show
        hiveqa Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12701202/HIVE-9743.04.patch SUCCESS: +1 7572 tests passed Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2892/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2892/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2892/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12701202 - PreCommit-HIVE-TRUNK-Build
        Hide
        mmccline Matt McCline added a comment -

        Fix test issues; add SMB test.

        Show
        mmccline Matt McCline added a comment - Fix test issues; add SMB test.
        Hide
        hiveqa Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12701098/HIVE-9743.03.patch

        ERROR: -1 due to 5 failed/errored test(s), 7569 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_mapjoin
        org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_mapjoin
        org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_left_outer_join2
        org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_decimal_mapjoin
        org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorized_mapjoin
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2883/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2883/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2883/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 5 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12701098 - PreCommit-HIVE-TRUNK-Build

        Show
        hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12701098/HIVE-9743.03.patch ERROR: -1 due to 5 failed/errored test(s), 7569 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_mapjoin org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_mapjoin org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_left_outer_join2 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_decimal_mapjoin org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorized_mapjoin Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2883/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2883/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2883/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed This message is automatically generated. ATTACHMENT ID: 12701098 - PreCommit-HIVE-TRUNK-Build
        Hide
        mmccline Matt McCline added a comment -

        This patch fixes the problem and continues to vectorize LEFT OUTER JOIN.

        Show
        mmccline Matt McCline added a comment - This patch fixes the problem and continues to vectorize LEFT OUTER JOIN.
        Hide
        mmccline Matt McCline added a comment -

        Postgres

        mmccline=# create table if not exists TJOIN1 (RNUM int , C1 int, C2 int);
        CREATE TABLE
        mmccline=# create table if not exists TJOIN2 (RNUM int , C1 int, C2 char(2));
        CREATE TABLE
        mmccline=# insert into TJOIN1 values (0,10,15),(1,20,25),(2,null,50);
        INSERT 0 3
        mmccline=# insert into TJOIN2 values (0,10,'BB'),(1,15,'DD'),(2,null,'EE'),(3,10,'FF');
        INSERT 0 4
        mmccline=# select tjoin1.rnum, tjoin1.c1, tjoin1.c2, tjoin2.c2 as c2j2 from tjoin1 left outer join tjoin2 on ( tjoin1.c1 = tjoin2.c1 and tjoin1.c2 > 15 );
         rnum | c1 | c2 | c2j2 
        ------+----+----+------
            0 | 10 | 15 | 
            1 | 20 | 25 | 
            2 |    | 50 | 
        (3 rows)
        
        mmccline=# 
        
        Show
        mmccline Matt McCline added a comment - Postgres mmccline=# create table if not exists TJOIN1 (RNUM int , C1 int, C2 int); CREATE TABLE mmccline=# create table if not exists TJOIN2 (RNUM int , C1 int, C2 char(2)); CREATE TABLE mmccline=# insert into TJOIN1 values (0,10,15),(1,20,25),(2,null,50); INSERT 0 3 mmccline=# insert into TJOIN2 values (0,10,'BB'),(1,15,'DD'),(2,null,'EE'),(3,10,'FF'); INSERT 0 4 mmccline=# select tjoin1.rnum, tjoin1.c1, tjoin1.c2, tjoin2.c2 as c2j2 from tjoin1 left outer join tjoin2 on ( tjoin1.c1 = tjoin2.c1 and tjoin1.c2 > 15 ); rnum | c1 | c2 | c2j2 ------+----+----+------ 0 | 10 | 15 | 1 | 20 | 25 | 2 | | 50 | (3 rows) mmccline=#
        Hide
        hiveqa Hive QA added a comment -

        Overall: +1 all checks pass

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12700696/HIVE-9743.02.patch

        SUCCESS: +1 7569 tests passed

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2868/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2868/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2868/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        

        This message is automatically generated.

        ATTACHMENT ID: 12700696 - PreCommit-HIVE-TRUNK-Build

        Show
        hiveqa Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12700696/HIVE-9743.02.patch SUCCESS: +1 7569 tests passed Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2868/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2868/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2868/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12700696 - PreCommit-HIVE-TRUNK-Build
        Hide
        mmccline Matt McCline added a comment -

        Fix q file results and test issues.

        Show
        mmccline Matt McCline added a comment - Fix q file results and test issues.
        Hide
        hiveqa Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12700639/HIVE-9743.01.patch

        ERROR: -1 due to 6 failed/errored test(s), 7569 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_left_outer_join
        org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_left_outer_join
        org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
        org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_left_outer_join
        org.apache.hadoop.hive.ql.optimizer.physical.TestVectorizer.testValidateMapJoinOperator
        org.apache.hadoop.hive.ql.optimizer.physical.TestVectorizer.testValidateSMBJoinOperator
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2865/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2865/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2865/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 6 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12700639 - PreCommit-HIVE-TRUNK-Build

        Show
        hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12700639/HIVE-9743.01.patch ERROR: -1 due to 6 failed/errored test(s), 7569 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_left_outer_join org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_left_outer_join org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_dynamic_partition_pruning org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_left_outer_join org.apache.hadoop.hive.ql.optimizer.physical.TestVectorizer.testValidateMapJoinOperator org.apache.hadoop.hive.ql.optimizer.physical.TestVectorizer.testValidateSMBJoinOperator Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2865/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2865/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2865/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 6 tests failed This message is automatically generated. ATTACHMENT ID: 12700639 - PreCommit-HIVE-TRUNK-Build
        Hide
        mmccline Matt McCline added a comment -

        The patch simply turns off vectorization for OUTER JOIN. Looking into an actual fix.

        Show
        mmccline Matt McCline added a comment - The patch simply turns off vectorization for OUTER JOIN. Looking into an actual fix.
        Hide
        mmccline Matt McCline added a comment -

        There are 3 fields in MapJoinDesc I need to understand: filters, valueTableDescs, valueFilteredTableDesc and their usage in (regular) inner join and outer join.

        Show
        mmccline Matt McCline added a comment - There are 3 fields in MapJoinDesc I need to understand: filters, valueTableDescs, valueFilteredTableDesc and their usage in (regular) inner join and outer join.
        Hide
        prasanth_j Prasanth Jayachandran added a comment -

        This issue happens only when vectorization is enabled. Without vectorization (but with ORC) the query runs correct. Matt McCline VectorMapJoin operator always applies the filter on big table side without checking for type of join. Can you check if thats the issue?

        Show
        prasanth_j Prasanth Jayachandran added a comment - This issue happens only when vectorization is enabled. Without vectorization (but with ORC) the query runs correct. Matt McCline VectorMapJoin operator always applies the filter on big table side without checking for type of join. Can you check if thats the issue?
        Hide
        cartershanklin Carter Shanklin added a comment -

        Prasanth Jayachandran ^^ Right syntax this time

        Show
        cartershanklin Carter Shanklin added a comment - Prasanth Jayachandran ^^ Right syntax this time
        Hide
        cartershanklin Carter Shanklin added a comment -

        Confirmed. Though the problem only happens when tjoin1 is ORC and not text. @prasanth_j any ideas?

        Show
        cartershanklin Carter Shanklin added a comment - Confirmed. Though the problem only happens when tjoin1 is ORC and not text. @prasanth_j any ideas?

          People

          • Assignee:
            mmccline Matt McCline
            Reporter:
            the6campbells N Campbell
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development