Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-13861

Fix up nullability issue that might be created by pull up constants rules

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.1.0
    • 2.1.0
    • CBO
    • None

    Description

      When we pull up constants through Union or Sort operators, we might end up rewriting the original expression into an expression whose schema has different nullability properties for some of its columns.

      This results in AssertionError of the following kind:

      ...
      org.apache.hive.service.cli.HiveSQLException: Error running query: java.lang.AssertionError: Internal error: Cannot add expression of different type to set:
      ...
      

      Attachments

        1. HIVE-13861.01.patch
          2 kB
          jcamachorodriguez

        Issue Links

          Activity

            +1 pending tests

            ashutoshc Ashutosh Chauhan added a comment - +1 pending tests
            jcamacho Jesús Camacho Rodríguez added a comment - - edited
            Test Name	Duration	Age
             org.apache.hive.hcatalog.listener.TestDbNotificationListener.cleanupNotifs	1 min 0 sec	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dynpart_hashjoin_1	79 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dynpart_hashjoin_2	65 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_3	1 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_4	1 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_5	1 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_constprog_dpp	0 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_mrr	1 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_smb_main	1 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_smb_1	1 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union	1 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_join_part_col_char	0 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_schema_evolution	1 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_bucket_map_join_tez1	2 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_bucket_map_join_tez2	1 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_mapjoin_decimal	1 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_tests	1 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_lvj_mapjoin	3 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_vector_dynpart_hashjoin_1	1 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_vector_dynpart_hashjoin_2	1 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_1	0 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_2	1 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_3	1 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_4	1 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_5	0 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_hash	1 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_insert_overwrite_local_directory_1	2 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_ppd_basic	1 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_bmj_schema_evolution	1 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_llapdecider	0 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_llap_nullscan	0 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_result_complex	0 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_hybridgrace_hashjoin_2	1 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_fsstat	0 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union_multiinsert	0 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dml	1 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join	0 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union_dynamic_partition	0 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_llap_udf	0 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_multi_union	0 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union2	1 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union_decimal	0 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union_group_by	1 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_joins_explain	0 ms	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver	1 ms	1
             org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority	5 sec	1
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_1	5 ms	2
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_2	2 ms	2
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_self_join	9 ms	2
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_hybridgrace_hashjoin_1	0 ms	2
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning	0 ms	2
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning_2	0 ms	2
             org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vectorized_dynamic_partition_pruning	1 ms	2
             org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testPreemptionQueueComparator	5 sec	2
             org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner	2.7 sec	31
             org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static	1 min 40 sec	47
             org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic	1 min 20 sec	47
             org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForMemoryTokenStore	1.1 sec	47
             org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForDBTokenStore	0.35 sec	47
             org.apache.hive.minikdc.TestMiniHiveKdc.testLogin	1 min 29 sec	47
             org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec	71 ms	67
             org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_selectindate	12 sec	99
             org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avrocountemptytbl	10 sec	99
             org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order_null	44 sec	99
             org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys	1 min 36 sec	99
             org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3	8.3 sec	99
             org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver	1 min 21 sec	99
            
            jcamacho Jesús Camacho Rodríguez added a comment - - edited Test Name Duration Age org.apache.hive.hcatalog.listener.TestDbNotificationListener.cleanupNotifs 1 min 0 sec 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dynpart_hashjoin_1 79 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dynpart_hashjoin_2 65 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_3 1 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_4 1 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_5 1 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_constprog_dpp 0 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_mrr 1 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_smb_main 1 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_smb_1 1 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union 1 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_join_part_col_char 0 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_schema_evolution 1 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_bucket_map_join_tez1 2 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_bucket_map_join_tez2 1 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_mapjoin_decimal 1 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_tests 1 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_lvj_mapjoin 3 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_vector_dynpart_hashjoin_1 1 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_vector_dynpart_hashjoin_2 1 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_1 0 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_2 1 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_3 1 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_4 1 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_5 0 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_hash 1 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_insert_overwrite_local_directory_1 2 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_ppd_basic 1 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_bmj_schema_evolution 1 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_llapdecider 0 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_llap_nullscan 0 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_result_complex 0 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_hybridgrace_hashjoin_2 1 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_fsstat 0 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union_multiinsert 0 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dml 1 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join 0 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union_dynamic_partition 0 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_llap_udf 0 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_multi_union 0 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union2 1 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union_decimal 0 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union_group_by 1 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_joins_explain 0 ms 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver 1 ms 1 org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority 5 sec 1 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_1 5 ms 2 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_2 2 ms 2 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_self_join 9 ms 2 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_hybridgrace_hashjoin_1 0 ms 2 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning 0 ms 2 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning_2 0 ms 2 org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vectorized_dynamic_partition_pruning 1 ms 2 org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testPreemptionQueueComparator 5 sec 2 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner 2.7 sec 31 org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static 1 min 40 sec 47 org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic 1 min 20 sec 47 org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForMemoryTokenStore 1.1 sec 47 org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForDBTokenStore 0.35 sec 47 org.apache.hive.minikdc.TestMiniHiveKdc.testLogin 1 min 29 sec 47 org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec 71 ms 67 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_selectindate 12 sec 99 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avrocountemptytbl 10 sec 99 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order_null 44 sec 99 org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys 1 min 36 sec 99 org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 8.3 sec 99 org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver 1 min 21 sec 99
            jcamacho Jesús Camacho Rodríguez added a comment - - edited

            Fails are not related. Pushed to master, branch-2.1. Thanks for reviewing ashutoshc!

            jcamacho Jesús Camacho Rodríguez added a comment - - edited Fails are not related. Pushed to master, branch-2.1. Thanks for reviewing ashutoshc !

            People

              jcamacho Jesús Camacho Rodríguez
              jcamacho Jesús Camacho Rodríguez
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: