Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-28120

When insert overwrite the iceberg table, data will loss if the sql contains union all

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Cannot Reproduce
    • 4.0.0-beta-1
    • Not Applicable
    • Iceberg integration
    • None
    • hadoop version: 3.3.1

      hive version: 4.0.0-beta-1

      iceberg version: 1.3.0

    Description

      (1)

      create table tmp.test_iceberg_overwrite_union_all(
      a string
      )
      stored by iceberg
      ;

      (2)

      insert overwrite table tmp.test_iceberg_overwrite_union_all
      select distinct 'a' union all select distinct 'b';

      (3)

      select * from tmp.test_iceberg_overwrite_union_all;

       

      the result only has one record:

      -------------------------------------

      test_iceberg_overwrite_union_all.a  

      -------------------------------------

      a                                  

      -------------------------------------

      According to the hiveserver log, this query will start two jobs, and each job will be committed. The problem is that the job that is committed later is also an overwrite, causing the result of the first commit to be overwritten. like this:

      2024-03-05T22:10:12,995 INFO  [iceberg-commit-table-pool-0]: hive.HiveIcebergOutputCommitter () - Committing job has started for table: default_iceberg.tmp.test_iceberg_overwrite_union_all
      2024-03-05T22:10:13,081 INFO  [iceberg-commit-table-pool-1]: hive.HiveIcebergOutputCommitter () - Committing job has started for table: default_iceberg.tmp.test_iceberg_overwrite_union_all
      2024-03-05T22:10:15,152 INFO  [iceberg-commit-table-pool-0]: hive.HiveIcebergOutputCommitter () - Overwrite commit took 2157 ms for table: default_iceberg.tmp.test_iceberg_overwrite_union_all with 1 file(s)
      2024-03-05T22:10:16,980 INFO  [iceberg-commit-table-pool-1]: hive.HiveIcebergOutputCommitter () - Overwrite commit took 3899 ms for table: default_iceberg.tmp.test_iceberg_overwrite_union_all with 1 file(s)

      Attachments

        Activity

          People

            Unassigned Unassigned
            xinmingchang xinmingchang
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: