Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-16784

Missing lineage information when hive.blobstore.optimizations.enabled is true

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0
    • Component/s: None
    • Labels:
      None

      Description

      Running the commands of the add_part_multiple.q test on S3 with hive.blobstore.optimizations.enabled=true fails because of missing lineage information.
      Running the command on HDFS

      from src TABLESAMPLE (1 ROWS)
      insert into table add_part_test PARTITION (ds='2010-01-01') select 100,100
      insert into table add_part_test PARTITION (ds='2010-02-01') select 200,200
      insert into table add_part_test PARTITION (ds='2010-03-01') select 400,300
      insert into table add_part_test PARTITION (ds='2010-04-01') select 500,400;
      

      results the following posthook outputs

      POSTHOOK: Lineage: add_part_test2 PARTITION(ds=2010-01-01).key EXPRESSION []
      POSTHOOK: Lineage: add_part_test2 PARTITION(ds=2010-01-01).value EXPRESSION []
      POSTHOOK: Lineage: add_part_test2 PARTITION(ds=2010-02-01).key EXPRESSION []
      POSTHOOK: Lineage: add_part_test2 PARTITION(ds=2010-02-01).value EXPRESSION []
      POSTHOOK: Lineage: add_part_test2 PARTITION(ds=2010-03-01).key EXPRESSION []
      POSTHOOK: Lineage: add_part_test2 PARTITION(ds=2010-03-01).value EXPRESSION []
      POSTHOOK: Lineage: add_part_test2 PARTITION(ds=2010-04-01).key EXPRESSION []
      POSTHOOK: Lineage: add_part_test2 PARTITION(ds=2010-04-01).value EXPRESSION []
      

      These lines are not printed when running the command on the table located in S3.
      If hive.blobstore.optimizations.enabled=false, the lineage information is printed.

        Attachments

        1. HIVE-16784.02.patch
          33 kB
          Barna Zsombor Klara
        2. HIVE-16784.02.patch
          33 kB
          Barna Zsombor Klara
        3. HIVE-16784.01.patch
          2 kB
          Barna Zsombor Klara

          Issue Links

            Activity

              People

              • Assignee:
                zsombor.klara Barna Zsombor Klara
                Reporter:
                kuczoram Marta Kuczora
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: