Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-16784

Missing lineage information when hive.blobstore.optimizations.enabled is true

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 3.0.0
    • None
    • None

    Description

      Running the commands of the add_part_multiple.q test on S3 with hive.blobstore.optimizations.enabled=true fails because of missing lineage information.
      Running the command on HDFS

      from src TABLESAMPLE (1 ROWS)
      insert into table add_part_test PARTITION (ds='2010-01-01') select 100,100
      insert into table add_part_test PARTITION (ds='2010-02-01') select 200,200
      insert into table add_part_test PARTITION (ds='2010-03-01') select 400,300
      insert into table add_part_test PARTITION (ds='2010-04-01') select 500,400;
      

      results the following posthook outputs

      POSTHOOK: Lineage: add_part_test2 PARTITION(ds=2010-01-01).key EXPRESSION []
      POSTHOOK: Lineage: add_part_test2 PARTITION(ds=2010-01-01).value EXPRESSION []
      POSTHOOK: Lineage: add_part_test2 PARTITION(ds=2010-02-01).key EXPRESSION []
      POSTHOOK: Lineage: add_part_test2 PARTITION(ds=2010-02-01).value EXPRESSION []
      POSTHOOK: Lineage: add_part_test2 PARTITION(ds=2010-03-01).key EXPRESSION []
      POSTHOOK: Lineage: add_part_test2 PARTITION(ds=2010-03-01).value EXPRESSION []
      POSTHOOK: Lineage: add_part_test2 PARTITION(ds=2010-04-01).key EXPRESSION []
      POSTHOOK: Lineage: add_part_test2 PARTITION(ds=2010-04-01).value EXPRESSION []
      

      These lines are not printed when running the command on the table located in S3.
      If hive.blobstore.optimizations.enabled=false, the lineage information is printed.

      Attachments

        1. HIVE-16784.02.patch
          33 kB
          Barna Zsombor Klara
        2. HIVE-16784.02.patch
          33 kB
          Barna Zsombor Klara
        3. HIVE-16784.01.patch
          2 kB
          Barna Zsombor Klara

        Issue Links

          Activity

            People

              zsombor.klara Barna Zsombor Klara
              kuczoram Marta Kuczora
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: