Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-16784

Missing lineage information when hive.blobstore.optimizations.enabled is true

Log workAgile BoardRank to TopRank to BottomVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 3.0.0
    • None
    • None

    Description

      Running the commands of the add_part_multiple.q test on S3 with hive.blobstore.optimizations.enabled=true fails because of missing lineage information.
      Running the command on HDFS

      from src TABLESAMPLE (1 ROWS)
      insert into table add_part_test PARTITION (ds='2010-01-01') select 100,100
      insert into table add_part_test PARTITION (ds='2010-02-01') select 200,200
      insert into table add_part_test PARTITION (ds='2010-03-01') select 400,300
      insert into table add_part_test PARTITION (ds='2010-04-01') select 500,400;
      

      results the following posthook outputs

      POSTHOOK: Lineage: add_part_test2 PARTITION(ds=2010-01-01).key EXPRESSION []
      POSTHOOK: Lineage: add_part_test2 PARTITION(ds=2010-01-01).value EXPRESSION []
      POSTHOOK: Lineage: add_part_test2 PARTITION(ds=2010-02-01).key EXPRESSION []
      POSTHOOK: Lineage: add_part_test2 PARTITION(ds=2010-02-01).value EXPRESSION []
      POSTHOOK: Lineage: add_part_test2 PARTITION(ds=2010-03-01).key EXPRESSION []
      POSTHOOK: Lineage: add_part_test2 PARTITION(ds=2010-03-01).value EXPRESSION []
      POSTHOOK: Lineage: add_part_test2 PARTITION(ds=2010-04-01).key EXPRESSION []
      POSTHOOK: Lineage: add_part_test2 PARTITION(ds=2010-04-01).value EXPRESSION []
      

      These lines are not printed when running the command on the table located in S3.
      If hive.blobstore.optimizations.enabled=false, the lineage information is printed.

      Attachments

        1. HIVE-16784.02.patch
          33 kB
          Barna Zsombor Klara
        2. HIVE-16784.02.patch
          33 kB
          Barna Zsombor Klara
        3. HIVE-16784.01.patch
          2 kB
          Barna Zsombor Klara

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            zsombor.klara Barna Zsombor Klara Assign to me
            kuczoram Marta Kuczora
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment