Hive
  1. Hive
  2. HIVE-3136

QTestUtil diff masking masks out most of 'DESCRIBE EXTENDED' output

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Testing Infrastructure
    • Labels:
      None

      Issue Links

        Activity

        Navis made changes -
        Link This issue relates to HIVE-3209 [ HIVE-3209 ]
        Hide
        Shane Huang added a comment -

        @Carl @JQ HIVE-3148 test case needs to use DESCRIBE FORMATTED to show the partition Locations. But the original QTestUtil masks out the Location related lines. I changed QTestUtil in HIVE-3148's patch to allow transforming output before result is checked. This could be generalized to support other test cases.

        Show
        Shane Huang added a comment - @Carl @JQ HIVE-3148 test case needs to use DESCRIBE FORMATTED to show the partition Locations. But the original QTestUtil masks out the Location related lines. I changed QTestUtil in HIVE-3148 's patch to allow transforming output before result is checked. This could be generalized to support other test cases.
        Shane Huang made changes -
        Field Original Value New Value
        Link This issue is related to HIVE-3148 [ HIVE-3148 ]
        Hide
        Carl Steinbach added a comment -

        Here's what the output of DESCRIBE EXTENDED looks like after it gets processed with QTestUtil.checkCliDriverResults:

        PREHOOK: query: desc extended tbl2
        PREHOOK: type: DESCTABLE
        POSTHOOK: query: desc extended tbl2
        POSTHOOK: type: DESCTABLE
        col	string	
        p	string	
        	 	 
        #### A masked pattern was here ####
        

        And here's what it looked like before diff masking:

        col	string	
        p	string	
        	 	 
        Detailed Table Information	Table(tableName:tbl2, dbName:default, owner:carl, createTime:1339652050, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:col, type:string, comment:null), FieldSchema(name:p, type:string, comment:null)], location:pfile:/Users/carl/Work/repos/hive-test/build/ql/test/data/warehouse/tbl2, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}), partitionKeys:[FieldSchema(name:p, type:string, comment:null)], parameters:{last_modified_by=carl, last_modified_time=1339652051, PROTECT_MODE=NO_DROP_CASCADE, transient_lastDdlTime=1339652051}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE)
        

        The problem with this is that many tests depend on DESC EXTENDED to verify the contents of the StorageDescriptor, view properties, tableType, etc, etc, and all of that information is currently getting masked out.

        Show
        Carl Steinbach added a comment - Here's what the output of DESCRIBE EXTENDED looks like after it gets processed with QTestUtil.checkCliDriverResults: PREHOOK: query: desc extended tbl2 PREHOOK: type: DESCTABLE POSTHOOK: query: desc extended tbl2 POSTHOOK: type: DESCTABLE col string p string #### A masked pattern was here #### And here's what it looked like before diff masking: col string p string Detailed Table Information Table(tableName:tbl2, dbName:default, owner:carl, createTime:1339652050, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:col, type:string, comment:null), FieldSchema(name:p, type:string, comment:null)], location:pfile:/Users/carl/Work/repos/hive-test/build/ql/test/data/warehouse/tbl2, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}), partitionKeys:[FieldSchema(name:p, type:string, comment:null)], parameters:{last_modified_by=carl, last_modified_time=1339652051, PROTECT_MODE=NO_DROP_CASCADE, transient_lastDdlTime=1339652051}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE) The problem with this is that many tests depend on DESC EXTENDED to verify the contents of the StorageDescriptor, view properties, tableType, etc, etc, and all of that information is currently getting masked out.
        Carl Steinbach created issue -

          People

          • Assignee:
            Unassigned
            Reporter:
            Carl Steinbach
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:

              Development