Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-21564

Load data into a bucketed table is ignoring partitions specs and loads data into default partition.

    XMLWordPrintableJSON

    Details

    • Target Version/s:

      Description

      When running below command to load data into bucketed tables it is not loading into specified partition instead loaded into default partition.
      LOAD DATA INPATH '/tmp/files/000000_0' OVERWRITE INTO TABLE call PARTITION(year_partition=2012, month=12);

      SELECT * FROM call WHERE year_partition=2012 AND month=12; --> returns 0 rows.

      CREATE TABLE call( 
      date_time_date date, 
      ssn string, 
      name string, 
      location string) 
      PARTITIONED BY ( 
      year_partition int, 
      month int) 
      CLUSTERED BY ( 
      date_time_date) 
      SORTED BY ( 
      date_time_date ASC) 
      INTO 1 BUCKETS 
      STORED AS ORC;
      

      If set hive.exec.dynamic.partition to false, it fails with below error.

      Error: Error while compiling statement: FAILED: SemanticException 1:18 Dynamic partition is disabled. Either enable it by setting hive.exec.dynamic.partition=true or specify partition column values. Error encountered near token 'month' (state=42000,code=40000)
      

      When we "set hive.strict.checks.bucketing=false;", the load works fine.
      This is a behaviour imposed by HIVE-15148 to avoid incorrectly named data files being loaded to the bucketed tables. In customer use case, if the files are named properly with bucket_id (00000_0, 00000_1 etc), then it is safe to set this flag to false.
      However, current behaviour of loading into default partitions when hive.strict.checks.bucketing=true and partitions specified, was a bug injected by HIVE-19311 where the given query is re-written into a insert query (to handle incorrect file names and Orc versions) but missed to incorporate the partitions specs to it.

        Attachments

        1. HIVE-21564.01.patch
          33 kB
          Sankar Hariappan
        2. HIVE-21564.02.patch
          42 kB
          Sankar Hariappan

          Issue Links

            Activity

              People

              • Assignee:
                sankarh Sankar Hariappan
                Reporter:
                sankarh Sankar Hariappan
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m