Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-24128

transactions cannot recognize bucket file

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.1.1
    • None
    • None

    Description

       

      • Error while compiling statement: FAILED: SemanticException [Error 10141]: Bucketed table metadata is not correct. Fix the metadata or don't use bucketed mapjoin, by setting hive.enforce.bucketmapjoin to false. The number of buckets for table dcp partition ods_load_date=20200701 is 2, whereas the number of files is 1
      • the trnasaaction table  manage file like below 
        -rw-r--r--   3 hadoop supergroup          1 2020-09-08 10:20 /usr/hive/warehouse/test_etl_dwd.db/dcp/ods_load_date=20200701/_orc_acid_version
        drwxr-xr-x   - hadoop supergroup          0 2020-09-07 19:28 /usr/hive/warehouse/test_etl_dwd.db/dcp/ods_load_date=20200701/base_0000021
        -rw-r--r--   3 hadoop supergroup   15401449 2020-09-08 10:20 /usr/hive/warehouse/test_etl_dwd.db/dcp/ods_load_date=20200701/bucket_00000
        -rw-r--r--   3 hadoop supergroup   15408471 2020-09-08 10:20 /usr/hive/warehouse/test_etl_dwd.db/dcp/ods_load_date=20200701/bucket_00001
        
      • it put the bucket file into.      dir  base**, 
      • but when I use merge  the table , hive will check the bucket file number , 
      • public static List<String> getBucketFilePathsOfPartition(
              Path location, ParseContext pGraphContext) throws SemanticException {
            List<String> fileNames = new ArrayList<String>();
            try {
              FileSystem fs = location.getFileSystem(pGraphContext.getConf());
              FileStatus[] files = fs.listStatus(new Path(location.toString()), FileUtils.HIDDEN_FILES_PATH_FILTER);
              if (files != null) {
                for (FileStatus file : files) {
                  fileNames.add(file.getPath().toString());
                }
              }
            } catch (IOException e) {
              throw new SemanticException(e);
            }
            return fileNames;
          }
        

        it only. check the file under the partition , not check the base** directory 

      Attachments

        Activity

          People

            Unassigned Unassigned
            yantianyu richt richt
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: