Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.1.1
-
None
-
None
Description
- Error while compiling statement: FAILED: SemanticException [Error 10141]: Bucketed table metadata is not correct. Fix the metadata or don't use bucketed mapjoin, by setting hive.enforce.bucketmapjoin to false. The number of buckets for table dcp partition ods_load_date=20200701 is 2, whereas the number of files is 1
- the trnasaaction table manage file like below
-rw-r--r-- 3 hadoop supergroup 1 2020-09-08 10:20 /usr/hive/warehouse/test_etl_dwd.db/dcp/ods_load_date=20200701/_orc_acid_version drwxr-xr-x - hadoop supergroup 0 2020-09-07 19:28 /usr/hive/warehouse/test_etl_dwd.db/dcp/ods_load_date=20200701/base_0000021 -rw-r--r-- 3 hadoop supergroup 15401449 2020-09-08 10:20 /usr/hive/warehouse/test_etl_dwd.db/dcp/ods_load_date=20200701/bucket_00000 -rw-r--r-- 3 hadoop supergroup 15408471 2020-09-08 10:20 /usr/hive/warehouse/test_etl_dwd.db/dcp/ods_load_date=20200701/bucket_00001
- it put the bucket file into. dir base**,
- but when I use merge the table , hive will check the bucket file number ,
public static List<String> getBucketFilePathsOfPartition( Path location, ParseContext pGraphContext) throws SemanticException { List<String> fileNames = new ArrayList<String>(); try { FileSystem fs = location.getFileSystem(pGraphContext.getConf()); FileStatus[] files = fs.listStatus(new Path(location.toString()), FileUtils.HIDDEN_FILES_PATH_FILTER); if (files != null) { for (FileStatus file : files) { fileNames.add(file.getPath().toString()); } } } catch (IOException e) { throw new SemanticException(e); } return fileNames; }
it only. check the file under the partition , not check the base** directory