[IMPALA-13340] COPY TESTCASE in LocalCatalog mode doesn't dump the partition and file metadata

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: Impala 4.5.0
• Component/s: Catalog
• Epic Color: ghx-label-1
• Assignee: Quanlong Huang
• Reporter: Quanlong Huang
• Created: 28/Aug/24 08:21
• Updated: 31/Oct/24 00:01
• Resolved: 31/Oct/24 00:01

Description

IMPALA-11901 fixes the failures of using COPY TESTCASE statements in LocalCatalog mode. However, only the table metadata is dumped, e.g. table schema, column stats. The partition and file metadata are missing.

To reproduce the issue locally, start the Impala cluster in LocalCatalog mode.

bin/start-impala-cluster.py --catalogd_args=--catalog_topic_mode=minimal --impalad_args=--use_local_catalog

Dump the metadata of a query on a partitioned table:

copy testcase to '/tmp' select * from functional_parquet.alltypes;
+--------------------------------------------------------------------------------------+
| Test case data output path                                                           |
+--------------------------------------------------------------------------------------+
| hdfs://localhost:20500/tmp/impala-testcase-data-c8316356-6448-4458-acad-c2f72f43c3e1 |
+--------------------------------------------------------------------------------------+ 

Check the metadata from the source cluster

show partitions functional_parquet.alltypes
+-------+-------+-------+--------+----------+--------------+-------------------+---------+-------------------+---------------------------------------------------------------------------+-----------+
| year  | month | #Rows | #Files | Size     | Bytes Cached | Cache Replication | Format  | Incremental stats | Location                                                                  | EC Policy |
+-------+-------+-------+--------+----------+--------------+-------------------+---------+-------------------+---------------------------------------------------------------------------+-----------+
| 2009  | 1     | -1    | 1      | 8.60KB   | NOT CACHED   | NOT CACHED        | PARQUET | false             | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=1  | NONE      |
| 2009  | 2     | -1    | 1      | 8.09KB   | NOT CACHED   | NOT CACHED        | PARQUET | false             | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=2  | NONE      |
| 2009  | 3     | -1    | 1      | 8.60KB   | NOT CACHED   | NOT CACHED        | PARQUET | false             | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=3  | NONE      |
| 2009  | 4     | -1    | 1      | 8.20KB   | NOT CACHED   | NOT CACHED        | PARQUET | false             | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=4  | NONE      |
| 2009  | 5     | -1    | 1      | 8.55KB   | NOT CACHED   | NOT CACHED        | PARQUET | false             | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=5  | NONE      |
| 2009  | 6     | -1    | 1      | 8.23KB   | NOT CACHED   | NOT CACHED        | PARQUET | false             | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=6  | NONE      |
| 2009  | 7     | -1    | 1      | 8.25KB   | NOT CACHED   | NOT CACHED        | PARQUET | false             | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=7  | NONE      |
| 2009  | 8     | -1    | 1      | 8.60KB   | NOT CACHED   | NOT CACHED        | PARQUET | false             | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=8  | NONE      |
| 2009  | 9     | -1    | 1      | 8.41KB   | NOT CACHED   | NOT CACHED        | PARQUET | false             | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=9  | NONE      |
| 2009  | 10    | -1    | 1      | 8.60KB   | NOT CACHED   | NOT CACHED        | PARQUET | false             | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=10 | NONE      |
| 2009  | 11    | -1    | 1      | 8.44KB   | NOT CACHED   | NOT CACHED        | PARQUET | false             | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=11 | NONE      |
| 2009  | 12    | -1    | 1      | 8.59KB   | NOT CACHED   | NOT CACHED        | PARQUET | false             | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=12 | NONE      |
| 2010  | 1     | -1    | 1      | 8.38KB   | NOT CACHED   | NOT CACHED        | PARQUET | false             | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=1  | NONE      |
| 2010  | 2     | -1    | 1      | 7.80KB   | NOT CACHED   | NOT CACHED        | PARQUET | false             | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=2  | NONE      |
| 2010  | 3     | -1    | 1      | 8.29KB   | NOT CACHED   | NOT CACHED        | PARQUET | false             | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=3  | NONE      |
| 2010  | 4     | -1    | 1      | 8.20KB   | NOT CACHED   | NOT CACHED        | PARQUET | false             | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=4  | NONE      |
| 2010  | 5     | -1    | 1      | 8.62KB   | NOT CACHED   | NOT CACHED        | PARQUET | false             | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=5  | NONE      |
| 2010  | 6     | -1    | 1      | 8.26KB   | NOT CACHED   | NOT CACHED        | PARQUET | false             | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=6  | NONE      |
| 2010  | 7     | -1    | 1      | 8.60KB   | NOT CACHED   | NOT CACHED        | PARQUET | false             | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=7  | NONE      |
| 2010  | 8     | -1    | 1      | 8.64KB   | NOT CACHED   | NOT CACHED        | PARQUET | false             | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=8  | NONE      |
| 2010  | 9     | -1    | 1      | 8.20KB   | NOT CACHED   | NOT CACHED        | PARQUET | false             | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=9  | NONE      |
| 2010  | 10    | -1    | 1      | 8.59KB   | NOT CACHED   | NOT CACHED        | PARQUET | false             | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=10 | NONE      |
| 2010  | 11    | -1    | 1      | 8.20KB   | NOT CACHED   | NOT CACHED        | PARQUET | false             | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=11 | NONE      |
| 2010  | 12    | -1    | 1      | 8.60KB   | NOT CACHED   | NOT CACHED        | PARQUET | false             | hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=12 | NONE      |
| Total |       | -1    | 24     | 201.52KB | 0B           |                   |         |                   |                                                                           |           |
+-------+-------+-------+--------+----------+--------------+-------------------+---------+-------------------+---------------------------------------------------------------------------+-----------+

Import the metadata

copy testcase from 'hdfs://localhost:20500/tmp/impala-testcase-data-c8316356-6448-4458-acad-c2f72f43c3e1';
+----------------------------------------------------------------------------------------------------------------+
| summary                                                                                                        |
+----------------------------------------------------------------------------------------------------------------+
| Testcase generated using Impala version 4.5.0-SNAPSHOT. 1 db(s), 1 table(s) and 0 view(s) imported for query:  |
|                                                                                                                |
| SELECT * FROM functional_parquet.alltypes                                                                      |
+----------------------------------------------------------------------------------------------------------------+

Check the partition metadata, they are missing:

set PLANNER_TESTCASE_MODE=true;
show partitions functional_parquet.alltypes;
Query: show partitions functional_parquet.alltypes
+-------+-------+-------+--------+------+--------------+-------------------+--------+-------------------+----------+-----------+
| year  | month | #Rows | #Files | Size | Bytes Cached | Cache Replication | Format | Incremental stats | Location | EC Policy |
+-------+-------+-------+--------+------+--------------+-------------------+--------+-------------------+----------+-----------+
| Total |       | -1    | 0      | 0B   | 0B           |                   |        |                   |          |           |
+-------+-------+-------+--------+------+--------------+-------------------+--------+-------------------+----------+-----------+