[HAWQ-991] "HAWQ register" could register tables according to .yml configuration file - ASF JIRA

Add vote

Watch issue

XML

Word

Printable

JSON

Details

Type: New Feature
Status: Open
Priority: Major
Resolution: Unresolved
Affects Version/s: 2.1.0.0-incubating
Fix Version/s: backlog
Component/s: Command Line Tools
Labels:
None

Description

Scenario:
1. For cluster Disaster Recovery. Two clusters co-exist, periodically import data from Cluster A to Cluster B. Need Register data to Cluster B.
2. For the rollback of table. Do checkpoints somewhere, and need to rollback to previous checkpoint.

Description:
Register according to .yml configuration file.
hawq register [-h hostname] [-p port] [-U username] [-d databasename] [-c config] [--force][--repair] <tablename>

Behaviors:
1. If table doesn't exist, will automatically create the table and register the files in .yml configuration file. Will use the filesize specified in .yml to update the catalog table.

2. If table already exist, and neither --force nor --repair configured. Do not create any table, and directly register the files specified in .yml file to the table. Note that if the file is under table directory in HDFS, will throw error, say, to-be-registered files should not under the table path.

3. If table already exist, and --force is specified. Will clear all the catalog contents in pg_aoseg.pg_paqseg_$relid while keep the files on HDFS, and then re-register all the files to the table. This is for scenario 2.

4. If table already exist, and --repair is specified. Will change both file folder and catalog table pg_aoseg.pg_paqseg_$relid to the state which .yml file configures. Note may some new generated files since the checkpoint may be deleted here. Also note the all the files in .yml file should all under the table folder on HDFS. Limitation: Do not support cases for hash table redistribution, table truncate and table drop. This is for scenario 3.

Requirements:
1. To be registered file path has to colocate with HAWQ in the same HDFS cluster.
2. If to be registered is a hash table, the registered file number should be one or multiple times or hash table bucket number.

Attachments

Issue Links

Is contained by

HAWQ-760 Hawq register

Reopened

is related to

HAWQ-1112 Error message is not accurate when hawq register with single file and the size is larger than real size

Closed

HAWQ-1113 In force mode, hawq register error when files in yaml is disordered

Closed

HAWQ-1127 HAWQ should print error message instead of python function stack when yaml file is invalid.

Closed

HAWQ-1144 Register into a 2-level partition table, hawq register didn't throw error, and indicates that hawq register succeed, but no data can be selected out.