[HBASE-11409] Add more flexibility for input directory structure to LoadIncrementalHFiles

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha-1
• Fix Version/s: 1.4.1, 2.0.0-beta-1, 2.0.0

Release Note

Allows for users to bulk load entire tables from hdfs by specifying the parameter -loadTable. This allows you to pass in a table level directory and have all regions column families bulk loaded, if you do not specify the -loadTable parameter LoadIncrementalHFiles will work as before. Note: you must have a pre-created table to run with -loadTable it will not create one for you.

• Assignee: churro morales
• Reporter: churro morales
• Created: 25/Jun/14 00:15
• Updated: 17/Jun/22 06:00
• Resolved: 12/Jan/18 21:36

Description

Use case:

We were trying to combine two very large tables into a single table. Thus we ran jobs in one datacenter that populated certain column families and another datacenter which populated other column families. Took a snapshot and exported them to their respective datacenters. Wanted to simply take the hdfs restored snapshot and use LoadIncremental to merge the data.

It would be nice to add support where we could run LoadIncremental on a directory where the depth of store files is something other than two (current behavior).

With snapshots it would be nice if you could pass a restored hdfs snapshot's directory and have the tool run.

I am attaching a patch where I parameterize the bulkLoad timeout as well as the default store file depth.

Issue Links

• relates to: HBASE-19754 Backport HBASE-11409 to branch-1 and branch-1.4