Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Duplicate
-
1.6.4, 1.7.0
-
None
-
None
-
300 node test cluster
Description
On a large test cluster, a table was storing very large row keys that were similar for 1 - 10 M per row id (which, is a bad schema... but not the problem, yet).
Large row keys made the tablet large, so it split. And the first 1-10M of the row keys were stored in the metadata table.
The metadata table has a small split size, so it split.
This ended up recording several keys in the root tablet that were very large. For example, a single metadata table file was 700M (compressed) and contained 34 keys.
The problem is that everyone wants to read the root tablet to find the metadata tablets. And that was causing the tablet server hosting the tablet to run out of heap.
Possible solution: bring down the cluster, put it in "safe mode" where only the metadata table is brought online. Raise the split size of the metadata table to something large (1G?). Merge the metadata table which should remove the large records from the root tablet.
There's a utility (SplitLarge) than can be used to remove large keys from the RFiles of the offending table. Once the ridiculous keys are stripped out the table can be brought online and merged, which will remove the large keys from the metadata table.
As long is this is done on a small number of nodes, the servers should have enough memory to satisfy the requests to perform the metadata table queries and updates.
We may want to consider adding key size to the metadata table constraint to prevent these things in the future.
Attachments
Issue Links
- duplicates
-
ACCUMULO-3021 Limit size of split rows
-
- Resolved
-