[ACCUMULO-31] pattern match for bulk-import in-progress markers is not scalable

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.3.5-incubating
• Component/s: gc
• Assignee: Eric C. Newton
• Reporter: Eric C. Newton
• Created: 13/Oct/11 18:42
• Updated: 18/Nov/11 15:16
• Resolved: 13/Oct/11 19:49

Description

When you ask HDFS for tables//bulk_/processing_proc_*, the NameNode seems to be shipping back all the file names, and the client is performing the pattern match. This results in very bad NameNode performance during accumulo garbage collection.

This is fixed in the trunk: markers are put in the METADATA table.

However, the fix is needed in very large accumulo installations presently using 1.3.x.