[ACCUMULO-416] reevaluate limiting the number of open files given HDFS improvements

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Abandoned
• Component/s: tserver
• Assignee: Keith Turner
• Reporter: Adam Fuchs
• Created: 17/Feb/12 20:07
• Updated: 02/Nov/22 19:06
• Resolved: 02/Nov/22 19:06

Description

Tablet servers limit the number of files that can be opened for scans and for major compactions. The two main reasons for this limit was to reduce our impact on HDFS, primarily regarding connections to data nodes, and to limit our memory usage related to preloading file indexes. A third reason might be that disk thrashing could become a problem if we try to read from too many places at once.

Two improvements may have made (or may soon make) this limit obsolete: HDFS now pools connections, and RFile now uses a multi-level index. With these improvements, is it reasonable to lift some of our open file restrictions? The tradeoff on query side might be availability vs. overall resource usage. On the compaction side, the tradeoff is probably write replication vs. thrashing on reads. I think we can make an argument that queries should be available at almost any cost, but the compaction tradeoff is not as clear. We should test the efficiency of compacting a large number of files to get a better feeling for how the two extremes effect read and write performance across the system.

Issue Links

• is related to: ACCUMULO-747 Allow more configuration changes without restarting TServers
• supercedes: ACCUMULO-97 !METADATA scans could be delayed by regular scans