[HBASE-1024] Rearchitect regionserver I/O

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.20.0
• Component/s: io, regionserver
• Assignee: Unassigned
• Reporter: Andrew Kyle Purtell
• Created: 23/Nov/08 22:40
• Updated: 12/Oct/12 06:12
• Resolved: 16/Dec/09 00:12

Description

Consider supporting:

• 2GB store files

• 1TB per node (500 store files)

• Cell values up to ~100MB

• Typical use case of RS running with 1GB of heap only

Some ideas:

• Drop MapFile and make a custom store file format with (competing) design goals:
  • heap efficiency
  • fast lookups
  • minimize I/O operations
  • optimize for typical DFS blocksizes (8MB, 64MB)

• MRU cache for filehandles and store file indexes

• Memory mapped store file indexes – don't hold the indexes in heap; rely on the OS blockcache for performance

• "Zero copy" I/O from IPC to store file and vice versa, like NIO buffers