Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The FileInputStreamCache currently defaults to holding only 10 input stream pairs (corresponding to 10 blocks). In many HBase workloads, the region server will be issuing random reads against a local file which is 2-4GB in size or even larger (hence 20+ blocks).

      Given that the memory usage for caching these input streams is low, and applications like HBase tend to already increase their ulimit -n substantially (eg up to 32,000), I think we should raise the default cache size to 50 or more. In the rare case that someone has an application which uses local reads with hundreds of open blocks and can't feasibly raise their ulimit -n, they can lower the limit appropriately.

      1. hdfs-4418.txt
        2 kB
        Todd Lipcon

        Activity

        Hide
        Todd Lipcon added a comment -

        Attached patch increases the cache size to 100 blocks (12GB file at default 128mb block size).

        I also dropped the cache expiry interval to 5 seconds, so that unless the workload is truly hitting a large number of blocks frequently, it won't accumulate a lot of file descriptors. Given that caching the socket only matters when you're doing lots of short reads that hit buffer cache, I think it's reasonable to have a short expiry period. For comparison, the keepalive period for non-local reads is only 1sec.

        Show
        Todd Lipcon added a comment - Attached patch increases the cache size to 100 blocks (12GB file at default 128mb block size). I also dropped the cache expiry interval to 5 seconds, so that unless the workload is truly hitting a large number of blocks frequently, it won't accumulate a lot of file descriptors. Given that caching the socket only matters when you're doing lots of short reads that hit buffer cache, I think it's reasonable to have a short expiry period. For comparison, the keepalive period for non-local reads is only 1sec.
        Hide
        stack added a comment -

        +1 on patch. LGTM.

        Show
        stack added a comment - +1 on patch. LGTM.
        Hide
        Colin Patrick McCabe added a comment -

        looks reasonable to me; thanks, Todd.

        Show
        Colin Patrick McCabe added a comment - looks reasonable to me; thanks, Todd.
        Hide
        Todd Lipcon added a comment -

        Committed to branch.

        Show
        Todd Lipcon added a comment - Committed to branch.

          People

          • Assignee:
            Todd Lipcon
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development