HBase
  1. HBase
  2. HBASE-24

Scaling: Too many open file handles to datanodes

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Blocker Blocker
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: regionserver
    • Labels:
      None

      Description

      We've been here before (HADOOP-2341).

      Today the rapleaf gave me an lsof listing from a regionserver. Had thousands of open sockets to datanodes all in ESTABLISHED and CLOSE_WAIT state. On average they seem to have about ten file descriptors/sockets open per region (They have 3 column families IIRC. Per family, can have between 1-5 or so mapfiles open per family – 3 is max... but compacting we open a new one, etc.).

      They have thousands of regions. 400 regions – ~100G, which is not that much – takes about 4k open file handles.

      If they want a regionserver to server a decent disk worths – 300-400G – then thats maybe 1600 regions... 16k file handles. If more than just 3 column families..... then we are in danger of blowing out limits if they are 32k.

      We've been here before with HADOOP-2341.

      A dfsclient that used non-blocking i/o would help applications like hbase (The datanode doesn't have this problem as bad – CLOSE_WAIT on regionserver side, the bulk of the open fds in the rapleaf log, don't have a corresponding open resource on datanode end).

      Could also just open mapfiles as needed, but that'd kill our random read performance and its bad enough already.

      1. HBASE-823.patch
        1 kB
        Luo Ning
      2. MonitoredReader.java
        10 kB
        Luo Ning

        Issue Links

          Activity

          stack created issue -
          Bryan Duxbury made changes -
          Field Original Value New Value
          Fix Version/s 0.17.0 [ 12312913 ]
          Owen O'Malley made changes -
          Fix Version/s 0.17.0 [ 12312913 ]
          Component/s contrib/hbase [ 12311752 ]
          Key HADOOP-2577 HBASE-24
          Project Hadoop Core [ 12310240 ] Hadoop HBase [ 12310753 ]
          Bryan Duxbury made changes -
          Component/s regionserver [ 12312139 ]
          Jim Kellerman made changes -
          Priority Major [ 3 ] Critical [ 2 ]
          Bryan Duxbury made changes -
          Fix Version/s 0.2.0 [ 12312955 ]
          Bryan Duxbury made changes -
          Summary [hbase] Scaling: Too many open file handles to datanodes Scaling: Too many open file handles to datanodes
          Bryan Duxbury made changes -
          Fix Version/s 0.2.0 [ 12312955 ]
          stack made changes -
          Fix Version/s 0.19.0 [ 12313364 ]
          Jim Kellerman made changes -
          Fix Version/s 0.19.0 [ 12313364 ]
          Fix Version/s 0.20.0 [ 12313474 ]
          stack made changes -
          Priority Critical [ 2 ] Blocker [ 1 ]
          Luo Ning made changes -
          Attachment hbase-0.18.0.iml [ 12397638 ]
          Luo Ning made changes -
          Attachment HBASE-823.patch [ 12397639 ]
          Luo Ning made changes -
          Attachment hbase-0.18.0.iml [ 12397638 ]
          Luo Ning made changes -
          Attachment MonitoredReader.java [ 12397640 ]
          stack made changes -
          Fix Version/s 0.21.0 [ 12313607 ]
          Fix Version/s 0.20.0 [ 12313474 ]
          stack made changes -
          Fix Version/s 0.22.0 [ 12314223 ]
          Fix Version/s 0.21.0 [ 12313607 ]
          stack made changes -
          Link This issue incorporates HBASE-2751 [ HBASE-2751 ]
          stack made changes -
          Fix Version/s 0.92.0 [ 12314223 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              stack
            • Votes:
              1 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

              • Created:
                Updated:

                Development