HBase
  1. HBase
  2. HBASE-24

Scaling: Too many open file handles to datanodes

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Blocker Blocker
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: regionserver
    • Labels:
      None

      Description

      We've been here before (HADOOP-2341).

      Today the rapleaf gave me an lsof listing from a regionserver. Had thousands of open sockets to datanodes all in ESTABLISHED and CLOSE_WAIT state. On average they seem to have about ten file descriptors/sockets open per region (They have 3 column families IIRC. Per family, can have between 1-5 or so mapfiles open per family – 3 is max... but compacting we open a new one, etc.).

      They have thousands of regions. 400 regions – ~100G, which is not that much – takes about 4k open file handles.

      If they want a regionserver to server a decent disk worths – 300-400G – then thats maybe 1600 regions... 16k file handles. If more than just 3 column families..... then we are in danger of blowing out limits if they are 32k.

      We've been here before with HADOOP-2341.

      A dfsclient that used non-blocking i/o would help applications like hbase (The datanode doesn't have this problem as bad – CLOSE_WAIT on regionserver side, the bulk of the open fds in the rapleaf log, don't have a corresponding open resource on datanode end).

      Could also just open mapfiles as needed, but that'd kill our random read performance and its bad enough already.

      1. MonitoredReader.java
        10 kB
        Luo Ning
      2. HBASE-823.patch
        1 kB
        Luo Ning

        Issue Links

          Activity

          stack created issue -
          Bryan Duxbury made changes -
          Field Original Value New Value
          Fix Version/s 0.17.0 [ 12312913 ]
          Owen O'Malley made changes -
          Fix Version/s 0.17.0 [ 12312913 ]
          Component/s contrib/hbase [ 12311752 ]
          Key HADOOP-2577 HBASE-24
          Project Hadoop Core [ 12310240 ] Hadoop HBase [ 12310753 ]
          Bryan Duxbury made changes -
          Component/s regionserver [ 12312139 ]
          Jim Kellerman made changes -
          Priority Major [ 3 ] Critical [ 2 ]
          Bryan Duxbury made changes -
          Fix Version/s 0.2.0 [ 12312955 ]
          Bryan Duxbury made changes -
          Summary [hbase] Scaling: Too many open file handles to datanodes Scaling: Too many open file handles to datanodes
          Bryan Duxbury made changes -
          Fix Version/s 0.2.0 [ 12312955 ]
          stack made changes -
          Fix Version/s 0.19.0 [ 12313364 ]
          Jim Kellerman made changes -
          Fix Version/s 0.19.0 [ 12313364 ]
          Fix Version/s 0.20.0 [ 12313474 ]
          stack made changes -
          Priority Critical [ 2 ] Blocker [ 1 ]
          Luo Ning made changes -
          Attachment hbase-0.18.0.iml [ 12397638 ]
          Luo Ning made changes -
          Attachment HBASE-823.patch [ 12397639 ]
          Luo Ning made changes -
          Attachment hbase-0.18.0.iml [ 12397638 ]
          Luo Ning made changes -
          Attachment MonitoredReader.java [ 12397640 ]
          stack made changes -
          Fix Version/s 0.21.0 [ 12313607 ]
          Fix Version/s 0.20.0 [ 12313474 ]
          stack made changes -
          Fix Version/s 0.22.0 [ 12314223 ]
          Fix Version/s 0.21.0 [ 12313607 ]
          stack made changes -
          Link This issue incorporates HBASE-2751 [ HBASE-2751 ]
          stack made changes -
          Fix Version/s 0.92.0 [ 12314223 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              stack
            • Votes:
              1 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

              • Created:
                Updated:

                Development