Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-17311

RBF: ConnectionManager creatorQueue should offer a pool that is not already in creatorQueue.

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.5.0
    • 3.5.0
    • rbf
    • Reviewed

    Description

      In the Router, find blow log

       
      2023-12-29 15:18:54,799 ERROR org.apache.hadoop.hdfs.server.federation.router.ConnectionManager: Cannot add more than 2048 connections at the same time

       
      The log indicates that ConnectionManager.creatorQueue is full at a certain point. But my cluster does not have so many users cloud reach up 2048 pair of <user,nn>.

      This may be due to the following reasons:

      1. ConnectionManager.creatorQueue is a queue that will be offered ConnectionPool if ConnectionContext is not enough.
      2. ConnectionCreator thread will consume from creatorQueue and make more ConnectionContexts for a ConnectionPool.
      3. Client will concurrent invoke for ConnectionManager.getConnection() for a same user. And this maybe lead to add many same ConnectionPool into ConnectionManager.creatorQueue.
      4. When creatorQueue is full, a new ConnectionPool will not be added in successfully and log this error. This maybe lead to a really new ConnectionPool clould not produce more ConnectionContexts for new user.

      So this pr try to make creatorQueue will not add same ConnectionPool at once.

      Attachments

        Issue Links

          Activity

            People

              liuguanghua liuguanghua
              liuguanghua liuguanghua
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: