Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-13891 HDFS RBF stabilization phase I
  3. HDFS-14114

RBF: MIN_ACTIVE_RATIO should be configurable

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.3.0, HDFS-13891
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The following code contains

            if (timeSinceLastActive > connectionCleanupPeriodMs ||
                active < MIN_ACTIVE_RATIO * total) {
              // Remove and close 1 connection
              List<ConnectionContext> conns = pool.removeConnections(1);
              for (ConnectionContext conn : conns) {
                conn.close();
              }
              LOG.debug("Removed connection {} used {} seconds ago. " +
                      "Pool has {}/{} connections", pool.getConnectionPoolId(),
                  TimeUnit.MILLISECONDS.toSeconds(timeSinceLastActive),
                  pool.getNumConnections(), pool.getMaxSize());
            }
      ...
                  if (pool.getNumConnections() < pool.getMaxSize() &&
                      active >= MIN_ACTIVE_RATIO * total) {
                    ConnectionContext conn = pool.newConnection();
                    pool.addConnection(conn);
                  } else {
                    LOG.debug("Cannot add more than {} connections to {}",
                        pool.getMaxSize(), pool);
                  }
      

      It affects cleanup and creating Connections. Maybe it should be configurable so that we can reconfig it to improve performance

        Attachments

        1. HDFS-14114.001.patch
          10 kB
          Fei Hui
        2. HDFS-14114.002.patch
          10 kB
          Fei Hui
        3. HDFS-14114.003.patch
          11 kB
          Fei Hui
        4. HDFS-14114.004.patch
          11 kB
          Fei Hui
        5. HDFS-14114.005.patch
          12 kB
          Fei Hui
        6. HDFS-14114.006.patch
          12 kB
          Fei Hui
        7. HDFS-14114.007.patch
          12 kB
          Fei Hui
        8. HDFS-14114.008.patch
          12 kB
          Fei Hui
        9. HDFS-14114-HDFS-13891.001.patch
          12 kB
          Yiqun Lin
        10. HDFS-14114-HDFS-13891.002.patch
          13 kB
          Fei Hui

          Activity

            People

            • Assignee:
              ferhui Fei Hui
              Reporter:
              ferhui Fei Hui
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: