[HDFS-13834] RBF: Connection creator thread should catch Throwable

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 3.3.0, HDFS-13891
• Hadoop Flags: Reviewed
• Assignee: CR Hota
• Reporter: CR Hota
• Created: 17/Aug/18 22:21
• Updated: 12/Sep/19 19:51
• Resolved: 03/Nov/18 01:47

Description

Connection creator thread is a single thread thats responsible for creating all downstream namenode connections.

This is very critical thread and hence should not die understand exception/error scenarios.

We saw this behavior in production systems where the thread died leaving the router process in bad state.

The thread should also catch a generic error/exception.

    @Override
    public void run() {
      while (this.running) {
        try {
          ConnectionPool pool = this.queue.take();
          try {
            int total = pool.getNumConnections();
            int active = pool.getNumActiveConnections();
            if (pool.getNumConnections() < pool.getMaxSize() &&
                active >= MIN_ACTIVE_RATIO * total) {
              ConnectionContext conn = pool.newConnection();
              pool.addConnection(conn);
            } else {
              LOG.debug("Cannot add more than {} connections to {}",
                  pool.getMaxSize(), pool);
            }
          } catch (IOException e) {
            LOG.error("Cannot create a new connection", e);
          }
        } catch (InterruptedException e) {
          LOG.error("The connection creator was interrupted");
          this.running = false;
        }
      }