Accumulo
  1. Accumulo
  2. ACCUMULO-2716

Duplicate connection loss logging in Writer

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.5.1
    • Fix Version/s: 1.5.2, 1.6.0
    • Component/s: client
    • Labels:

      Description

      Running CI with agitation, I see lots of duplicated messages in the monitor whenever a tserver dies.

      WARN Error connecting to tserver1.example.com:10011: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
      ERROR error sending update to tserver1.example.com:10011: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused

      These always occur in pairs, at the same millisecond, and coming from the same tserver. I think that they are updates to the metadata table coming from these tservers, like flushes or compactions that fail because the dead server was hosting the corresponding metadata tablet, but it doesn't really matter.

      The culprit is in Writer.java where we log-and-rethrow in updateServer():

          } catch (TTransportException e) {
            log.warn("Error connecting to " + server + ": " + e);
            throw e;
          }
      

      and then later log again in update():

            } catch (TException e) {
              log.error("error sending update to " + tabLoc.tablet_location + ": " + e);
              TabletLocator.getLocator(instance, table).invalidateCache(tabLoc.tablet_extent);
            }
      

        Activity

        Hide
        Mike Drob added a comment -

        Does not affect 1.4.x because the second log statement there was at trace level.

        Show
        Mike Drob added a comment - Does not affect 1.4.x because the second log statement there was at trace level.
        Hide
        ASF subversion and git services added a comment -

        Commit c62d676e1bb3a0e0f17817f9839378c54c2e3acf in accumulo's branch refs/heads/1.5.2-SNAPSHOT from Mike Drob
        [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=c62d676 ]

        ACCUMULO-2716 Remove log-and-rethrow in Writer

        Show
        ASF subversion and git services added a comment - Commit c62d676e1bb3a0e0f17817f9839378c54c2e3acf in accumulo's branch refs/heads/1.5.2-SNAPSHOT from Mike Drob [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=c62d676 ] ACCUMULO-2716 Remove log-and-rethrow in Writer
        Hide
        ASF subversion and git services added a comment -

        Commit c62d676e1bb3a0e0f17817f9839378c54c2e3acf in accumulo's branch refs/heads/1.6.0-SNAPSHOT from Mike Drob
        [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=c62d676 ]

        ACCUMULO-2716 Remove log-and-rethrow in Writer

        Show
        ASF subversion and git services added a comment - Commit c62d676e1bb3a0e0f17817f9839378c54c2e3acf in accumulo's branch refs/heads/1.6.0-SNAPSHOT from Mike Drob [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=c62d676 ] ACCUMULO-2716 Remove log-and-rethrow in Writer
        Hide
        ASF subversion and git services added a comment -

        Commit c62d676e1bb3a0e0f17817f9839378c54c2e3acf in accumulo's branch refs/heads/master from Mike Drob
        [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=c62d676 ]

        ACCUMULO-2716 Remove log-and-rethrow in Writer

        Show
        ASF subversion and git services added a comment - Commit c62d676e1bb3a0e0f17817f9839378c54c2e3acf in accumulo's branch refs/heads/master from Mike Drob [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=c62d676 ] ACCUMULO-2716 Remove log-and-rethrow in Writer

          People

          • Assignee:
            Mike Drob
            Reporter:
            Mike Drob
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development