Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2594

Close Async HBase Client if there are large number of consecutive timeouts

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.0
    • Component/s: None
    • Labels:
      None

      Description

      A large number of timeouts can lead to several thousands of messages being buffered but never removed from the HBaseClient's buffers. This can cause a major heap spike. So we should close and dereference the HBaseClient if we hit many failures from the HBase side to clear up the buffers.

      1. FLUME-2594.patch
        6 kB
        Hari Shreedharan

        Issue Links

          Activity

          Hide
          jarcec Jarek Jarcec Cecho added a comment -

          +1

          Show
          jarcec Jarek Jarcec Cecho added a comment - +1
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 82631f811a7c6a4d8e6ec886f80d1a21876947e5 in flume's branch refs/heads/trunk from Jarek Jarcec Cecho
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=82631f8 ]

          FLUME-2594: Close Async HBase Client if there are large number of consecutive timeouts

          (Hari Shreedharan via Jarek Jarcec Cecho)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 82631f811a7c6a4d8e6ec886f80d1a21876947e5 in flume's branch refs/heads/trunk from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=82631f8 ] FLUME-2594 : Close Async HBase Client if there are large number of consecutive timeouts (Hari Shreedharan via Jarek Jarcec Cecho)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit da294a164c27d400dc0b7bc6754c883a4eb5ed24 in flume's branch refs/heads/flume-1.6 from Jarek Jarcec Cecho
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=da294a1 ]

          FLUME-2594: Close Async HBase Client if there are large number of consecutive timeouts

          (Hari Shreedharan via Jarek Jarcec Cecho)

          Show
          jira-bot ASF subversion and git services added a comment - Commit da294a164c27d400dc0b7bc6754c883a4eb5ed24 in flume's branch refs/heads/flume-1.6 from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=da294a1 ] FLUME-2594 : Close Async HBase Client if there are large number of consecutive timeouts (Hari Shreedharan via Jarek Jarcec Cecho)
          Hide
          jarcec Jarek Jarcec Cecho added a comment -

          Thank you for the contribution Hari Shreedharan!

          Show
          jarcec Jarek Jarcec Cecho added a comment - Thank you for the contribution Hari Shreedharan !
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in flume-trunk #708 (See https://builds.apache.org/job/flume-trunk/708/)
          FLUME-2594: Close Async HBase Client if there are large number of consecutive timeouts (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=82631f811a7c6a4d8e6ec886f80d1a21876947e5)

          • flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/AsyncHBaseSink.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in flume-trunk #708 (See https://builds.apache.org/job/flume-trunk/708/ ) FLUME-2594 : Close Async HBase Client if there are large number of consecutive timeouts (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=82631f811a7c6a4d8e6ec886f80d1a21876947e5 ) flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/AsyncHBaseSink.java
          Hide
          hudson Hudson added a comment -

          UNSTABLE: Integrated in Flume-trunk-hbase-98 #65 (See https://builds.apache.org/job/Flume-trunk-hbase-98/65/)
          FLUME-2594: Close Async HBase Client if there are large number of consecutive timeouts (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=82631f811a7c6a4d8e6ec886f80d1a21876947e5)

          • flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/AsyncHBaseSink.java
          Show
          hudson Hudson added a comment - UNSTABLE: Integrated in Flume-trunk-hbase-98 #65 (See https://builds.apache.org/job/Flume-trunk-hbase-98/65/ ) FLUME-2594 : Close Async HBase Client if there are large number of consecutive timeouts (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=82631f811a7c6a4d8e6ec886f80d1a21876947e5 ) flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/AsyncHBaseSink.java

            People

            • Assignee:
              hshreedharan Hari Shreedharan
              Reporter:
              hshreedharan Hari Shreedharan
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development