Flume
  1. Flume
  2. FLUME-1925

HDFS timeouts should not starve other threads

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v1.4.0
    • Component/s: None
    • Labels:
      None

      Description

      Currently it is possible to have one HDFS call cause another one from timing out because of the synchronized blocks inside the HDFS sink code. So when we calculate timeout, we end up calculating the time to enter the monitor + hdfs calls. We must change it to make sure only the HDFS calls count towards the timeout.

      1. FLUME-1925-1.patch
        20 kB
        Hari Shreedharan
      2. FLUME-1925.patch
        20 kB
        Hari Shreedharan

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in flume-trunk #365 (See https://builds.apache.org/job/flume-trunk/365/)
          FLUME-1925. HDFS timeouts should not starve other threads. (Revision ed4cae79f8ab66a71960dfa2bdb848f8141ca6bd)

          Result = SUCCESS
          mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=ed4cae79f8ab66a71960dfa2bdb848f8141ca6bd
          Files :

          • flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java
          • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
          • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
          Show
          Hudson added a comment - Integrated in flume-trunk #365 (See https://builds.apache.org/job/flume-trunk/365/ ) FLUME-1925 . HDFS timeouts should not starve other threads. (Revision ed4cae79f8ab66a71960dfa2bdb848f8141ca6bd) Result = SUCCESS mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=ed4cae79f8ab66a71960dfa2bdb848f8141ca6bd Files : flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
          Hide
          Mike Percy added a comment -

          Pushed to trunk & flume-1.4 branches. Thanks for the patch Hari!

          Show
          Mike Percy added a comment - Pushed to trunk & flume-1.4 branches. Thanks for the patch Hari!
          Hide
          Mike Percy added a comment -

          +1

          Show
          Mike Percy added a comment - +1
          Hide
          Mike Percy added a comment -

          Linking RB to the ticket for posterity

          Show
          Mike Percy added a comment - Linking RB to the ticket for posterity
          Hide
          Hari Shreedharan added a comment -

          I verified that the rebased patch works against secure HDFS.

          Show
          Hari Shreedharan added a comment - I verified that the rebased patch works against secure HDFS.

            People

            • Assignee:
              Hari Shreedharan
              Reporter:
              Hari Shreedharan
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development