Flume
  1. Flume
  2. FLUME-1925

HDFS timeouts should not starve other threads

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v1.4.0
    • Component/s: None
    • Labels:
      None

      Description

      Currently it is possible to have one HDFS call cause another one from timing out because of the synchronized blocks inside the HDFS sink code. So when we calculate timeout, we end up calculating the time to enter the monitor + hdfs calls. We must change it to make sure only the HDFS calls count towards the timeout.

      1. FLUME-1925.patch
        20 kB
        Hari Shreedharan
      2. FLUME-1925-1.patch
        20 kB
        Hari Shreedharan

        Issue Links

          Activity

          Hari Shreedharan created issue -
          Hari Shreedharan made changes -
          Field Original Value New Value
          Attachment FLUME-1925.patch [ 12570614 ]
          Hari Shreedharan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Assignee Hari Shreedharan [ hshreedharan ]
          Hide
          Hari Shreedharan added a comment -

          I verified that the rebased patch works against secure HDFS.

          Show
          Hari Shreedharan added a comment - I verified that the rebased patch works against secure HDFS.
          Hari Shreedharan made changes -
          Attachment FLUME-1925-1.patch [ 12571339 ]
          Mike Percy made changes -
          Remote Link This issue links to "Review Board (Web Link)" [ 12048 ]
          Hide
          Mike Percy added a comment -

          Linking RB to the ticket for posterity

          Show
          Mike Percy added a comment - Linking RB to the ticket for posterity
          Hide
          Mike Percy added a comment -

          +1

          Show
          Mike Percy added a comment - +1
          Hide
          Mike Percy added a comment -

          Pushed to trunk & flume-1.4 branches. Thanks for the patch Hari!

          Show
          Mike Percy added a comment - Pushed to trunk & flume-1.4 branches. Thanks for the patch Hari!
          Mike Percy made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s v1.4.0 [ 12323372 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in flume-trunk #365 (See https://builds.apache.org/job/flume-trunk/365/)
          FLUME-1925. HDFS timeouts should not starve other threads. (Revision ed4cae79f8ab66a71960dfa2bdb848f8141ca6bd)

          Result = SUCCESS
          mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=ed4cae79f8ab66a71960dfa2bdb848f8141ca6bd
          Files :

          • flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java
          • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
          • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
          Show
          Hudson added a comment - Integrated in flume-trunk #365 (See https://builds.apache.org/job/flume-trunk/365/ ) FLUME-1925 . HDFS timeouts should not starve other threads. (Revision ed4cae79f8ab66a71960dfa2bdb848f8141ca6bd) Result = SUCCESS mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=ed4cae79f8ab66a71960dfa2bdb848f8141ca6bd Files : flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java

            People

            • Assignee:
              Hari Shreedharan
              Reporter:
              Hari Shreedharan
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development