Flume
  1. Flume
  2. FLUME-2443

org.apache.hadoop.fs.FSDataOutputStream.sync() is deprecated in hadoop 2.4

    Details

    • Type: Dependency upgrade Dependency upgrade
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v1.5.0, v1.5.0.1, v1.6.0
    • Fix Version/s: v1.6.0
    • Labels:
      None
    • Environment:

      Linux PPC / x86

      Description

      HDFS sink uses this method, which is deprecated in hadoop 2.4.1, and no longer present in the current hadoop developments:

      java.lang.NoSuchMethodError: org/apache/hadoop/fs/FSDataOutputStream.sync()V
      at org.apache.flume.sink.hdfs.HDFSDataStream.sync(HDFSDataStream.java:131)
      at org.apache.flume.sink.hdfs.BucketWriter$6.call(BucketWriter.java:502)
      at org.apache.flume.sink.hdfs.BucketWriter$6.call(BucketWriter.java:499)
      at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:718)
      at org.apache.flume.sink.hdfs.BucketWriter.runPrivileged(BucketWriter.java:183)
      at org.apache.flume.sink.hdfs.BucketWriter.access$1700(BucketWriter.java:59)
      at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:715)
      at java.util.concurrent.FutureTask.run(FutureTask.java:273)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1176)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
      at java.lang.Thread.run(Thread.java:853)

      1. FLUME-2443.patch
        6 kB
        Hari Shreedharan

        Activity

        Hide
        Hudson added a comment -

        SUCCESS: Integrated in flume-trunk #712 (See https://builds.apache.org/job/flume-trunk/712/)
        FLUME-2443: org.apache.hadoop.fs.FSDataOutputStream.sync() is deprecated in hadoop 2.4 (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=990776427d0bb12e8775c31fc78d47bea8f6501f)

        • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java
        • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java
        • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java
        • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java
        Show
        Hudson added a comment - SUCCESS: Integrated in flume-trunk #712 (See https://builds.apache.org/job/flume-trunk/712/ ) FLUME-2443 : org.apache.hadoop.fs.FSDataOutputStream.sync() is deprecated in hadoop 2.4 (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=990776427d0bb12e8775c31fc78d47bea8f6501f ) flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Flume-trunk-hbase-98 #69 (See https://builds.apache.org/job/Flume-trunk-hbase-98/69/)
        FLUME-2443: org.apache.hadoop.fs.FSDataOutputStream.sync() is deprecated in hadoop 2.4 (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=990776427d0bb12e8775c31fc78d47bea8f6501f)

        • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java
        • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java
        • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java
        • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java
        Show
        Hudson added a comment - FAILURE: Integrated in Flume-trunk-hbase-98 #69 (See https://builds.apache.org/job/Flume-trunk-hbase-98/69/ ) FLUME-2443 : org.apache.hadoop.fs.FSDataOutputStream.sync() is deprecated in hadoop 2.4 (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=990776427d0bb12e8775c31fc78d47bea8f6501f ) flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java
        Hide
        Jarek Jarcec Cecho added a comment -

        Thank you for the contribution Hari Shreedharan, sorry that the review took so long!

        Show
        Jarek Jarcec Cecho added a comment - Thank you for the contribution Hari Shreedharan , sorry that the review took so long!
        Hide
        ASF subversion and git services added a comment -

        Commit d47ee46b0f76d61f15e153f159e19387da46cc85 in flume's branch refs/heads/flume-1.6 from Jarek Jarcec Cecho
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=d47ee46 ]

        FLUME-2443: org.apache.hadoop.fs.FSDataOutputStream.sync() is deprecated in hadoop 2.4

        (Hari Shreedharan via Jarek Jarcec Cecho)

        Show
        ASF subversion and git services added a comment - Commit d47ee46b0f76d61f15e153f159e19387da46cc85 in flume's branch refs/heads/flume-1.6 from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=d47ee46 ] FLUME-2443 : org.apache.hadoop.fs.FSDataOutputStream.sync() is deprecated in hadoop 2.4 (Hari Shreedharan via Jarek Jarcec Cecho)
        Hide
        Jarek Jarcec Cecho added a comment -

        +1

        Show
        Jarek Jarcec Cecho added a comment - +1
        Hide
        ASF subversion and git services added a comment -

        Commit 990776427d0bb12e8775c31fc78d47bea8f6501f in flume's branch refs/heads/trunk from Jarek Jarcec Cecho
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=9907764 ]

        FLUME-2443: org.apache.hadoop.fs.FSDataOutputStream.sync() is deprecated in hadoop 2.4

        (Hari Shreedharan via Jarek Jarcec Cecho)

        Show
        ASF subversion and git services added a comment - Commit 990776427d0bb12e8775c31fc78d47bea8f6501f in flume's branch refs/heads/trunk from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=9907764 ] FLUME-2443 : org.apache.hadoop.fs.FSDataOutputStream.sync() is deprecated in hadoop 2.4 (Hari Shreedharan via Jarek Jarcec Cecho)
        Hide
        Corentin Baron added a comment -

        Hi Hari,

        This has been freezed for quite some time now, and this issue would prevent Flume from working with newer versions of Hadoop until resolved. Do you know anyone who could help reviewing your patch so we could mark the ticket as resolved?

        Thanks,
        Corentin.

        Show
        Corentin Baron added a comment - Hi Hari, This has been freezed for quite some time now, and this issue would prevent Flume from working with newer versions of Hadoop until resolved. Do you know anyone who could help reviewing your patch so we could mark the ticket as resolved? Thanks, Corentin.
        Hide
        Luke Browning added a comment -

        Hi Hari,

        Any luck getting a second reviewer?

        Thanks,
        Luke

        Show
        Luke Browning added a comment - Hi Hari, Any luck getting a second reviewer? Thanks, Luke
        Hide
        Hari Shreedharan added a comment -

        I need to wait for some other committer to +1 this before it can be committed

        Show
        Hari Shreedharan added a comment - I need to wait for some other committer to +1 this before it can be committed
        Hide
        Corentin Baron added a comment -

        Hi Hari,

        We've successfully tested your patch on both hadoop 2.4.1 and master branch, I guess you can commit your patch.

        Corentin.

        Show
        Corentin Baron added a comment - Hi Hari, We've successfully tested your patch on both hadoop 2.4.1 and master branch, I guess you can commit your patch. Corentin.
        Hide
        Hari Shreedharan added a comment -

        I am working on a patch for this one.

        Show
        Hari Shreedharan added a comment - I am working on a patch for this one.
        Hide
        Hari Shreedharan added a comment -

        Looks like we'd have to use reflection.

        Show
        Hari Shreedharan added a comment - Looks like we'd have to use reflection.
        Hide
        Hari Shreedharan added a comment -

        We should be able to switch it out with hflush - though I am not sure if hadoop 1.x had hflush

        Show
        Hari Shreedharan added a comment - We should be able to switch it out with hflush - though I am not sure if hadoop 1.x had hflush

          People

          • Assignee:
            Hari Shreedharan
            Reporter:
            Corentin Baron
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development