Flume
  1. Flume
  2. FLUME-2443

org.apache.hadoop.fs.FSDataOutputStream.sync() is deprecated in hadoop 2.4

        Details

        • Type: Dependency upgrade Dependency upgrade
        • Status: Resolved
        • Priority: Major Major
        • Resolution: Fixed
        • Affects Version/s: v1.5.0, v1.5.0.1, v1.6.0
        • Fix Version/s: v1.6.0
        • Component/s: Sinks+Sources, Technical Debt
        • Labels:
          None
        • Environment:

          Linux PPC / x86

          Description

          HDFS sink uses this method, which is deprecated in hadoop 2.4.1, and no longer present in the current hadoop developments:

          java.lang.NoSuchMethodError: org/apache/hadoop/fs/FSDataOutputStream.sync()V
          at org.apache.flume.sink.hdfs.HDFSDataStream.sync(HDFSDataStream.java:131)
          at org.apache.flume.sink.hdfs.BucketWriter$6.call(BucketWriter.java:502)
          at org.apache.flume.sink.hdfs.BucketWriter$6.call(BucketWriter.java:499)
          at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:718)
          at org.apache.flume.sink.hdfs.BucketWriter.runPrivileged(BucketWriter.java:183)
          at org.apache.flume.sink.hdfs.BucketWriter.access$1700(BucketWriter.java:59)
          at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:715)
          at java.util.concurrent.FutureTask.run(FutureTask.java:273)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1176)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
          at java.lang.Thread.run(Thread.java:853)

          1. Text File
            FLUME-2443.patch
            6 kB
            Hari Shreedharan

            Activity

            Descending order - Click to sort in ascending order
            Hide
            Permalink
            Hudson added a comment -

            SUCCESS: Integrated in flume-trunk #712 (See https://builds.apache.org/job/flume-trunk/712/)
            FLUME-2443: org.apache.hadoop.fs.FSDataOutputStream.sync() is deprecated in hadoop 2.4 (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=990776427d0bb12e8775c31fc78d47bea8f6501f)

            • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java
            • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java
            • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java
            • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java
            Show
            Hudson added a comment - SUCCESS: Integrated in flume-trunk #712 (See https://builds.apache.org/job/flume-trunk/712/ ) FLUME-2443 : org.apache.hadoop.fs.FSDataOutputStream.sync() is deprecated in hadoop 2.4 (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=990776427d0bb12e8775c31fc78d47bea8f6501f ) flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java
            Hide
            Permalink
            Hudson added a comment -

            FAILURE: Integrated in Flume-trunk-hbase-98 #69 (See https://builds.apache.org/job/Flume-trunk-hbase-98/69/)
            FLUME-2443: org.apache.hadoop.fs.FSDataOutputStream.sync() is deprecated in hadoop 2.4 (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=990776427d0bb12e8775c31fc78d47bea8f6501f)

            • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java
            • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java
            • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java
            • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java
            Show
            Hudson added a comment - FAILURE: Integrated in Flume-trunk-hbase-98 #69 (See https://builds.apache.org/job/Flume-trunk-hbase-98/69/ ) FLUME-2443 : org.apache.hadoop.fs.FSDataOutputStream.sync() is deprecated in hadoop 2.4 (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=990776427d0bb12e8775c31fc78d47bea8f6501f ) flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java
            Jarek Jarcec Cecho made changes -
            Status Patch Available [ 10002 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            Hide
            Permalink
            Jarek Jarcec Cecho added a comment -

            Thank you for the contribution Hari Shreedharan, sorry that the review took so long!

            Show
            Jarek Jarcec Cecho added a comment - Thank you for the contribution Hari Shreedharan , sorry that the review took so long!
            Hide
            Permalink
            ASF subversion and git services added a comment -

            Commit d47ee46b0f76d61f15e153f159e19387da46cc85 in flume's branch refs/heads/flume-1.6 from Jarek Jarcec Cecho
            [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=d47ee46 ]

            FLUME-2443: org.apache.hadoop.fs.FSDataOutputStream.sync() is deprecated in hadoop 2.4

            (Hari Shreedharan via Jarek Jarcec Cecho)

            Show
            ASF subversion and git services added a comment - Commit d47ee46b0f76d61f15e153f159e19387da46cc85 in flume's branch refs/heads/flume-1.6 from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=d47ee46 ] FLUME-2443 : org.apache.hadoop.fs.FSDataOutputStream.sync() is deprecated in hadoop 2.4 (Hari Shreedharan via Jarek Jarcec Cecho)
            Hide
            Permalink
            Jarek Jarcec Cecho added a comment -

            +1

            Show
            Jarek Jarcec Cecho added a comment - +1
            Hide
            Permalink
            ASF subversion and git services added a comment -

            Commit 990776427d0bb12e8775c31fc78d47bea8f6501f in flume's branch refs/heads/trunk from Jarek Jarcec Cecho
            [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=9907764 ]

            FLUME-2443: org.apache.hadoop.fs.FSDataOutputStream.sync() is deprecated in hadoop 2.4

            (Hari Shreedharan via Jarek Jarcec Cecho)

            Show
            ASF subversion and git services added a comment - Commit 990776427d0bb12e8775c31fc78d47bea8f6501f in flume's branch refs/heads/trunk from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=9907764 ] FLUME-2443 : org.apache.hadoop.fs.FSDataOutputStream.sync() is deprecated in hadoop 2.4 (Hari Shreedharan via Jarek Jarcec Cecho)
            Hide
            Permalink
            Corentin Baron added a comment -

            Hi Hari,

            This has been freezed for quite some time now, and this issue would prevent Flume from working with newer versions of Hadoop until resolved. Do you know anyone who could help reviewing your patch so we could mark the ticket as resolved?

            Thanks,
            Corentin.

            Show
            Corentin Baron added a comment - Hi Hari, This has been freezed for quite some time now, and this issue would prevent Flume from working with newer versions of Hadoop until resolved. Do you know anyone who could help reviewing your patch so we could mark the ticket as resolved? Thanks, Corentin.
            Hide
            Permalink
            Luke Browning added a comment -

            Hi Hari,

            Any luck getting a second reviewer?

            Thanks,
            Luke

            Show
            Luke Browning added a comment - Hi Hari, Any luck getting a second reviewer? Thanks, Luke
            Hide
            Permalink
            Hari Shreedharan added a comment -

            I need to wait for some other committer to +1 this before it can be committed

            Show
            Hari Shreedharan added a comment - I need to wait for some other committer to +1 this before it can be committed
            Hide
            Permalink
            Corentin Baron added a comment -

            Hi Hari,

            We've successfully tested your patch on both hadoop 2.4.1 and master branch, I guess you can commit your patch.

            Corentin.

            Show
            Corentin Baron added a comment - Hi Hari, We've successfully tested your patch on both hadoop 2.4.1 and master branch, I guess you can commit your patch. Corentin.
            Hari Shreedharan made changes -
            Status Open [ 1 ] Patch Available [ 10002 ]
            Fix Version/s v1.6.0 [ 12327047 ]
            Hari Shreedharan made changes -
            Attachment FLUME-2443.patch [ 12660717 ]
            Hari Shreedharan made changes -
            Field Original Value New Value
            Assignee Hari Shreedharan [ hshreedharan ]
            Hide
            Permalink
            Hari Shreedharan added a comment -

            I am working on a patch for this one.

            Show
            Hari Shreedharan added a comment - I am working on a patch for this one.
            Hide
            Permalink
            Hari Shreedharan added a comment -

            Looks like we'd have to use reflection.

            Show
            Hari Shreedharan added a comment - Looks like we'd have to use reflection.
            Hide
            Permalink
            Hari Shreedharan added a comment -

            We should be able to switch it out with hflush - though I am not sure if hadoop 1.x had hflush

            Show
            Hari Shreedharan added a comment - We should be able to switch it out with hflush - though I am not sure if hadoop 1.x had hflush
            Corentin Baron created issue -

              People

              • Assignee:
                Hari Shreedharan
                Reporter:
                Corentin Baron
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Development