Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2443

org.apache.hadoop.fs.FSDataOutputStream.sync() is deprecated in hadoop 2.4

        Details

        • Type: Dependency upgrade
        • Status: Resolved
        • Priority: Major
        • Resolution: Fixed
        • Affects Version/s: 1.5.0, 1.5.0.1, 1.6.0
        • Fix Version/s: 1.6.0
        • Component/s: Sinks+Sources, Technical Debt
        • Labels:
          None
        • Environment:

          Linux PPC / x86

          Description

          HDFS sink uses this method, which is deprecated in hadoop 2.4.1, and no longer present in the current hadoop developments:

          java.lang.NoSuchMethodError: org/apache/hadoop/fs/FSDataOutputStream.sync()V
          at org.apache.flume.sink.hdfs.HDFSDataStream.sync(HDFSDataStream.java:131)
          at org.apache.flume.sink.hdfs.BucketWriter$6.call(BucketWriter.java:502)
          at org.apache.flume.sink.hdfs.BucketWriter$6.call(BucketWriter.java:499)
          at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:718)
          at org.apache.flume.sink.hdfs.BucketWriter.runPrivileged(BucketWriter.java:183)
          at org.apache.flume.sink.hdfs.BucketWriter.access$1700(BucketWriter.java:59)
          at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:715)
          at java.util.concurrent.FutureTask.run(FutureTask.java:273)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1176)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
          at java.lang.Thread.run(Thread.java:853)

          1. Text File
            FLUME-2443.patch
            6 kB
            Hari Shreedharan

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            hshreedharan Hari Shreedharan added a comment -

            We should be able to switch it out with hflush - though I am not sure if hadoop 1.x had hflush

            Show
            hshreedharan Hari Shreedharan added a comment - We should be able to switch it out with hflush - though I am not sure if hadoop 1.x had hflush
            Hide
            Permalink
            hshreedharan Hari Shreedharan added a comment -

            Looks like we'd have to use reflection.

            Show
            hshreedharan Hari Shreedharan added a comment - Looks like we'd have to use reflection.
            Hide
            Permalink
            hshreedharan Hari Shreedharan added a comment -

            I am working on a patch for this one.

            Show
            hshreedharan Hari Shreedharan added a comment - I am working on a patch for this one.
            Hide
            Permalink
            ctrl_alt_suppr Corentin Baron added a comment -

            Hi Hari,

            We've successfully tested your patch on both hadoop 2.4.1 and master branch, I guess you can commit your patch.

            Corentin.

            Show
            ctrl_alt_suppr Corentin Baron added a comment - Hi Hari, We've successfully tested your patch on both hadoop 2.4.1 and master branch, I guess you can commit your patch. Corentin.
            Hide
            Permalink
            hshreedharan Hari Shreedharan added a comment -

            I need to wait for some other committer to +1 this before it can be committed

            Show
            hshreedharan Hari Shreedharan added a comment - I need to wait for some other committer to +1 this before it can be committed
            Hide
            Permalink
            lukebrowning Luke Browning added a comment -

            Hi Hari,

            Any luck getting a second reviewer?

            Thanks,
            Luke

            Show
            lukebrowning Luke Browning added a comment - Hi Hari, Any luck getting a second reviewer? Thanks, Luke
            Hide
            Permalink
            ctrl_alt_suppr Corentin Baron added a comment -

            Hi Hari,

            This has been freezed for quite some time now, and this issue would prevent Flume from working with newer versions of Hadoop until resolved. Do you know anyone who could help reviewing your patch so we could mark the ticket as resolved?

            Thanks,
            Corentin.

            Show
            ctrl_alt_suppr Corentin Baron added a comment - Hi Hari, This has been freezed for quite some time now, and this issue would prevent Flume from working with newer versions of Hadoop until resolved. Do you know anyone who could help reviewing your patch so we could mark the ticket as resolved? Thanks, Corentin.
            Hide
            Permalink
            jira-bot ASF subversion and git services added a comment -

            Commit 990776427d0bb12e8775c31fc78d47bea8f6501f in flume's branch refs/heads/trunk from Jarek Jarcec Cecho
            [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=9907764 ]

            FLUME-2443: org.apache.hadoop.fs.FSDataOutputStream.sync() is deprecated in hadoop 2.4

            (Hari Shreedharan via Jarek Jarcec Cecho)

            Show
            jira-bot ASF subversion and git services added a comment - Commit 990776427d0bb12e8775c31fc78d47bea8f6501f in flume's branch refs/heads/trunk from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=9907764 ] FLUME-2443 : org.apache.hadoop.fs.FSDataOutputStream.sync() is deprecated in hadoop 2.4 (Hari Shreedharan via Jarek Jarcec Cecho)
            Hide
            Permalink
            jarcec Jarek Jarcec Cecho added a comment -

            +1

            Show
            jarcec Jarek Jarcec Cecho added a comment - +1
            Hide
            Permalink
            jira-bot ASF subversion and git services added a comment -

            Commit d47ee46b0f76d61f15e153f159e19387da46cc85 in flume's branch refs/heads/flume-1.6 from Jarek Jarcec Cecho
            [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=d47ee46 ]

            FLUME-2443: org.apache.hadoop.fs.FSDataOutputStream.sync() is deprecated in hadoop 2.4

            (Hari Shreedharan via Jarek Jarcec Cecho)

            Show
            jira-bot ASF subversion and git services added a comment - Commit d47ee46b0f76d61f15e153f159e19387da46cc85 in flume's branch refs/heads/flume-1.6 from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=d47ee46 ] FLUME-2443 : org.apache.hadoop.fs.FSDataOutputStream.sync() is deprecated in hadoop 2.4 (Hari Shreedharan via Jarek Jarcec Cecho)
            Hide
            Permalink
            jarcec Jarek Jarcec Cecho added a comment -

            Thank you for the contribution Hari Shreedharan, sorry that the review took so long!

            Show
            jarcec Jarek Jarcec Cecho added a comment - Thank you for the contribution Hari Shreedharan , sorry that the review took so long!
            Hide
            Permalink
            hudson Hudson added a comment -

            FAILURE: Integrated in Flume-trunk-hbase-98 #69 (See https://builds.apache.org/job/Flume-trunk-hbase-98/69/)
            FLUME-2443: org.apache.hadoop.fs.FSDataOutputStream.sync() is deprecated in hadoop 2.4 (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=990776427d0bb12e8775c31fc78d47bea8f6501f)

            • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java
            • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java
            • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java
            • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java
            Show
            hudson Hudson added a comment - FAILURE: Integrated in Flume-trunk-hbase-98 #69 (See https://builds.apache.org/job/Flume-trunk-hbase-98/69/ ) FLUME-2443 : org.apache.hadoop.fs.FSDataOutputStream.sync() is deprecated in hadoop 2.4 (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=990776427d0bb12e8775c31fc78d47bea8f6501f ) flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java
            Hide
            Permalink
            hudson Hudson added a comment -

            SUCCESS: Integrated in flume-trunk #712 (See https://builds.apache.org/job/flume-trunk/712/)
            FLUME-2443: org.apache.hadoop.fs.FSDataOutputStream.sync() is deprecated in hadoop 2.4 (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=990776427d0bb12e8775c31fc78d47bea8f6501f)

            • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java
            • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java
            • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java
            • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java
            Show
            hudson Hudson added a comment - SUCCESS: Integrated in flume-trunk #712 (See https://builds.apache.org/job/flume-trunk/712/ ) FLUME-2443 : org.apache.hadoop.fs.FSDataOutputStream.sync() is deprecated in hadoop 2.4 (jarcec: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=990776427d0bb12e8775c31fc78d47bea8f6501f ) flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java

              People

              • Assignee:
                hshreedharan Hari Shreedharan
                Reporter:
                ctrl_alt_suppr Corentin Baron
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Development