Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2427

java.lang.NoSuchMethodException and warning on HDFS (S3) sink

    Details

    • Type: Question
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.8.0
    • Component/s: None
    • Labels:
      None

      Description

      The below warning and Exception is thrown every time a file is written to S3 using HDFS sink. Looks like a jar mismatch to me. Tried latest hadoop and jets3 jars but didn't work

      17 Jul 2014 23:30:18,159 INFO [hdfs-s3sink-engagements-call-runner-6] (org.apache.flume.sink.hdfs.AbstractHDFSWriter.reflectGetNumCurrentReplicas:184) - FileSystem's output stream doesn't support getNumCurrentReplicas; --HDFS-826 not available; fsOut=org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream; err=java.lang.NoSuchMethodException: org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream.getNumCurrentReplicas()
      17 Jul 2014 23:30:18,160 WARN [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hdfs.BucketWriter.getRefIsClosed:210) - isFileClosed is not available in the version of HDFS being used. Flume will not attempt to close files if the close fails on the first attempt
      java.lang.NoSuchMethodException: org.apache.hadoop.fs.s3native.NativeS3FileSystem.isFileClosed(org.apache.hadoop.fs.Path)
      at java.lang.Class.getMethod(Class.java:1665)
      at org.apache.flume.sink.hdfs.BucketWriter.getRefIsClosed(BucketWriter.java:207)
      at org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:295)
      at org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:554)
      at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:426)
      at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
      at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
      at java.lang.Thread.run(Thread.java:744)

        Activity

        Hide
        wpwang Ping Wang added a comment -

        Thanks Mike!

        Show
        wpwang Ping Wang added a comment - Thanks Mike!
        Hide
        hudson Hudson added a comment -

        UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #240 (See https://builds.apache.org/job/Flume-trunk-hbase-1/240/)
        FLUME-2427. Be less verbose in isFileClosed() notification (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=630083228364f6cd5dc18839fbd4b167cb615b38)

        • (edit) flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
        Show
        hudson Hudson added a comment - UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #240 (See https://builds.apache.org/job/Flume-trunk-hbase-1/240/ ) FLUME-2427 . Be less verbose in isFileClosed() notification (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=630083228364f6cd5dc18839fbd4b167cb615b38 ) (edit) flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 630083228364f6cd5dc18839fbd4b167cb615b38 in flume's branch refs/heads/trunk from Ping Wang
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=6300832 ]

        FLUME-2427. Be less verbose in isFileClosed() notification

        Some versions of HDFS and implementations of FileSystem do not implement
        the isFileClosed() method.

        We perform a check for that method each time a new file is opened. This
        patch lowers the severity of the message printed when we detect that the
        method is not present from LOG.warn() to LOG.info().

        We also remove the stack trace from the log message.

        (Ping Wang via Mike Percy)

        Show
        jira-bot ASF subversion and git services added a comment - Commit 630083228364f6cd5dc18839fbd4b167cb615b38 in flume's branch refs/heads/trunk from Ping Wang [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=6300832 ] FLUME-2427 . Be less verbose in isFileClosed() notification Some versions of HDFS and implementations of FileSystem do not implement the isFileClosed() method. We perform a check for that method each time a new file is opened. This patch lowers the severity of the message printed when we detect that the method is not present from LOG.warn() to LOG.info(). We also remove the stack trace from the log message. (Ping Wang via Mike Percy)
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 9604c5ae1c9f2b266200889219b5604fe0e756e4 in flume's branch refs/heads/trunk from Ping Wang
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=9604c5a ]

        FLUME-2427. Be less verbose in isFileClosed() notification

        Some versions of HDFS and implementations of Filesystem do not implement
        the isFileClosed() method.

        We perform a check for that method each time a new file is opened. This
        patch lowers the severity of the message printed when we detect that the
        method id not present from LOG.warn() to LOG.info().

        We also remove the stack trace from the log message.

        (Ping Wang via Mike Percy)

        Show
        jira-bot ASF subversion and git services added a comment - Commit 9604c5ae1c9f2b266200889219b5604fe0e756e4 in flume's branch refs/heads/trunk from Ping Wang [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=9604c5a ] FLUME-2427 . Be less verbose in isFileClosed() notification Some versions of HDFS and implementations of Filesystem do not implement the isFileClosed() method. We perform a check for that method each time a new file is opened. This patch lowers the severity of the message printed when we detect that the method id not present from LOG.warn() to LOG.info(). We also remove the stack trace from the log message. (Ping Wang via Mike Percy)
        Hide
        mpercy Mike Percy added a comment -

        +1 on Ping Wang's patch, I am also going to lower this to info() and tweak the message to say "version of distributed filesystem" instead of "version of HDFS".

        Show
        mpercy Mike Percy added a comment - +1 on Ping Wang 's patch, I am also going to lower this to info() and tweak the message to say "version of distributed filesystem" instead of "version of HDFS".
        Hide
        nperiwal Narayan Periwal added a comment -

        I also encountered this issue while working with LocalFileSystem. The exception coming in the log looks confusing as suggested by everyone. It will be appreciable if it can be fixed.

        Show
        nperiwal Narayan Periwal added a comment - I also encountered this issue while working with LocalFileSystem. The exception coming in the log looks confusing as suggested by everyone. It will be appreciable if it can be fixed.
        Hide
        wpwang Ping Wang added a comment -

        I also hit this issue, and agree that it will cause confusion and it’s better to hidden the exception from log. I attached my fix for this, please review it. Thanks!

        Show
        wpwang Ping Wang added a comment - I also hit this issue, and agree that it will cause confusion and it’s better to hidden the exception from log. I attached my fix for this, please review it. Thanks!
        Hide
        diranged Matt Wise added a comment -

        This may only be a warning – but it shows up on nearly every single file-write in the logs. Its extremely confusing for engineers, and its just bad-practice to leave an exception being printed like this that cannot be hidden through log4j. This really should be fixed, it should be a simple patch.

        Show
        diranged Matt Wise added a comment - This may only be a warning – but it shows up on nearly every single file-write in the logs. Its extremely confusing for engineers, and its just bad-practice to leave an exception being printed like this that cannot be hidden through log4j. This really should be fixed, it should be a simple patch.
        Hide
        hshreedharan Hari Shreedharan added a comment -

        It is only a warning that it was not found. It does not cause any issues.

        Show
        hshreedharan Hari Shreedharan added a comment - It is only a warning that it was not found. It does not cause any issues.
        Hide
        russellcardullo Russell Cardullo added a comment -

        After upgrading to Flume 1.5 I also see this message when writing to S3 but this hasn't caused any issues for me.

        This looks to have been introduced in FLUME-2357 and is caused by the fact that NativeS3FileSystem does not have an isFileClosed method. The message just indicates that Flume will only attempt to close the file once regardless of the settings for hdfs.closeTries.

        Show
        russellcardullo Russell Cardullo added a comment - After upgrading to Flume 1.5 I also see this message when writing to S3 but this hasn't caused any issues for me. This looks to have been introduced in FLUME-2357 and is caused by the fact that NativeS3FileSystem does not have an isFileClosed method. The message just indicates that Flume will only attempt to close the file once regardless of the settings for hdfs.closeTries .

          People

          • Assignee:
            wpwang Ping Wang
            Reporter:
            bijithkumar Bijith Kumar
          • Votes:
            2 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development