Flume
  1. Flume
  2. FLUME-2027

Check for default replication fails on federated cluster in hdfs sink

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v1.4.0
    • Component/s: None
    • Labels:
      None

      Description

      Seeing the following exception when calling fs.getDefaultReplication() on a federated fs:

      13/04/26 08:52:12 ERROR hdfs.HDFSEventSink: process failed
      org.apache.hadoop.fs.viewfs.NotInMountpointException:
      getDefaultReplication on empty path is invalid
              at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDefaultReplication(ViewFileSystem.java:509)
              at org.apache.flume.sink.hdfs.AbstractHDFSWriter.getFsDesiredReplication(AbstractHDFSWriter.java:104)
              at org.apache.flume.sink.hdfs.AbstractHDFSWriter.isUnderReplicated(AbstractHDFSWriter.java:73)
              at org.apache.flume.sink.hdfs.BucketWriter.shouldRotate(BucketWriter.java:486)
              at org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:421)
              at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:392)
              at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
              at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
              at java.lang.Thread.run(Thread.java:722)
      
      1. FLUME-2027.patch
        5 kB
        Mike Percy
      2. FLUME-2027-1.patch
        6 kB
        Mike Percy

        Issue Links

          Activity

          Hide
          Mike Percy added a comment -

          It looks like we should be using fs.getDefaultReplication(Path f) instead. This is available in Hadoop 1.1.0 and Hadoop 2 I believe

          Show
          Mike Percy added a comment - It looks like we should be using fs.getDefaultReplication(Path f) instead. This is available in Hadoop 1.1.0 and Hadoop 2 I believe
          Hide
          Mike Percy added a comment -

          This fix requires kicking the Hadoop version up to 1.1.0

          Show
          Mike Percy added a comment - This fix requires kicking the Hadoop version up to 1.1.0
          Hide
          Mike Percy added a comment -

          Updated patch

          Show
          Mike Percy added a comment - Updated patch
          Hide
          Hari Shreedharan added a comment -

          +1. Running tests, will commit soon.

          Show
          Hari Shreedharan added a comment - +1. Running tests, will commit soon.
          Hide
          Hari Shreedharan added a comment -

          Committed, rev: fac09aa5f543f251acbe3aa205b470562b93b344. Thanks Mike!

          Show
          Hari Shreedharan added a comment - Committed, rev: fac09aa5f543f251acbe3aa205b470562b93b344. Thanks Mike!
          Hide
          Hudson added a comment -

          Integrated in flume-trunk #404 (See https://builds.apache.org/job/flume-trunk/404/)
          FLUME-2027. Check for default replication fails on federated cluster in hdfs sink (Revision fac09aa5f543f251acbe3aa205b470562b93b344)

          Result = SUCCESS
          hshreedharan : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=fac09aa5f543f251acbe3aa205b470562b93b344
          Files :

          • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java
          • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java
          • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java
          • flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java
          Show
          Hudson added a comment - Integrated in flume-trunk #404 (See https://builds.apache.org/job/flume-trunk/404/ ) FLUME-2027 . Check for default replication fails on federated cluster in hdfs sink (Revision fac09aa5f543f251acbe3aa205b470562b93b344) Result = SUCCESS hshreedharan : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=fac09aa5f543f251acbe3aa205b470562b93b344 Files : flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java

            People

            • Assignee:
              Mike Percy
              Reporter:
              Mike Percy
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development