Flume
  1. Flume
  2. FLUME-774

Move HDFS sink into a separate module

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: NG alpha 1
    • Fix Version/s: NG alpha 1
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      The HDFS sink is currently in the flume-ng-core maven module. This means that Flume core depends on Hadoop. Ideally, we would move this to a separate maven module (e.g. flume-ng-hadoop) so this dependency was optional.

      1. Flume-774.patch.4
        91 kB
        Prasad Mujumdar
      2. Flume-774.patch.3
        91 kB
        Prasad Mujumdar
      3. Flume-774.patch.2
        91 kB
        Prasad Mujumdar
      4. Flume-774.patch
        88 kB
        Prasad Mujumdar

        Activity

        Hide
        Hudson added a comment -

        Integrated in flume-728 #36 (See https://builds.apache.org/job/flume-728/36/)
        FLUME-774. Move HDFS sink into a separate module

        (Prasad Mujumdar via Arvind Prabhakar)

        arvind : http://svn.apache.org/viewvc/?view=rev&rev=1182548
        Files :

        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/hdfs/HDFSFormatterFactory.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/hdfs/HDFSTextFormatter.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/hdfs/HDFSWritableFormatter.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/hdfs/HDFSWriterFactory.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java
        • /incubator/flume/branches/flume-728/flume-ng-node/pom.xml
        • /incubator/flume/branches/flume-728/flume-ng-sinks
        • /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink
        • /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/pom.xml
        • /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src
        • /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/main
        • /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/main/java
        • /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/main/java/org
        • /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache
        • /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume
        • /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink
        • /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs
        • /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
        • /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java
        • /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java
        • /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
        • /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSFormatterFactory.java
        • /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java
        • /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSTextFormatter.java
        • /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWritableFormatter.java
        • /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java
        • /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriterFactory.java
        • /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/test
        • /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/test/java
        • /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/test/java/org
        • /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache
        • /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume
        • /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink
        • /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs
        • /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java
        • /incubator/flume/branches/flume-728/flume-ng-sinks/pom.xml
        • /incubator/flume/branches/flume-728/pom.xml
        Show
        Hudson added a comment - Integrated in flume-728 #36 (See https://builds.apache.org/job/flume-728/36/ ) FLUME-774 . Move HDFS sink into a separate module (Prasad Mujumdar via Arvind Prabhakar) arvind : http://svn.apache.org/viewvc/?view=rev&rev=1182548 Files : /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/hdfs/HDFSFormatterFactory.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/hdfs/HDFSTextFormatter.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/hdfs/HDFSWritableFormatter.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/hdfs/HDFSWriterFactory.java /incubator/flume/branches/flume-728/flume-ng-core/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java /incubator/flume/branches/flume-728/flume-ng-node/pom.xml /incubator/flume/branches/flume-728/flume-ng-sinks /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/pom.xml /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/main /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/main/java /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/main/java/org /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSFormatterFactory.java /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSTextFormatter.java /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWritableFormatter.java /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriterFactory.java /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/test /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/test/java /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/test/java/org /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs /incubator/flume/branches/flume-728/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java /incubator/flume/branches/flume-728/flume-ng-sinks/pom.xml /incubator/flume/branches/flume-728/pom.xml
        Hide
        Arvind Prabhakar added a comment -

        Patch committed. Thanks Prasad!

        Show
        Arvind Prabhakar added a comment - Patch committed. Thanks Prasad!
        Hide
        Prasad Mujumdar added a comment -

        Corrected group id in flume-ng-node/pom.xml

        Show
        Prasad Mujumdar added a comment - Corrected group id in flume-ng-node/pom.xml
        Hide
        Prasad Mujumdar added a comment -

        corrected the artifact Id in flume-ng-node/pom.xml and group id in flume-ng-sinks/flume-hdfs-sink/pom.xml

        Show
        Prasad Mujumdar added a comment - corrected the artifact Id in flume-ng-node/pom.xml and group id in flume-ng-sinks/flume-hdfs-sink/pom.xml
        Hide
        Prasad Mujumdar added a comment -

        Updated patch per review comments

        Show
        Prasad Mujumdar added a comment - Updated patch per review comments
        Hide
        Arvind Prabhakar added a comment -

        Some feedback on the patch:

        1. the new module is directly introduced at the top level as flume-ng-hdfs. Rather than doing that, I suggest you introduce a top-level module called flume-ng-sinks and make this a submodule of that. Also, it should be renamed to flume-hdfs-sink instead.

        2. the new module is not yet wired into the top level pom so it won't build from the top level. Need to modify the pom and introduce it in the ng-profile.

        3. the flume-ng-node depends upon this, so its pom needs to be modifed as well to call out an explicit dependency.

        One style suggest - since we are using $

        {project.version}

        for internal dependencies, it will be better to use that instead of explicit versions.

        Thanks,
        Arvind

        Show
        Arvind Prabhakar added a comment - Some feedback on the patch: 1. the new module is directly introduced at the top level as flume-ng-hdfs. Rather than doing that, I suggest you introduce a top-level module called flume-ng-sinks and make this a submodule of that. Also, it should be renamed to flume-hdfs-sink instead. 2. the new module is not yet wired into the top level pom so it won't build from the top level. Need to modify the pom and introduce it in the ng-profile. 3. the flume-ng-node depends upon this, so its pom needs to be modifed as well to call out an explicit dependency. One style suggest - since we are using $ {project.version} for internal dependencies, it will be better to use that instead of explicit versions. Thanks, Arvind
        Hide
        Prasad Mujumdar added a comment -

        Patch for Flume--774.
        Note that this is no top of Flume-777. Please commit this after 777 is applied.

        Show
        Prasad Mujumdar added a comment - Patch for Flume--774. Note that this is no top of Flume-777. Please commit this after 777 is applied.

          People

          • Assignee:
            Prasad Mujumdar
            Reporter:
            E. Sammer
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development