Flume
  1. Flume
  2. FLUME-1102

HDFSEventSink rollInterval is broken

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v1.2.0
    • Component/s: None
    • Labels:
      None

      Description

      rollInterval doesn't work.

        Activity

        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4638/
        -----------------------------------------------------------

        Review request for Flume.

        Summary
        -------

        Fix for rollInterval. Also added unit tests for the BucketWriter.

        This addresses bug FLUME-1102.
        https://issues.apache.org/jira/browse/FLUME-1102

        Diffs


        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 3d00e64
        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java PRE-CREATION
        flume-ng-sinks/flume-hdfs-sink/src/test/resources/log4j.properties PRE-CREATION
        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java PRE-CREATION

        Diff: https://reviews.apache.org/r/4638/diff

        Testing
        -------

        Unit tests pass.

        Thanks,

        Mike

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4638/ ----------------------------------------------------------- Review request for Flume. Summary ------- Fix for rollInterval. Also added unit tests for the BucketWriter. This addresses bug FLUME-1102 . https://issues.apache.org/jira/browse/FLUME-1102 Diffs flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 3d00e64 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/test/resources/log4j.properties PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java PRE-CREATION Diff: https://reviews.apache.org/r/4638/diff Testing ------- Unit tests pass. Thanks, Mike
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4638/#review6695
        -----------------------------------------------------------

        Ship it!

        +1
        My understanding is the close() followed by open() takes care of the counters getting reset, when the rotate happens and now talks to a new file. If that is the case, it looks good.

        • Hari

        On 2012-04-04 18:51:44, Mike Percy wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/4638/

        -----------------------------------------------------------

        (Updated 2012-04-04 18:51:44)

        Review request for Flume.

        Summary

        -------

        Fix for rollInterval. Also added unit tests for the BucketWriter.

        This addresses bug FLUME-1102.

        https://issues.apache.org/jira/browse/FLUME-1102

        Diffs

        -----

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 3d00e64

        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java PRE-CREATION

        flume-ng-sinks/flume-hdfs-sink/src/test/resources/log4j.properties PRE-CREATION

        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java PRE-CREATION

        Diff: https://reviews.apache.org/r/4638/diff

        Testing

        -------

        Unit tests pass.

        Thanks,

        Mike

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4638/#review6695 ----------------------------------------------------------- Ship it! +1 My understanding is the close() followed by open() takes care of the counters getting reset, when the rotate happens and now talks to a new file. If that is the case, it looks good. Hari On 2012-04-04 18:51:44, Mike Percy wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4638/ ----------------------------------------------------------- (Updated 2012-04-04 18:51:44) Review request for Flume. Summary ------- Fix for rollInterval. Also added unit tests for the BucketWriter. This addresses bug FLUME-1102 . https://issues.apache.org/jira/browse/FLUME-1102 Diffs ----- flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 3d00e64 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/test/resources/log4j.properties PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java PRE-CREATION Diff: https://reviews.apache.org/r/4638/diff Testing ------- Unit tests pass. Thanks, Mike
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4638/#review6699
        -----------------------------------------------------------

        Ship it!

        lgtm

        • Prasad

        On 2012-04-04 18:51:44, Mike Percy wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/4638/

        -----------------------------------------------------------

        (Updated 2012-04-04 18:51:44)

        Review request for Flume.

        Summary

        -------

        Fix for rollInterval. Also added unit tests for the BucketWriter.

        This addresses bug FLUME-1102.

        https://issues.apache.org/jira/browse/FLUME-1102

        Diffs

        -----

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 3d00e64

        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java PRE-CREATION

        flume-ng-sinks/flume-hdfs-sink/src/test/resources/log4j.properties PRE-CREATION

        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java PRE-CREATION

        Diff: https://reviews.apache.org/r/4638/diff

        Testing

        -------

        Unit tests pass.

        Thanks,

        Mike

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4638/#review6699 ----------------------------------------------------------- Ship it! lgtm Prasad On 2012-04-04 18:51:44, Mike Percy wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4638/ ----------------------------------------------------------- (Updated 2012-04-04 18:51:44) Review request for Flume. Summary ------- Fix for rollInterval. Also added unit tests for the BucketWriter. This addresses bug FLUME-1102 . https://issues.apache.org/jira/browse/FLUME-1102 Diffs ----- flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java 3d00e64 flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/test/resources/log4j.properties PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java PRE-CREATION Diff: https://reviews.apache.org/r/4638/diff Testing ------- Unit tests pass. Thanks, Mike
        Hide
        Prasad Mujumdar added a comment -

        Patch committed to trunk
        Thanks Mike !

        Show
        Prasad Mujumdar added a comment - Patch committed to trunk Thanks Mike !
        Hide
        Hudson added a comment -

        Integrated in flume-trunk #159 (See https://builds.apache.org/job/flume-trunk/159/)
        FLUME-1102: HDFSEventSink rollInterval is broken
        (Mike Percy via Prasad Mujumdar) (Revision 1309628)
        FLUME-1102: HDFSEventSink rollInterval is broken
        (Mike Percy via Prasad Mujumdar)
        . (Revision 1309627)

        Result = SUCCESS
        prasadm : http://svn.apache.org/viewvc/?view=rev&rev=1309628
        Files :

        • /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java
        • /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java
        • /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/resources
        • /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/resources/log4j.properties

        prasadm : http://svn.apache.org/viewvc/?view=rev&rev=1309627
        Files :

        • /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
        Show
        Hudson added a comment - Integrated in flume-trunk #159 (See https://builds.apache.org/job/flume-trunk/159/ ) FLUME-1102 : HDFSEventSink rollInterval is broken (Mike Percy via Prasad Mujumdar) (Revision 1309628) FLUME-1102 : HDFSEventSink rollInterval is broken (Mike Percy via Prasad Mujumdar) . (Revision 1309627) Result = SUCCESS prasadm : http://svn.apache.org/viewvc/?view=rev&rev=1309628 Files : /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/resources /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/resources/log4j.properties prasadm : http://svn.apache.org/viewvc/?view=rev&rev=1309627 Files : /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java

          People

          • Assignee:
            Mike Percy
            Reporter:
            Mike Percy
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development