Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2350

Consume Order tests need to space out file creation

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: v1.4.0
    • Fix Version/s: v1.5.0
    • Component/s: None
    • Labels:
      None

      Description

      Since the files end up getting created almost at the same time, their timestamps end up not being different even though they are created one after another. We need to add sleeps or make sure their last modified is different by adding 1 or 2 seconds to each consecutive file.

      1. FLUME-2350-0.patch
        8 kB
        Muhammad Ehsan ul Haque

        Activity

        Hide
        hshreedharan Hari Shreedharan added a comment -

        There a couple of things that need to be fixed:

        • Leave time gaps so last modified do not match.
        • Re-introduce lexicographical comparison if timestamps are same (this is my bad, removing the comparator in the final patch removed this as well).

        I will submit a patch soon to fix this.

        Show
        hshreedharan Hari Shreedharan added a comment - There a couple of things that need to be fixed: Leave time gaps so last modified do not match. Re-introduce lexicographical comparison if timestamps are same (this is my bad, removing the comparator in the final patch removed this as well). I will submit a patch soon to fix this.
        Hide
        ehsan Muhammad Ehsan ul Haque added a comment -

        I have added patch for the fix of lexicographical comparison. Apologize for making that a bit obscure in the previous patch which made you do the change yourself.

        I have also added time gaps in two of the tests "testConsumeFileOldest" and "testConsumeFileYoungest", just to make it more explicit.
        Also added two more test to explicitly test the lexicographical order in case of YOUNGEST and OLDEST. (If they look redundant then you may remove as well)

        For the tests "testLargeNumberOfFilesOLDEST" and "testLargeNumberOfFilesYOUNGEST" I don't think that we need to add any time gaps. The test will take very long and also I think it is a good case for testing lexicographical ordering as well.

        Show
        ehsan Muhammad Ehsan ul Haque added a comment - I have added patch for the fix of lexicographical comparison. Apologize for making that a bit obscure in the previous patch which made you do the change yourself. I have also added time gaps in two of the tests "testConsumeFileOldest" and "testConsumeFileYoungest", just to make it more explicit. Also added two more test to explicitly test the lexicographical order in case of YOUNGEST and OLDEST. (If they look redundant then you may remove as well) For the tests "testLargeNumberOfFilesOLDEST" and "testLargeNumberOfFilesYOUNGEST" I don't think that we need to add any time gaps. The test will take very long and also I think it is a good case for testing lexicographical ordering as well.
        Hide
        hshreedharan Hari Shreedharan added a comment -

        This looks good. Long#compare is not available in Java 6, so I am replacing that with selected.lastModified() - candidate.lastModified(), which is basically the same thing.

        Running tests now.

        Show
        hshreedharan Hari Shreedharan added a comment - This looks good. Long#compare is not available in Java 6, so I am replacing that with selected.lastModified() - candidate.lastModified(), which is basically the same thing. Running tests now.
        Hide
        hshreedharan Hari Shreedharan added a comment -

        +1.

        Show
        hshreedharan Hari Shreedharan added a comment - +1.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 62b383a00c3f678b0f504dc71bf36091ddd4067a in flume's branch refs/heads/trunk from Hari Shreedharan
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=62b383a ]

        FLUME-2350. Consume Order tests need to space out file creation.

        (Muhammad Ehsan ul Haque via Hari Shreedharan)

        Show
        jira-bot ASF subversion and git services added a comment - Commit 62b383a00c3f678b0f504dc71bf36091ddd4067a in flume's branch refs/heads/trunk from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=62b383a ] FLUME-2350 . Consume Order tests need to space out file creation. (Muhammad Ehsan ul Haque via Hari Shreedharan)
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 964ce67a839df164004219bda47f397e923f6ad8 in flume's branch refs/heads/flume-1.5 from Hari Shreedharan
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=964ce67 ]

        FLUME-2350. Consume Order tests need to space out file creation.

        (Muhammad Ehsan ul Haque via Hari Shreedharan)

        Show
        jira-bot ASF subversion and git services added a comment - Commit 964ce67a839df164004219bda47f397e923f6ad8 in flume's branch refs/heads/flume-1.5 from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=964ce67 ] FLUME-2350 . Consume Order tests need to space out file creation. (Muhammad Ehsan ul Haque via Hari Shreedharan)
        Hide
        hshreedharan Hari Shreedharan added a comment -

        Committed. Thanks Muhammad!

        Show
        hshreedharan Hari Shreedharan added a comment - Committed. Thanks Muhammad!
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in flume-trunk #631 (See https://builds.apache.org/job/flume-trunk/631/)
        FLUME-2350. Consume Order tests need to space out file creation. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo/?p=flume.git&a=commit&h=62b383a00c3f678b0f504dc71bf36091ddd4067a)

        • flume-ng-core/src/test/java/org/apache/flume/client/avro/TestReliableSpoolingFileEventReader.java
        • flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in flume-trunk #631 (See https://builds.apache.org/job/flume-trunk/631/ ) FLUME-2350 . Consume Order tests need to space out file creation. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo/?p=flume.git&a=commit&h=62b383a00c3f678b0f504dc71bf36091ddd4067a ) flume-ng-core/src/test/java/org/apache/flume/client/avro/TestReliableSpoolingFileEventReader.java flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java

          People

          • Assignee:
            ehsan Muhammad Ehsan ul Haque
            Reporter:
            hshreedharan Hari Shreedharan
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development