Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2939

Upgrade recursive SpoolDir to use Java7 features

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.7.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      FLUME-1899 was committed without the DirectoryStream API

      This ticket is for "upgrading" the patch provided to use Java 7 features as the Java version for Flume has been recently bumped.

        Issue Links

          Activity

          Hide
          bessbd Bessenyei Balázs Donát added a comment -

          Patch submitted for review, please see https://reviews.apache.org/r/49506/

          Show
          bessbd Bessenyei Balázs Donát added a comment - Patch submitted for review, please see https://reviews.apache.org/r/49506/
          Hide
          mpercy Mike Percy added a comment -

          +1

          Thank you Bessenyei Balázs Donát for the patch and Attila Simon and Denes Arvay for reviewing it. I am about to commit this.

          Show
          mpercy Mike Percy added a comment - +1 Thank you Bessenyei Balázs Donát for the patch and Attila Simon and Denes Arvay for reviewing it. I am about to commit this.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 7013708baddc8ed7d861797d1fd8280a94b6025c in flume's branch refs/heads/trunk from Mike Percy
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=7013708 ]

          FLUME-2939. Update recursive SpoolDir source to use Java 7 APIs

          (Bessenyei Balázs Donát via Mike Percy)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 7013708baddc8ed7d861797d1fd8280a94b6025c in flume's branch refs/heads/trunk from Mike Percy [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=7013708 ] FLUME-2939 . Update recursive SpoolDir source to use Java 7 APIs (Bessenyei Balázs Donát via Mike Percy)
          Hide
          mpercy Mike Percy added a comment -

          This has been pushed to trunk.

          Show
          mpercy Mike Percy added a comment - This has been pushed to trunk.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Flume-trunk-hbase-1 #173 (See https://builds.apache.org/job/Flume-trunk-hbase-1/173/)
          FLUME-2939. Update recursive SpoolDir source to use Java 7 APIs (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=7013708baddc8ed7d861797d1fd8280a94b6025c)

          • flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Flume-trunk-hbase-1 #173 (See https://builds.apache.org/job/Flume-trunk-hbase-1/173/ ) FLUME-2939 . Update recursive SpoolDir source to use Java 7 APIs (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=7013708baddc8ed7d861797d1fd8280a94b6025c ) flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit ba64b12670f7b2c9ce079f8898f4601689a51def in flume's branch refs/heads/trunk from Lior Zeno
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=ba64b12 ]

          FLUME-2950. ReliableSpoolingFileEventReader.rollCurrentFile is broken

          FLUME-2939 introduced a Java 7 API (nio.file) to
          ReliableSpoolingFileEventReader. However, the committed patch breaks
          rollCurrentFile() on Windows machines. File rename might be not atomic,
          in such cases, and under the assumption that the rolled file and the
          current file store the same data, Flume only logs a warning and does not
          assume that the user broke the contract that the spooling directory
          source demands.

          In order to determine whether the files store the same data, we used the
          com.google.common.io.Files.equal() method. However, FLUME-2939 uses
          java.nio.file.Files.isSameFile() which has different semantics.
          Consequently, this relaxation does not hold anymore. In addition,
          TestSpoolingFileLineReader.testDestinationExistsAndSameFileWindows
          fails. This patch brings back the old check using Files.equal().

          (Lior Zeno via Mike Percy)

          Show
          jira-bot ASF subversion and git services added a comment - Commit ba64b12670f7b2c9ce079f8898f4601689a51def in flume's branch refs/heads/trunk from Lior Zeno [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=ba64b12 ] FLUME-2950 . ReliableSpoolingFileEventReader.rollCurrentFile is broken FLUME-2939 introduced a Java 7 API (nio.file) to ReliableSpoolingFileEventReader. However, the committed patch breaks rollCurrentFile() on Windows machines. File rename might be not atomic, in such cases, and under the assumption that the rolled file and the current file store the same data, Flume only logs a warning and does not assume that the user broke the contract that the spooling directory source demands. In order to determine whether the files store the same data, we used the com.google.common.io.Files.equal() method. However, FLUME-2939 uses java.nio.file.Files.isSameFile() which has different semantics. Consequently, this relaxation does not hold anymore. In addition, TestSpoolingFileLineReader.testDestinationExistsAndSameFileWindows fails. This patch brings back the old check using Files.equal(). (Lior Zeno via Mike Percy)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit ba64b12670f7b2c9ce079f8898f4601689a51def in flume's branch refs/heads/trunk from Lior Zeno
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=ba64b12 ]

          FLUME-2950. ReliableSpoolingFileEventReader.rollCurrentFile is broken

          FLUME-2939 introduced a Java 7 API (nio.file) to
          ReliableSpoolingFileEventReader. However, the committed patch breaks
          rollCurrentFile() on Windows machines. File rename might be not atomic,
          in such cases, and under the assumption that the rolled file and the
          current file store the same data, Flume only logs a warning and does not
          assume that the user broke the contract that the spooling directory
          source demands.

          In order to determine whether the files store the same data, we used the
          com.google.common.io.Files.equal() method. However, FLUME-2939 uses
          java.nio.file.Files.isSameFile() which has different semantics.
          Consequently, this relaxation does not hold anymore. In addition,
          TestSpoolingFileLineReader.testDestinationExistsAndSameFileWindows
          fails. This patch brings back the old check using Files.equal().

          (Lior Zeno via Mike Percy)

          Show
          jira-bot ASF subversion and git services added a comment - Commit ba64b12670f7b2c9ce079f8898f4601689a51def in flume's branch refs/heads/trunk from Lior Zeno [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=ba64b12 ] FLUME-2950 . ReliableSpoolingFileEventReader.rollCurrentFile is broken FLUME-2939 introduced a Java 7 API (nio.file) to ReliableSpoolingFileEventReader. However, the committed patch breaks rollCurrentFile() on Windows machines. File rename might be not atomic, in such cases, and under the assumption that the rolled file and the current file store the same data, Flume only logs a warning and does not assume that the user broke the contract that the spooling directory source demands. In order to determine whether the files store the same data, we used the com.google.common.io.Files.equal() method. However, FLUME-2939 uses java.nio.file.Files.isSameFile() which has different semantics. Consequently, this relaxation does not hold anymore. In addition, TestSpoolingFileLineReader.testDestinationExistsAndSameFileWindows fails. This patch brings back the old check using Files.equal(). (Lior Zeno via Mike Percy)

            People

            • Assignee:
              bessbd Bessenyei Balázs Donát
              Reporter:
              bessbd Bessenyei Balázs Donát
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development