Flume
  1. Flume
  2. FLUME-1632

Persist progress on each file in file spooling client/source

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v1.4.0
    • Component/s: None
    • Labels:
      None

      Description

      Add the ability to persist progress on a partially consumed file in spooling client/source

      1. FLUME-1632-9b.patch
        150 kB
        Mike Percy
      2. FLUME-1632-7.patch
        107 kB
        Mike Percy
      3. FLUME-1632-11b.patch
        156 kB
        Mike Percy
      4. FLUME-1632-10a.patch
        155 kB
        Mike Percy

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in flume-trunk #344 (See https://builds.apache.org/job/flume-trunk/344/)
          FLUME-1632: Persist progress on each file in file spooling client/source (Revision fad7251dbc13e74bb070691e7a5b643f2402b901)

          Result = SUCCESS
          brock : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=fad7251dbc13e74bb070691e7a5b643f2402b901
          Files :

          • flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java
          • flume-ng-core/src/main/avro/TransferStateFileMeta.avsc
          • flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java
          • flume-ng-core/src/test/resources/TestResettableFileInputStream_1.truncated.avro
          • flume-ng-core/src/main/java/org/apache/flume/client/avro/SimpleTextLineEventReader.java
          • flume-ng-core/src/test/java/org/apache/flume/serialization/ResettableTestStringInputStream.java
          • flume-ng-core/src/main/java/org/apache/flume/serialization/DurablePositionTracker.java
          • flume-ng-core/src/test/java/org/apache/flume/serialization/TestLineDeserializer.java
          • flume-ng-core/src/test/java/org/apache/flume/client/avro/TestBufferedLineReader.java
          • flume-ng-core/src/test/java/org/apache/flume/serialization/TestReliableSpoolingFileEventReader.java
          • flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableInputStream.java
          • flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableEventReader.java
          • flume-ng-core/src/test/java/org/apache/flume/serialization/TestResettableFileInputStream.java
          • flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java
          • flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java
          • flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java
          • flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java
          • flume-ng-core/src/main/java/org/apache/flume/serialization/LineDeserializer.java
          • flume-ng-core/src/main/java/org/apache/flume/tools/PlatformDetect.java
          • flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java
          • flume-ng-core/src/test/resources/TestResettableFileInputStream_1.avro
          • flume-ng-core/src/main/java/org/apache/flume/serialization/Resettable.java
          • flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java
          • flume-ng-core/src/main/java/org/apache/flume/serialization/PositionTracker.java
          • flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java
          • flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializer.java
          • flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java
          • flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java
          • flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerDe.java
          • pom.xml
          • flume-ng-core/src/test/java/org/apache/flume/serialization/TestDurablePositionTracker.java
          • flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerFactory.java
          • flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java
          • flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java
          • flume-ng-core/src/main/java/org/apache/flume/client/avro/EventReader.java
          • flume-ng-core/pom.xml
          • flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java
          Show
          Hudson added a comment - Integrated in flume-trunk #344 (See https://builds.apache.org/job/flume-trunk/344/ ) FLUME-1632 : Persist progress on each file in file spooling client/source (Revision fad7251dbc13e74bb070691e7a5b643f2402b901) Result = SUCCESS brock : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=fad7251dbc13e74bb070691e7a5b643f2402b901 Files : flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java flume-ng-core/src/main/avro/TransferStateFileMeta.avsc flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java flume-ng-core/src/test/resources/TestResettableFileInputStream_1.truncated.avro flume-ng-core/src/main/java/org/apache/flume/client/avro/SimpleTextLineEventReader.java flume-ng-core/src/test/java/org/apache/flume/serialization/ResettableTestStringInputStream.java flume-ng-core/src/main/java/org/apache/flume/serialization/DurablePositionTracker.java flume-ng-core/src/test/java/org/apache/flume/serialization/TestLineDeserializer.java flume-ng-core/src/test/java/org/apache/flume/client/avro/TestBufferedLineReader.java flume-ng-core/src/test/java/org/apache/flume/serialization/TestReliableSpoolingFileEventReader.java flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableInputStream.java flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableEventReader.java flume-ng-core/src/test/java/org/apache/flume/serialization/TestResettableFileInputStream.java flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java flume-ng-core/src/main/java/org/apache/flume/serialization/LineDeserializer.java flume-ng-core/src/main/java/org/apache/flume/tools/PlatformDetect.java flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java flume-ng-core/src/test/resources/TestResettableFileInputStream_1.avro flume-ng-core/src/main/java/org/apache/flume/serialization/Resettable.java flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java flume-ng-core/src/main/java/org/apache/flume/serialization/PositionTracker.java flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializer.java flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerDe.java pom.xml flume-ng-core/src/test/java/org/apache/flume/serialization/TestDurablePositionTracker.java flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerFactory.java flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java flume-ng-core/src/main/java/org/apache/flume/client/avro/EventReader.java flume-ng-core/pom.xml flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java
          Hide
          Brock Noland added a comment -

          Thank you for the contribution Mike! Committed to trunk and 1.4!

          Show
          Brock Noland added a comment - Thank you for the contribution Mike! Committed to trunk and 1.4!
          Hide
          Mike Percy added a comment -

          Patch 10 + a couple whitespace changes

          Show
          Mike Percy added a comment - Patch 10 + a couple whitespace changes
          Hide
          Mike Percy added a comment -

          Updated patch.

          Show
          Mike Percy added a comment - Updated patch.
          Hide
          Mike Percy added a comment -

          Early patch. Missing a bunch of unit tests and not all unit tests are passing yet.

          Defines EventDeserializer interface and uses it from the spooling source. Progress is persisted as bytes are read from the underlying file.

          Show
          Mike Percy added a comment - Early patch. Missing a bunch of unit tests and not all unit tests are passing yet. Defines EventDeserializer interface and uses it from the spooling source. Progress is persisted as bytes are read from the underlying file.

            People

            • Assignee:
              Mike Percy
              Reporter:
              Mike Percy
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development