Flume
  1. Flume
  2. FLUME-1632

Persist progress on each file in file spooling client/source

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v1.4.0
    • Component/s: None
    • Labels:
      None

      Description

      Add the ability to persist progress on a partially consumed file in spooling client/source

      1. FLUME-1632-11b.patch
        156 kB
        Mike Percy
      2. FLUME-1632-10a.patch
        155 kB
        Mike Percy
      3. FLUME-1632-9b.patch
        150 kB
        Mike Percy
      4. FLUME-1632-7.patch
        107 kB
        Mike Percy

        Issue Links

          Activity

          Mike Percy created issue -
          Mike Percy made changes -
          Field Original Value New Value
          Link This issue requires FLUME-1425 [ FLUME-1425 ]
          Mike Percy made changes -
          Assignee Mike Percy [ mpercy ]
          Hide
          Mike Percy added a comment -

          Early patch. Missing a bunch of unit tests and not all unit tests are passing yet.

          Defines EventDeserializer interface and uses it from the spooling source. Progress is persisted as bytes are read from the underlying file.

          Show
          Mike Percy added a comment - Early patch. Missing a bunch of unit tests and not all unit tests are passing yet. Defines EventDeserializer interface and uses it from the spooling source. Progress is persisted as bytes are read from the underlying file.
          Mike Percy made changes -
          Attachment FLUME-1632-7.patch [ 12560941 ]
          Hide
          Mike Percy added a comment -

          Updated patch.

          Show
          Mike Percy added a comment - Updated patch.
          Mike Percy made changes -
          Attachment FLUME-1632-9b.patch [ 12561260 ]
          Mike Percy made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Fix Version/s v1.4.0 [ 12323372 ]
          Hide
          Mike Percy added a comment -

          Patch 10 + a couple whitespace changes

          Show
          Mike Percy added a comment - Patch 10 + a couple whitespace changes
          Mike Percy made changes -
          Attachment FLUME-1632-10a.patch [ 12561600 ]
          Mike Percy made changes -
          Attachment FLUME-1632-11b.patch [ 12561687 ]
          Mike Percy made changes -
          Link This issue blocks FLUME-1800 [ FLUME-1800 ]
          Mike Percy made changes -
          Link This issue blocks FLUME-1801 [ FLUME-1801 ]
          Brock Noland made changes -
          Remote Link This issue links to "Review Board (Web Link)" [ 11921 ]
          Hide
          Brock Noland added a comment -

          Thank you for the contribution Mike! Committed to trunk and 1.4!

          Show
          Brock Noland added a comment - Thank you for the contribution Mike! Committed to trunk and 1.4!
          Brock Noland made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Mike Percy made changes -
          Link This issue is duplicated by FLUME-1633 [ FLUME-1633 ]
          Hide
          Hudson added a comment -

          Integrated in flume-trunk #344 (See https://builds.apache.org/job/flume-trunk/344/)
          FLUME-1632: Persist progress on each file in file spooling client/source (Revision fad7251dbc13e74bb070691e7a5b643f2402b901)

          Result = SUCCESS
          brock : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=fad7251dbc13e74bb070691e7a5b643f2402b901
          Files :

          • flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java
          • flume-ng-core/src/main/avro/TransferStateFileMeta.avsc
          • flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java
          • flume-ng-core/src/test/resources/TestResettableFileInputStream_1.truncated.avro
          • flume-ng-core/src/main/java/org/apache/flume/client/avro/SimpleTextLineEventReader.java
          • flume-ng-core/src/test/java/org/apache/flume/serialization/ResettableTestStringInputStream.java
          • flume-ng-core/src/main/java/org/apache/flume/serialization/DurablePositionTracker.java
          • flume-ng-core/src/test/java/org/apache/flume/serialization/TestLineDeserializer.java
          • flume-ng-core/src/test/java/org/apache/flume/client/avro/TestBufferedLineReader.java
          • flume-ng-core/src/test/java/org/apache/flume/serialization/TestReliableSpoolingFileEventReader.java
          • flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableInputStream.java
          • flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableEventReader.java
          • flume-ng-core/src/test/java/org/apache/flume/serialization/TestResettableFileInputStream.java
          • flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java
          • flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java
          • flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java
          • flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java
          • flume-ng-core/src/main/java/org/apache/flume/serialization/LineDeserializer.java
          • flume-ng-core/src/main/java/org/apache/flume/tools/PlatformDetect.java
          • flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java
          • flume-ng-core/src/test/resources/TestResettableFileInputStream_1.avro
          • flume-ng-core/src/main/java/org/apache/flume/serialization/Resettable.java
          • flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java
          • flume-ng-core/src/main/java/org/apache/flume/serialization/PositionTracker.java
          • flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java
          • flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializer.java
          • flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java
          • flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java
          • flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerDe.java
          • pom.xml
          • flume-ng-core/src/test/java/org/apache/flume/serialization/TestDurablePositionTracker.java
          • flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerFactory.java
          • flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java
          • flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java
          • flume-ng-core/src/main/java/org/apache/flume/client/avro/EventReader.java
          • flume-ng-core/pom.xml
          • flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java
          Show
          Hudson added a comment - Integrated in flume-trunk #344 (See https://builds.apache.org/job/flume-trunk/344/ ) FLUME-1632 : Persist progress on each file in file spooling client/source (Revision fad7251dbc13e74bb070691e7a5b643f2402b901) Result = SUCCESS brock : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=fad7251dbc13e74bb070691e7a5b643f2402b901 Files : flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java flume-ng-core/src/main/avro/TransferStateFileMeta.avsc flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java flume-ng-core/src/test/resources/TestResettableFileInputStream_1.truncated.avro flume-ng-core/src/main/java/org/apache/flume/client/avro/SimpleTextLineEventReader.java flume-ng-core/src/test/java/org/apache/flume/serialization/ResettableTestStringInputStream.java flume-ng-core/src/main/java/org/apache/flume/serialization/DurablePositionTracker.java flume-ng-core/src/test/java/org/apache/flume/serialization/TestLineDeserializer.java flume-ng-core/src/test/java/org/apache/flume/client/avro/TestBufferedLineReader.java flume-ng-core/src/test/java/org/apache/flume/serialization/TestReliableSpoolingFileEventReader.java flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableInputStream.java flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableEventReader.java flume-ng-core/src/test/java/org/apache/flume/serialization/TestResettableFileInputStream.java flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java flume-ng-core/src/main/java/org/apache/flume/serialization/LineDeserializer.java flume-ng-core/src/main/java/org/apache/flume/tools/PlatformDetect.java flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java flume-ng-core/src/test/resources/TestResettableFileInputStream_1.avro flume-ng-core/src/main/java/org/apache/flume/serialization/Resettable.java flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java flume-ng-core/src/main/java/org/apache/flume/serialization/PositionTracker.java flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializer.java flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerDe.java pom.xml flume-ng-core/src/test/java/org/apache/flume/serialization/TestDurablePositionTracker.java flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerFactory.java flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java flume-ng-core/src/main/java/org/apache/flume/client/avro/EventReader.java flume-ng-core/pom.xml flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java
          Brock Noland made changes -
          Link This issue is related to FLUME-1825 [ FLUME-1825 ]
          Gavin made changes -
          Link This issue blocks FLUME-1800 [ FLUME-1800 ]
          Gavin made changes -
          Link This issue is depended upon by FLUME-1800 [ FLUME-1800 ]
          Gavin made changes -
          Link This issue blocks FLUME-1801 [ FLUME-1801 ]
          Gavin made changes -
          Link This issue is depended upon by FLUME-1801 [ FLUME-1801 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          67d 9h 14m 1 Mike Percy 17/Dec/12 08:59
          Patch Available Patch Available Resolved Resolved
          2d 9h 14m 1 Brock Noland 19/Dec/12 18:14

            People

            • Assignee:
              Mike Percy
              Reporter:
              Mike Percy
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development