Flume
  1. Flume
  2. FLUME-1632

Persist progress on each file in file spooling client/source

        Details

        • Type: New Feature New Feature
        • Status: Resolved
        • Priority: Major Major
        • Resolution: Fixed
        • Affects Version/s: None
        • Fix Version/s: v1.4.0
        • Component/s: None
        • Labels:
          None

          Description

          Add the ability to persist progress on a partially consumed file in spooling client/source

          1. Text File
            FLUME-1632-10a.patch
            155 kB
            Mike Percy
          2. Text File
            FLUME-1632-11b.patch
            156 kB
            Mike Percy
          3. Text File
            FLUME-1632-7.patch
            107 kB
            Mike Percy
          4. Text File
            FLUME-1632-9b.patch
            150 kB
            Mike Percy

            Issue Links

            is depended upon by

            Improvement - An improvement or enhancement to an existing feature or task. FLUME-1801 Configurable batch size for spooling client

            • Major - Major loss of function.
            • Open

            Improvement - An improvement or enhancement to an existing feature or task. FLUME-1800 Docs for spooling source durability changes

            • Major - Major loss of function.
            • Resolved
            is duplicated by

            New Feature - A new feature of the product, which has yet to be developed. FLUME-1633 Support pluggable deserializers in spooling client / source

            • Major - Major loss of function.
            • Resolved
            is related to

            Bug - A problem which impairs or prevents the functions of the product. FLUME-1825 Avro-cient command with -F always drops 16% lines

            • Major - Major loss of function.
            • Open
            requires

            New Feature - A new feature of the product, which has yet to be developed. FLUME-1425 Create a SpoolDirectory Source and Client

            • Major - Major loss of function.
            • Resolved
            links to

            Web Link Review Board

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              Mike Percy added a comment -

              Early patch. Missing a bunch of unit tests and not all unit tests are passing yet.

              Defines EventDeserializer interface and uses it from the spooling source. Progress is persisted as bytes are read from the underlying file.

              Show
              Mike Percy added a comment - Early patch. Missing a bunch of unit tests and not all unit tests are passing yet. Defines EventDeserializer interface and uses it from the spooling source. Progress is persisted as bytes are read from the underlying file.
              Hide
              Permalink
              Mike Percy added a comment -

              Updated patch.

              Show
              Mike Percy added a comment - Updated patch.
              Hide
              Permalink
              Mike Percy added a comment -

              Patch 10 + a couple whitespace changes

              Show
              Mike Percy added a comment - Patch 10 + a couple whitespace changes
              Hide
              Permalink
              Brock Noland added a comment -

              Thank you for the contribution Mike! Committed to trunk and 1.4!

              Show
              Brock Noland added a comment - Thank you for the contribution Mike! Committed to trunk and 1.4!
              Hide
              Permalink
              Hudson added a comment -

              Integrated in flume-trunk #344 (See https://builds.apache.org/job/flume-trunk/344/)
              FLUME-1632: Persist progress on each file in file spooling client/source (Revision fad7251dbc13e74bb070691e7a5b643f2402b901)

              Result = SUCCESS
              brock : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=fad7251dbc13e74bb070691e7a5b643f2402b901
              Files :

              • flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java
              • flume-ng-core/src/main/avro/TransferStateFileMeta.avsc
              • flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java
              • flume-ng-core/src/test/resources/TestResettableFileInputStream_1.truncated.avro
              • flume-ng-core/src/main/java/org/apache/flume/client/avro/SimpleTextLineEventReader.java
              • flume-ng-core/src/test/java/org/apache/flume/serialization/ResettableTestStringInputStream.java
              • flume-ng-core/src/main/java/org/apache/flume/serialization/DurablePositionTracker.java
              • flume-ng-core/src/test/java/org/apache/flume/serialization/TestLineDeserializer.java
              • flume-ng-core/src/test/java/org/apache/flume/client/avro/TestBufferedLineReader.java
              • flume-ng-core/src/test/java/org/apache/flume/serialization/TestReliableSpoolingFileEventReader.java
              • flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableInputStream.java
              • flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableEventReader.java
              • flume-ng-core/src/test/java/org/apache/flume/serialization/TestResettableFileInputStream.java
              • flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java
              • flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java
              • flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java
              • flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java
              • flume-ng-core/src/main/java/org/apache/flume/serialization/LineDeserializer.java
              • flume-ng-core/src/main/java/org/apache/flume/tools/PlatformDetect.java
              • flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java
              • flume-ng-core/src/test/resources/TestResettableFileInputStream_1.avro
              • flume-ng-core/src/main/java/org/apache/flume/serialization/Resettable.java
              • flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java
              • flume-ng-core/src/main/java/org/apache/flume/serialization/PositionTracker.java
              • flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java
              • flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializer.java
              • flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java
              • flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java
              • flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerDe.java
              • pom.xml
              • flume-ng-core/src/test/java/org/apache/flume/serialization/TestDurablePositionTracker.java
              • flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerFactory.java
              • flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java
              • flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java
              • flume-ng-core/src/main/java/org/apache/flume/client/avro/EventReader.java
              • flume-ng-core/pom.xml
              • flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java
              Show
              Hudson added a comment - Integrated in flume-trunk #344 (See https://builds.apache.org/job/flume-trunk/344/ ) FLUME-1632 : Persist progress on each file in file spooling client/source (Revision fad7251dbc13e74bb070691e7a5b643f2402b901) Result = SUCCESS brock : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=fad7251dbc13e74bb070691e7a5b643f2402b901 Files : flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializer.java flume-ng-core/src/main/avro/TransferStateFileMeta.avsc flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java flume-ng-core/src/test/resources/TestResettableFileInputStream_1.truncated.avro flume-ng-core/src/main/java/org/apache/flume/client/avro/SimpleTextLineEventReader.java flume-ng-core/src/test/java/org/apache/flume/serialization/ResettableTestStringInputStream.java flume-ng-core/src/main/java/org/apache/flume/serialization/DurablePositionTracker.java flume-ng-core/src/test/java/org/apache/flume/serialization/TestLineDeserializer.java flume-ng-core/src/test/java/org/apache/flume/client/avro/TestBufferedLineReader.java flume-ng-core/src/test/java/org/apache/flume/serialization/TestReliableSpoolingFileEventReader.java flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableInputStream.java flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableEventReader.java flume-ng-core/src/test/java/org/apache/flume/serialization/TestResettableFileInputStream.java flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerType.java flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java flume-ng-core/src/main/java/org/apache/flume/serialization/LineDeserializer.java flume-ng-core/src/main/java/org/apache/flume/tools/PlatformDetect.java flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java flume-ng-core/src/test/resources/TestResettableFileInputStream_1.avro flume-ng-core/src/main/java/org/apache/flume/serialization/Resettable.java flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java flume-ng-core/src/main/java/org/apache/flume/serialization/PositionTracker.java flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializer.java flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerializerFactory.java flume-ng-core/src/main/java/org/apache/flume/serialization/EventSerDe.java pom.xml flume-ng-core/src/test/java/org/apache/flume/serialization/TestDurablePositionTracker.java flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerFactory.java flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java flume-ng-core/src/main/java/org/apache/flume/client/avro/EventReader.java flume-ng-core/pom.xml flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java

                People

                • Assignee:
                  Mike Percy
                  Reporter:
                  Mike Percy
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  3 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development