Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v1.4.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      It would be great to support an avro container format deserializer in the spool directory source.

      1. FLUME-2048-3.patch
        25 kB
        Mike Percy
      2. FLUME-2048-2.patch
        21 kB
        Mike Percy
      3. FLUME-2048-1a.patch
        16 kB
        Mike Percy
      4. FLUME-2048.patch
        16 kB
        Mike Percy

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in flume-trunk #430 (See https://builds.apache.org/job/flume-trunk/430/)
          FLUME-2048. Add an Avro container file deserializer for Spool Directory Source. (Revision ad68350ffcc9a6cb47b7b555b9ebf6dacfa7c129)

          Result = SUCCESS
          hshreedharan : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=ad68350ffcc9a6cb47b7b555b9ebf6dacfa7c129
          Files :

          • flume-ng-core/src/test/java/org/apache/flume/serialization/TransientPositionTracker.java
          • flume-ng-core/src/test/java/org/apache/flume/serialization/TestAvroEventDeserializer.java
          • flume-ng-core/pom.xml
          • flume-ng-core/src/main/java/org/apache/flume/serialization/LengthMeasurable.java
          • pom.xml
          • flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java
          • flume-ng-core/src/main/java/org/apache/flume/serialization/RemoteMarkable.java
          • flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java
          • flume-ng-core/src/main/java/org/apache/flume/serialization/AvroEventDeserializer.java
          • flume-ng-doc/sphinx/FlumeUserGuide.rst
          Show
          Hudson added a comment - Integrated in flume-trunk #430 (See https://builds.apache.org/job/flume-trunk/430/ ) FLUME-2048 . Add an Avro container file deserializer for Spool Directory Source. (Revision ad68350ffcc9a6cb47b7b555b9ebf6dacfa7c129) Result = SUCCESS hshreedharan : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=ad68350ffcc9a6cb47b7b555b9ebf6dacfa7c129 Files : flume-ng-core/src/test/java/org/apache/flume/serialization/TransientPositionTracker.java flume-ng-core/src/test/java/org/apache/flume/serialization/TestAvroEventDeserializer.java flume-ng-core/pom.xml flume-ng-core/src/main/java/org/apache/flume/serialization/LengthMeasurable.java pom.xml flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java flume-ng-core/src/main/java/org/apache/flume/serialization/RemoteMarkable.java flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java flume-ng-core/src/main/java/org/apache/flume/serialization/AvroEventDeserializer.java flume-ng-doc/sphinx/FlumeUserGuide.rst
          Hide
          Hari Shreedharan added a comment -

          Patch committed, rev: ad68350. Thanks Mike!

          Show
          Hari Shreedharan added a comment - Patch committed, rev: ad68350. Thanks Mike!
          Hide
          Hari Shreedharan added a comment -

          +1. This looks good. Can you please file the doc jira, Mike?

          Show
          Hari Shreedharan added a comment - +1. This looks good. Can you please file the doc jira, Mike?
          Hide
          Mike Percy added a comment -

          Hmm, I think we should do that in a follow-up doc JIRA because we haven't documented the new AvroEventSerializer at all. I also don't think we have provided an alias for it yet.

          Show
          Mike Percy added a comment - Hmm, I think we should do that in a follow-up doc JIRA because we haven't documented the new AvroEventSerializer at all. I also don't think we have provided an alias for it yet.
          Hide
          Tom White added a comment -

          This looks good to me. Is it worth pointing out the connection with the Avro event serializer in the docs? Also, if you use schema hashes there's (currently) no way to retrieve the schema.

          Show
          Tom White added a comment - This looks good to me. Is it worth pointing out the connection with the Avro event serializer in the docs? Also, if you use schema hashes there's (currently) no way to retrieve the schema.
          Hide
          Mike Percy added a comment -

          Updated avro deserializer with literal mode to be compatible with literal mode in the avro serializer.

          Show
          Mike Percy added a comment - Updated avro deserializer with literal mode to be compatible with literal mode in the avro serializer.
          Hide
          Mike Percy added a comment -

          Unit test now passes.

          Show
          Mike Percy added a comment - Unit test now passes.
          Hide
          Mike Percy added a comment -

          Updated patch

          Show
          Mike Percy added a comment - Updated patch
          Hide
          Mike Percy added a comment -

          Unit test is currently failing, but here is some partial progress on this feature.

          Show
          Mike Percy added a comment - Unit test is currently failing, but here is some partial progress on this feature.

            People

            • Assignee:
              Mike Percy
              Reporter:
              Mike Percy
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development