Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v1.4.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      It would be great to support an avro container format deserializer in the spool directory source.

      1. FLUME-2048-3.patch
        25 kB
        Mike Percy
      2. FLUME-2048-2.patch
        21 kB
        Mike Percy
      3. FLUME-2048-1a.patch
        16 kB
        Mike Percy
      4. FLUME-2048.patch
        16 kB
        Mike Percy

        Issue Links

          Activity

          Mike Percy created issue -
          Mike Percy made changes -
          Field Original Value New Value
          Assignee Mike Percy [ mpercy ]
          Hide
          Mike Percy added a comment -

          Unit test is currently failing, but here is some partial progress on this feature.

          Show
          Mike Percy added a comment - Unit test is currently failing, but here is some partial progress on this feature.
          Mike Percy made changes -
          Attachment FLUME-2048.patch [ 12584268 ]
          Mike Percy made changes -
          Attachment FLUME-2048.patch [ 12584270 ]
          Mike Percy made changes -
          Attachment FLUME-2048.patch [ 12584268 ]
          Hide
          Mike Percy added a comment -

          Updated patch

          Show
          Mike Percy added a comment - Updated patch
          Mike Percy made changes -
          Attachment FLUME-2048-1a.patch [ 12584459 ]
          Mike Percy made changes -
          Attachment FLUME-2048-2.patch [ 12584472 ]
          Mike Percy made changes -
          Remote Link This issue links to "Review Board (Web Link)" [ 12302 ]
          Mike Percy made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Fix Version/s v1.4.0 [ 12323372 ]
          Hide
          Mike Percy added a comment -

          Unit test now passes.

          Show
          Mike Percy added a comment - Unit test now passes.
          Tom White made changes -
          Link This issue relates to FLUME-2091 [ FLUME-2091 ]
          Hide
          Mike Percy added a comment -

          Updated avro deserializer with literal mode to be compatible with literal mode in the avro serializer.

          Show
          Mike Percy added a comment - Updated avro deserializer with literal mode to be compatible with literal mode in the avro serializer.
          Mike Percy made changes -
          Attachment FLUME-2048-3.patch [ 12588525 ]
          Hide
          Tom White added a comment -

          This looks good to me. Is it worth pointing out the connection with the Avro event serializer in the docs? Also, if you use schema hashes there's (currently) no way to retrieve the schema.

          Show
          Tom White added a comment - This looks good to me. Is it worth pointing out the connection with the Avro event serializer in the docs? Also, if you use schema hashes there's (currently) no way to retrieve the schema.
          Hide
          Mike Percy added a comment -

          Hmm, I think we should do that in a follow-up doc JIRA because we haven't documented the new AvroEventSerializer at all. I also don't think we have provided an alias for it yet.

          Show
          Mike Percy added a comment - Hmm, I think we should do that in a follow-up doc JIRA because we haven't documented the new AvroEventSerializer at all. I also don't think we have provided an alias for it yet.
          Hide
          Hari Shreedharan added a comment -

          +1. This looks good. Can you please file the doc jira, Mike?

          Show
          Hari Shreedharan added a comment - +1. This looks good. Can you please file the doc jira, Mike?
          Hide
          Hari Shreedharan added a comment -

          Patch committed, rev: ad68350. Thanks Mike!

          Show
          Hari Shreedharan added a comment - Patch committed, rev: ad68350. Thanks Mike!
          Hari Shreedharan made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in flume-trunk #430 (See https://builds.apache.org/job/flume-trunk/430/)
          FLUME-2048. Add an Avro container file deserializer for Spool Directory Source. (Revision ad68350ffcc9a6cb47b7b555b9ebf6dacfa7c129)

          Result = SUCCESS
          hshreedharan : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=ad68350ffcc9a6cb47b7b555b9ebf6dacfa7c129
          Files :

          • flume-ng-core/src/test/java/org/apache/flume/serialization/TransientPositionTracker.java
          • flume-ng-core/src/test/java/org/apache/flume/serialization/TestAvroEventDeserializer.java
          • flume-ng-core/pom.xml
          • flume-ng-core/src/main/java/org/apache/flume/serialization/LengthMeasurable.java
          • pom.xml
          • flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java
          • flume-ng-core/src/main/java/org/apache/flume/serialization/RemoteMarkable.java
          • flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java
          • flume-ng-core/src/main/java/org/apache/flume/serialization/AvroEventDeserializer.java
          • flume-ng-doc/sphinx/FlumeUserGuide.rst
          Show
          Hudson added a comment - Integrated in flume-trunk #430 (See https://builds.apache.org/job/flume-trunk/430/ ) FLUME-2048 . Add an Avro container file deserializer for Spool Directory Source. (Revision ad68350ffcc9a6cb47b7b555b9ebf6dacfa7c129) Result = SUCCESS hshreedharan : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=ad68350ffcc9a6cb47b7b555b9ebf6dacfa7c129 Files : flume-ng-core/src/test/java/org/apache/flume/serialization/TransientPositionTracker.java flume-ng-core/src/test/java/org/apache/flume/serialization/TestAvroEventDeserializer.java flume-ng-core/pom.xml flume-ng-core/src/main/java/org/apache/flume/serialization/LengthMeasurable.java pom.xml flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java flume-ng-core/src/main/java/org/apache/flume/serialization/RemoteMarkable.java flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java flume-ng-core/src/main/java/org/apache/flume/serialization/AvroEventDeserializer.java flume-ng-doc/sphinx/FlumeUserGuide.rst
          Mike Percy made changes -
          Link This issue blocks FLUME-2097 [ FLUME-2097 ]

            People

            • Assignee:
              Mike Percy
              Reporter:
              Mike Percy
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development