Flume
  1. Flume
  2. FLUME-2048

Avro container file deserializer

        Details

        • Type: New Feature New Feature
        • Status: Resolved
        • Priority: Major Major
        • Resolution: Fixed
        • Affects Version/s: None
        • Fix Version/s: v1.4.0
        • Component/s: Sinks+Sources
        • Labels:
          None

          Description

          It would be great to support an avro container format deserializer in the spool directory source.

          1. Text File
            FLUME-2048-3.patch
            25 kB
            Mike Percy
          2. Text File
            FLUME-2048-2.patch
            21 kB
            Mike Percy
          3. Text File
            FLUME-2048-1a.patch
            16 kB
            Mike Percy
          4. Text File
            FLUME-2048.patch
            16 kB
            Mike Percy

            Issue Links

            blocks

            Documentation - Documentation or Website FLUME-2097 Add docs for Avro Serializer and describe how to make the Avro Deserializer work with the Avro Serializer

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Open
            relates to

            New Feature - A new feature of the product, which has yet to be developed. FLUME-2091 Use Avro schema hashes in Log4jAppender and the HDFS Sink

            • Major - Major loss of function.
            • Open
            links to

            Web Link Review Board

              Activity

              Ascending order - Click to sort in descending order
              Mike Percy created issue -
              Mike Percy made changes -
              Field Original Value New Value
              Assignee Mike Percy [ mpercy ]
              Hide
              Permalink
              Mike Percy added a comment -

              Unit test is currently failing, but here is some partial progress on this feature.

              Show
              Mike Percy added a comment - Unit test is currently failing, but here is some partial progress on this feature.
              Mike Percy made changes -
              Attachment FLUME-2048.patch [ 12584268 ]
              Mike Percy made changes -
              Attachment FLUME-2048.patch [ 12584270 ]
              Mike Percy made changes -
              Attachment FLUME-2048.patch [ 12584268 ]
              Hide
              Permalink
              Mike Percy added a comment -

              Updated patch

              Show
              Mike Percy added a comment - Updated patch
              Mike Percy made changes -
              Attachment FLUME-2048-1a.patch [ 12584459 ]
              Mike Percy made changes -
              Attachment FLUME-2048-2.patch [ 12584472 ]
              Mike Percy made changes -
              Remote Link This issue links to "Review Board (Web Link)" [ 12302 ]
              Mike Percy made changes -
              Status Open [ 1 ] Patch Available [ 10002 ]
              Fix Version/s v1.4.0 [ 12323372 ]
              Hide
              Permalink
              Mike Percy added a comment -

              Unit test now passes.

              Show
              Mike Percy added a comment - Unit test now passes.
              Tom White made changes -
              Link This issue relates to FLUME-2091 [ FLUME-2091 ]
              Hide
              Permalink
              Mike Percy added a comment -

              Updated avro deserializer with literal mode to be compatible with literal mode in the avro serializer.

              Show
              Mike Percy added a comment - Updated avro deserializer with literal mode to be compatible with literal mode in the avro serializer.
              Mike Percy made changes -
              Attachment FLUME-2048-3.patch [ 12588525 ]
              Hide
              Permalink
              Tom White added a comment -

              This looks good to me. Is it worth pointing out the connection with the Avro event serializer in the docs? Also, if you use schema hashes there's (currently) no way to retrieve the schema.

              Show
              Tom White added a comment - This looks good to me. Is it worth pointing out the connection with the Avro event serializer in the docs? Also, if you use schema hashes there's (currently) no way to retrieve the schema.
              Hide
              Permalink
              Mike Percy added a comment -

              Hmm, I think we should do that in a follow-up doc JIRA because we haven't documented the new AvroEventSerializer at all. I also don't think we have provided an alias for it yet.

              Show
              Mike Percy added a comment - Hmm, I think we should do that in a follow-up doc JIRA because we haven't documented the new AvroEventSerializer at all. I also don't think we have provided an alias for it yet.
              Hide
              Permalink
              Hari Shreedharan added a comment -

              +1. This looks good. Can you please file the doc jira, Mike?

              Show
              Hari Shreedharan added a comment - +1. This looks good. Can you please file the doc jira, Mike?
              Hide
              Permalink
              Hari Shreedharan added a comment -

              Patch committed, rev: ad68350. Thanks Mike!

              Show
              Hari Shreedharan added a comment - Patch committed, rev: ad68350. Thanks Mike!
              Hari Shreedharan made changes -
              Status Patch Available [ 10002 ] Resolved [ 5 ]
              Resolution Fixed [ 1 ]
              Hide
              Permalink
              Hudson added a comment -

              Integrated in flume-trunk #430 (See https://builds.apache.org/job/flume-trunk/430/)
              FLUME-2048. Add an Avro container file deserializer for Spool Directory Source. (Revision ad68350ffcc9a6cb47b7b555b9ebf6dacfa7c129)

              Result = SUCCESS
              hshreedharan : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=ad68350ffcc9a6cb47b7b555b9ebf6dacfa7c129
              Files :

              • flume-ng-core/src/test/java/org/apache/flume/serialization/TransientPositionTracker.java
              • flume-ng-core/src/test/java/org/apache/flume/serialization/TestAvroEventDeserializer.java
              • flume-ng-core/pom.xml
              • flume-ng-core/src/main/java/org/apache/flume/serialization/LengthMeasurable.java
              • pom.xml
              • flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java
              • flume-ng-core/src/main/java/org/apache/flume/serialization/RemoteMarkable.java
              • flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java
              • flume-ng-core/src/main/java/org/apache/flume/serialization/AvroEventDeserializer.java
              • flume-ng-doc/sphinx/FlumeUserGuide.rst
              Show
              Hudson added a comment - Integrated in flume-trunk #430 (See https://builds.apache.org/job/flume-trunk/430/ ) FLUME-2048 . Add an Avro container file deserializer for Spool Directory Source. (Revision ad68350ffcc9a6cb47b7b555b9ebf6dacfa7c129) Result = SUCCESS hshreedharan : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=ad68350ffcc9a6cb47b7b555b9ebf6dacfa7c129 Files : flume-ng-core/src/test/java/org/apache/flume/serialization/TransientPositionTracker.java flume-ng-core/src/test/java/org/apache/flume/serialization/TestAvroEventDeserializer.java flume-ng-core/pom.xml flume-ng-core/src/main/java/org/apache/flume/serialization/LengthMeasurable.java pom.xml flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java flume-ng-core/src/main/java/org/apache/flume/serialization/RemoteMarkable.java flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java flume-ng-core/src/main/java/org/apache/flume/serialization/AvroEventDeserializer.java flume-ng-doc/sphinx/FlumeUserGuide.rst
              Mike Percy made changes -
              Link This issue blocks FLUME-2097 [ FLUME-2097 ]

                People

                • Assignee:
                  Mike Percy
                  Reporter:
                  Mike Percy
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  5 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development