Flume
  1. Flume
  2. FLUME-2048

Avro container file deserializer

        Details

        • Type: New Feature New Feature
        • Status: Resolved
        • Priority: Major Major
        • Resolution: Fixed
        • Affects Version/s: None
        • Fix Version/s: v1.4.0
        • Component/s: Sinks+Sources
        • Labels:
          None

          Description

          It would be great to support an avro container format deserializer in the spool directory source.

          1. Text File
            FLUME-2048.patch
            16 kB
            Mike Percy
          2. Text File
            FLUME-2048-1a.patch
            16 kB
            Mike Percy
          3. Text File
            FLUME-2048-2.patch
            21 kB
            Mike Percy
          4. Text File
            FLUME-2048-3.patch
            25 kB
            Mike Percy

            Issue Links

            blocks

            Documentation - Documentation or Website FLUME-2097 Add docs for Avro Serializer and describe how to make the Avro Deserializer work with the Avro Serializer

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Open
            relates to

            New Feature - A new feature of the product, which has yet to be developed. FLUME-2091 Use Avro schema hashes in Log4jAppender and the HDFS Sink

            • Major - Major loss of function.
            • Open
            links to

            Web Link Review Board

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              Mike Percy added a comment -

              Unit test is currently failing, but here is some partial progress on this feature.

              Show
              Mike Percy added a comment - Unit test is currently failing, but here is some partial progress on this feature.
              Hide
              Permalink
              Mike Percy added a comment -

              Updated patch

              Show
              Mike Percy added a comment - Updated patch
              Hide
              Permalink
              Mike Percy added a comment -

              Unit test now passes.

              Show
              Mike Percy added a comment - Unit test now passes.
              Hide
              Permalink
              Mike Percy added a comment -

              Updated avro deserializer with literal mode to be compatible with literal mode in the avro serializer.

              Show
              Mike Percy added a comment - Updated avro deserializer with literal mode to be compatible with literal mode in the avro serializer.
              Hide
              Permalink
              Tom White added a comment -

              This looks good to me. Is it worth pointing out the connection with the Avro event serializer in the docs? Also, if you use schema hashes there's (currently) no way to retrieve the schema.

              Show
              Tom White added a comment - This looks good to me. Is it worth pointing out the connection with the Avro event serializer in the docs? Also, if you use schema hashes there's (currently) no way to retrieve the schema.
              Hide
              Permalink
              Mike Percy added a comment -

              Hmm, I think we should do that in a follow-up doc JIRA because we haven't documented the new AvroEventSerializer at all. I also don't think we have provided an alias for it yet.

              Show
              Mike Percy added a comment - Hmm, I think we should do that in a follow-up doc JIRA because we haven't documented the new AvroEventSerializer at all. I also don't think we have provided an alias for it yet.
              Hide
              Permalink
              Hari Shreedharan added a comment -

              +1. This looks good. Can you please file the doc jira, Mike?

              Show
              Hari Shreedharan added a comment - +1. This looks good. Can you please file the doc jira, Mike?
              Hide
              Permalink
              Hari Shreedharan added a comment -

              Patch committed, rev: ad68350. Thanks Mike!

              Show
              Hari Shreedharan added a comment - Patch committed, rev: ad68350. Thanks Mike!
              Hide
              Permalink
              Hudson added a comment -

              Integrated in flume-trunk #430 (See https://builds.apache.org/job/flume-trunk/430/)
              FLUME-2048. Add an Avro container file deserializer for Spool Directory Source. (Revision ad68350ffcc9a6cb47b7b555b9ebf6dacfa7c129)

              Result = SUCCESS
              hshreedharan : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=ad68350ffcc9a6cb47b7b555b9ebf6dacfa7c129
              Files :

              • flume-ng-core/src/test/java/org/apache/flume/serialization/TransientPositionTracker.java
              • flume-ng-core/src/test/java/org/apache/flume/serialization/TestAvroEventDeserializer.java
              • flume-ng-core/pom.xml
              • flume-ng-core/src/main/java/org/apache/flume/serialization/LengthMeasurable.java
              • pom.xml
              • flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java
              • flume-ng-core/src/main/java/org/apache/flume/serialization/RemoteMarkable.java
              • flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java
              • flume-ng-core/src/main/java/org/apache/flume/serialization/AvroEventDeserializer.java
              • flume-ng-doc/sphinx/FlumeUserGuide.rst
              Show
              Hudson added a comment - Integrated in flume-trunk #430 (See https://builds.apache.org/job/flume-trunk/430/ ) FLUME-2048 . Add an Avro container file deserializer for Spool Directory Source. (Revision ad68350ffcc9a6cb47b7b555b9ebf6dacfa7c129) Result = SUCCESS hshreedharan : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=ad68350ffcc9a6cb47b7b555b9ebf6dacfa7c129 Files : flume-ng-core/src/test/java/org/apache/flume/serialization/TransientPositionTracker.java flume-ng-core/src/test/java/org/apache/flume/serialization/TestAvroEventDeserializer.java flume-ng-core/pom.xml flume-ng-core/src/main/java/org/apache/flume/serialization/LengthMeasurable.java pom.xml flume-ng-core/src/main/java/org/apache/flume/serialization/ResettableFileInputStream.java flume-ng-core/src/main/java/org/apache/flume/serialization/RemoteMarkable.java flume-ng-core/src/main/java/org/apache/flume/serialization/EventDeserializerType.java flume-ng-core/src/main/java/org/apache/flume/serialization/AvroEventDeserializer.java flume-ng-doc/sphinx/FlumeUserGuide.rst

                People

                • Assignee:
                  Mike Percy
                  Reporter:
                  Mike Percy
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  5 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development