Crunch
  1. Crunch
  2. CRUNCH-243

Support easily extensibility for custom reading of Avro Datum

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.7.0
    • Component/s: Core
    • Labels:
      None

      Description

      Through some discussions on the Avro mailing lists[1], there are use cases that need the ability to inject a custom SpecificDatumReader. Crunch does not make it easy to inject the custom reader as it is currently implemented causing consumers to copy three classes (e.g. AvroFileSource, AvroFileSourceTarget, AvroFileReaderFactory) to gain this functionality. With a few minor changes to AvroFileSource and AvroFileReaderFactory, consumers can inject their custom reader fairly easily.

      [1] - http://search-hadoop.com/m/98HA2DitiF/avro+enum+passivity&subj=Avro+enum+and+passivity+

      1. CRUNCH-243_v2.patch
        5 kB
        Micah Whitacre
      2. CRUNCH-243.patch
        2 kB
        Micah Whitacre

        Activity

        Hide
        Micah Whitacre added a comment -

        Patch showing first pass at the changes.

        Show
        Micah Whitacre added a comment - Patch showing first pass at the changes.
        Hide
        Josh Wills added a comment -

        +1. Is there anything else you'd like it to do? Otherwise, I'd say commit as-is.

        Show
        Josh Wills added a comment - +1. Is there anything else you'd like it to do? Otherwise, I'd say commit as-is.
        Hide
        Micah Whitacre added a comment -

        Only other change I was toying with would be to add constructors to AvroFileSource and AvroFileSourceTarget that passes in a DatumReader, so consumers would not need subclassing.

        Show
        Micah Whitacre added a comment - Only other change I was toying with would be to add constructors to AvroFileSource and AvroFileSourceTarget that passes in a DatumReader, so consumers would not need subclassing.
        Hide
        Josh Wills added a comment -

        No objection to that.

        Show
        Josh Wills added a comment - No objection to that.
        Hide
        Micah Whitacre added a comment -

        Change with the constructors expanded.

        Show
        Micah Whitacre added a comment - Change with the constructors expanded.
        Hide
        Josh Wills added a comment -

        +1.

        Show
        Josh Wills added a comment - +1.
        Hide
        Micah Whitacre added a comment -

        Commit has been merged to master.

        Show
        Micah Whitacre added a comment - Commit has been merged to master.

          People

          • Assignee:
            Micah Whitacre
            Reporter:
            Micah Whitacre
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development