Uploaded image for project: 'Crunch'
  1. Crunch
  2. CRUNCH-243

Support easily extensibility for custom reading of Avro Datum

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.7.0
    • Component/s: Core
    • Labels:
      None

      Description

      Through some discussions on the Avro mailing lists[1], there are use cases that need the ability to inject a custom SpecificDatumReader. Crunch does not make it easy to inject the custom reader as it is currently implemented causing consumers to copy three classes (e.g. AvroFileSource, AvroFileSourceTarget, AvroFileReaderFactory) to gain this functionality. With a few minor changes to AvroFileSource and AvroFileReaderFactory, consumers can inject their custom reader fairly easily.

      [1] - http://search-hadoop.com/m/98HA2DitiF/avro+enum+passivity&subj=Avro+enum+and+passivity+

      1. CRUNCH-243_v2.patch
        5 kB
        Micah Whitacre
      2. CRUNCH-243.patch
        2 kB
        Micah Whitacre

        Activity

        Hide
        mkwhitacre Micah Whitacre added a comment -

        Patch showing first pass at the changes.

        Show
        mkwhitacre Micah Whitacre added a comment - Patch showing first pass at the changes.
        Hide
        jwills Josh Wills added a comment -

        +1. Is there anything else you'd like it to do? Otherwise, I'd say commit as-is.

        Show
        jwills Josh Wills added a comment - +1. Is there anything else you'd like it to do? Otherwise, I'd say commit as-is.
        Hide
        mkwhitacre Micah Whitacre added a comment -

        Only other change I was toying with would be to add constructors to AvroFileSource and AvroFileSourceTarget that passes in a DatumReader, so consumers would not need subclassing.

        Show
        mkwhitacre Micah Whitacre added a comment - Only other change I was toying with would be to add constructors to AvroFileSource and AvroFileSourceTarget that passes in a DatumReader, so consumers would not need subclassing.
        Hide
        jwills Josh Wills added a comment -

        No objection to that.

        Show
        jwills Josh Wills added a comment - No objection to that.
        Hide
        mkwhitacre Micah Whitacre added a comment -

        Change with the constructors expanded.

        Show
        mkwhitacre Micah Whitacre added a comment - Change with the constructors expanded.
        Hide
        jwills Josh Wills added a comment -

        +1.

        Show
        jwills Josh Wills added a comment - +1.
        Hide
        mkwhitacre Micah Whitacre added a comment -

        Commit has been merged to master.

        Show
        mkwhitacre Micah Whitacre added a comment - Commit has been merged to master.

          People

          • Assignee:
            mkwhitacre Micah Whitacre
            Reporter:
            mkwhitacre Micah Whitacre
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development