Uploaded image for project: 'Crunch (Retired)'
  1. Crunch (Retired)
  2. CRUNCH-242

Input/output conversion needs to be controlled by the Source/Target interfaces

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.7.0
    • None
    • None

    Description

      I was working on adding support for Parquet to Crunch, and ran into the issue that Parquet always assumes that the value it returns is on the "value" side of the key-value pair of an InputFormat/OutputFormat. Crunch, for semi-sensible historical reasons, makes this position dependent on the PTypeFamily (Avro PTypes write to the key, Writable PTypes write to the value). Since the Parquet InputFormat/OutputFormat treat the two types the same way, we need a way for the Source and Target implementations to override the default configuration of the PTypes and choose the right side for the given format.

      Attachments

        1. CRUNCH-242.patch
          13 kB
          Josh Wills

        Activity

          jwills Josh Wills added a comment -

          Patch that updates the Source and Target interfaces to enable them to override the Converter used by the PType as appropriate.

          jwills Josh Wills added a comment - Patch that updates the Source and Target interfaces to enable them to override the Converter used by the PType as appropriate.
          mkwhitacre Micah Whitacre added a comment -

          +1 to the patch. The changes look fine. Might make a few people grumpy with the non-passive change of adding new method to interface but nothing that can't be worked around.

          mkwhitacre Micah Whitacre added a comment - +1 to the patch. The changes look fine. Might make a few people grumpy with the non-passive change of adding new method to interface but nothing that can't be worked around.
          jwills Josh Wills added a comment -

          I'm one of those people, but I figured Parquet support was worth it.

          jwills Josh Wills added a comment - I'm one of those people, but I figured Parquet support was worth it.
          jwills Josh Wills added a comment -

          Committed to master.

          jwills Josh Wills added a comment - Committed to master.

          People

            Unassigned Unassigned
            jwills Josh Wills
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: