Uploaded image for project: 'Crunch'
  1. Crunch
  2. CRUNCH-433

Add support for reading specific/reflect data from an Avro MR file

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.4, 0.11.0
    • Component/s: None
    • Labels:
      None

      Description

      An Avro Key/Value file written via raw MapReduce contains records that follow the schema generated by the org.apache.avro.hadoop.io.AvroKeyValue class.

      If these files contain specific or reflection-based records, there is currently no easy way to read them in as specific or reflection records. Using the basic public Crunch APIs, they can only be read as generic records (that also contain generic records).

      A method should be added to the Avros class which allows specifying specific PTypes to be used for reading the underlying data types within a raw MR output file.

      Link to related discussion that inspired this ticket on the user list: http://s.apache.org/es

        Attachments

        1. CRUNCH-433.1.patch
          33 kB
          Gabriel Reid
        2. CRUNCH-433.patch
          33 kB
          Gabriel Reid

          Activity

            People

            • Assignee:
              gabriel.reid Gabriel Reid
              Reporter:
              gabriel.reid Gabriel Reid
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: