Uploaded image for project: 'Crunch'
  1. Crunch
  2. CRUNCH-314

Shuffle and Bundle configuration AvroMode settings clobber one another

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.9.0
    • Fix Version/s: 0.10.0, 0.8.3
    • Component/s: Core
    • Labels:
      None

      Description

      AvroMode uses the same configuration property for both shuffle and bundle configuration. If the bundle's AvroMode for a map/reduce round doesn't match the shuffle's AvroMode, then the bundle wins. In some cases, this prevents the shuffle from being able to write data, e.g., because GENERIC is used instead of REFLECT.

      A work-around if to set the strict mode to delegate reader and writer construction to the more permissive one:

      AvroMode.GENERIC.override(AvroMode.REFLECT);
      

        Attachments

          Activity

            People

            • Assignee:
              jwills Josh Wills
              Reporter:
              rdblue Ryan Blue
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: