Uploaded image for project: 'Crunch'
  1. Crunch
  2. CRUNCH-511

Scrunch product type support should use derived() instead of derivedImmutable()

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.0
    • Component/s: Scrunch
    • Labels:
      None

      Description

      Ran into a subtle bug related to Scrunch case class support: right now, I was using derivedImmutable to generate the mapping from an Avro record in to a Scala case class, thinking (foolishly, as it turned out) that case classes were always immutable. Although that's true, the arguments that get passed to the case class constructor (the fields of the Avro record) are not necessarily immutable (e.g., an Avro ByteBuffer gets reused from key to key in a reduce), and those values can cause join key errors.

      I wrote a test case to illustrate the bug and switched the case class PType generator to use derived instead of derivedImmutable.

        Attachments

        1. CRUNCH-511.patch
          5 kB
          Josh Wills
        2. CRUNCH-511b.patch
          4 kB
          Josh Wills

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jwills Josh Wills
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: