Uploaded image for project: 'Crunch'
  1. Crunch
  2. CRUNCH-448

Smart-ish case class serialization for Scrunch

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.4, 0.11.0
    • Component/s: Scrunch
    • Labels:
      None

      Description

      I took advantage of some of the new reflection features in Scala 2.10 to make automatic serialization of case classes much cooler, viz.,

      1) We can automatically map any case class to a TupleN and properly infer PTypes for Scala types (options, lists, etc.), just as we do for regular PCollections.
      2) For Avro serialization, we will also capture the names of the case class' fields and map them to the names of the fields in the Avro record we generate, which makes debugging intermediate outputs much easier.

        Attachments

        1. CRUNCH-448.patch
          19 kB
          Josh Wills
        2. CRUNCH-448b.patch
          21 kB
          Josh Wills

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jwills Josh Wills
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: