Uploaded image for project: 'Crunch'
  1. Crunch
  2. CRUNCH-667

Documentation should mention that cogrouping requires a common type family

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Core
    • Labels:
      None

      Description

      Attempting to cogroup an Avro-based table with a Writable-based table produces an exception like:

      java.lang.ClassCastException: org.apache.crunch.types.writable.WritableType cannot be cast
         to org.apache.crunch.types.avro.AvroType
      at org.apache.crunch.types.avro.Avros.collections(Avros.java:409)
      at org.apache.crunch.types.avro.AvroTypeFamily.collections(AvroTypeFamily.java:106)
      at org.apache.crunch.lib.Cogroup.cogroup(Cogroup.java:63)
      at org.apache.crunch.lib.Cogroup.cogroup(Cogroup.java:46)
      at org.apache.crunch.impl.dist.collect.PTableBase.cogroup(PTableBase.java:176)

      This technical limitation makes sense, and the exception message is relatively self-explanatory (at least for anyone who is familiar with Crunch concepts), so I don't think there's really a bug to be fixed. However both the documentation in the code javadoc (Cogroup class, Set#difference, and PTable#cogroup) and the user guide section about Cogroups (https://crunch.apache.org/user-guide.html#cogroups) could be updated to clarify that all tables involved in a cogroup operation must have the same type family.

        Attachments

          Activity

            People

            • Assignee:
              jwills Josh Wills
              Reporter:
              noslowerdna Andrew Olson
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: