Uploaded image for project: 'Crunch (Retired)'
  1. Crunch (Retired)
  2. CRUNCH-302

PType OutputMapFn not initialised when using MemPipeline

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 0.9.0
    • Fix Version/s: 0.9.0, 0.8.2
    • Component/s: Core, IO
    • Labels:
      None
    • Environment:
      Crunch synced to 2aa692e5299ee9d775218d4754ae73f3d58beed1

      java version "1.6.0_65"
      Java(TM) SE Runtime Environment (build 1.6.0_65-b14-462-11M4609)
      Java HotSpot(TM) 64-Bit Server VM (build 20.65-b04-462, mixed mode)

      Description

      It seems that the OutputMapFns for PTypes aren't initialised when running in local memory. For most types this doesn't matter, but it gets tricky when using a Tuple as a key or value type in a PTable or a PCollection of Tuples; the following exception gets thrown:

      Caused by: java.lang.NullPointerException
      at org.apache.crunch.types.writable.Writables$TupleTWMapFn.map(Writables.java:376)
      at org.apache.crunch.types.writable.Writables$TupleTWMapFn.map(Writables.java:337)
      at org.apache.crunch.impl.mem.MemPipeline.writeSequenceFileFromPTable(MemPipeline.java:270)
      at org.apache.crunch.impl.mem.MemPipeline.write(MemPipeline.java:196)
      at org.apache.crunch.impl.mem.MemPipeline.write(MemPipeline.java:177)

      I'm currently working around this issue by explicitly calling the initialize() method on MapFn, like this:
      table.getPTableType().getValueType().getOutputMapFn().initialize();

        Attachments

        1. CRUNCH-302.patch
          4 kB
          Josh Wills

          Activity

            People

            • Assignee:
              jwills Josh Wills
              Reporter:
              cvangysel Christophe Van Gysel
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: