Pig
  1. Pig
  2. PIG-226

Problem with streaming optimization

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.1.0
    • Component/s: None
    • Labels:
      None

      Description

      Current optimization code assumes that every storeage function implements both LoadFunc and StoreFunc interfaces and causes an exception if it does not.

      2008-05-01 13:31:23,662 [main] ERROR org.apache.pig.tools.grunt.Grunt - java.lang.RuntimeException: could not instantiate 'org.apache.pig.test.udf.storefunc.DumpLoader' with arguments '[]'
      at org.apache.pig.impl.PigContext.instantiateFunc(PigContext.java:515)
      at org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:521)
      at org.apache.pig.impl.logicalLayer.optimizer.streaming.StoreOptimizer.visitStore(StoreOptimizer.java:103)
      at org.apache.pig.impl.logicalLayer.LOStore.visit(LOStore.java:121)
      at org.apache.pig.impl.logicalLayer.optimizer.streaming.StoreOptimizer.optimize(StoreOptimizer.java:145)
      at org.apache.pig.PigServer.optimizeAndRunQuery(PigServer.java:411)
      at org.apache.pig.PigServer.registerQuery(PigServer.java:297)
      at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:450)
      at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:233)
      at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:62)
      at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:60)
      at org.apache.pig.Main.main(Main.java:296)

      The code that needs to change is in impl/logicalLayer/optimizer/streaming/LoadOptimizer.java and impl/logicalLayer/optimizer/streaming/StoreOptimizer.java.

      The logic that needs to be added is if the load/store interface is not implemented then disable optimization.

      1. PIG-226_0_20080501.patch
        7 kB
        Arun C Murthy
      2. PIG-226_1_20080502.patch
        5 kB
        Arun C Murthy

        Activity

        Hide
        Arun C Murthy added a comment -

        The 'right' patch ...

        Show
        Arun C Murthy added a comment - The 'right' patch ...
        Hide
        Arun C Murthy added a comment -

        Fixed the check to use ReversibleLoadStoreFunc as suggested by Pi...

        Show
        Arun C Murthy added a comment - Fixed the check to use ReversibleLoadStoreFunc as suggested by Pi...
        Hide
        Olga Natkovich added a comment -

        Thanks, Pi. I did not realize that the change already ent in. Arun, could you make the change please.

        Show
        Olga Natkovich added a comment - Thanks, Pi. I did not realize that the change already ent in. Arun, could you make the change please.
        Hide
        Pi Song added a comment -

        Actually we have an interface called "ReversibleLoadStoreFunc" that enforces the load/store operations to be reversible.

        Only checking "if (streamStorer instanceof LoadFunc)" is not enough because the load/store operations might not be symmetric.

        The ReversibleLoadStoreFunc interface guarantees the property that:-

        LoadFunc (StoreFunc(x)) = x
        
        Show
        Pi Song added a comment - Actually we have an interface called "ReversibleLoadStoreFunc" that enforces the load/store operations to be reversible. Only checking "if (streamStorer instanceof LoadFunc)" is not enough because the load/store operations might not be symmetric. The ReversibleLoadStoreFunc interface guarantees the property that:- LoadFunc (StoreFunc(x)) = x
        Hide
        Olga Natkovich added a comment -

        patch committed. thanks Arun

        Show
        Olga Natkovich added a comment - patch committed. thanks Arun
        Hide
        Arun C Murthy added a comment -

        I've fixed the

        {Load|Store}Optimizer to check if the StreamingCommand's {in|out}Specs implement {Load|Store}

        Func before casting them. The patch also contains a test case for the same. I have also checked that the test fails without the fix to the

        {Load|Store}

        Optimizer.

        Show
        Arun C Murthy added a comment - I've fixed the {Load|Store}Optimizer to check if the StreamingCommand's {in|out}Specs implement {Load|Store} Func before casting them. The patch also contains a test case for the same. I have also checked that the test fails without the fix to the {Load|Store} Optimizer.

          People

          • Assignee:
            Arun C Murthy
            Reporter:
            Olga Natkovich
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development