Pig
  1. Pig
  2. PIG-612

custom pig slicer not supported in local mode

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Won't Fix
    • Affects Version/s: 0.2.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Pig just ignores my custom slicer – i.e. never invokes it.

      I noticed that the test case for this is disabled:

      in TestCustomSlicer.java:
      @Test
      public void testUseRangeSlicer() throws ExecException, IOException {
      // FIXME : this should be tested in all modes
      if(execType == ExecType.LOCAL)
      return;

        Activity

        Hide
        Alan Gates added a comment -

        Two things are happening in 0.7 that will affect this. First, local mode is transitioning to Hadoop's LocalJobRunner. Second, the slicer interface is going away. So, we're solving this bug in spirit in that most things in local mode will now work the same as in map reduce mode.

        Show
        Alan Gates added a comment - Two things are happening in 0.7 that will affect this. First, local mode is transitioning to Hadoop's LocalJobRunner. Second, the slicer interface is going away. So, we're solving this bug in spirit in that most things in local mode will now work the same as in map reduce mode.
        Hide
        Christopher Olston added a comment -

        PARALLEL is different, because it doesn't involve a UDF.

        I just wrote a slicer UDF and want to step through it with the debugger before farming it out to hadoop.

        Show
        Christopher Olston added a comment - PARALLEL is different, because it doesn't involve a UDF. I just wrote a slicer UDF and want to step through it with the debugger before farming it out to hadoop.
        Hide
        Olga Natkovich added a comment -

        Sure but certain features like for instance PARALLEL keywords are ignored in local mode so certain things you can only test in parallel invironment and slicer seems to be like one of these features.

        Show
        Olga Natkovich added a comment - Sure but certain features like for instance PARALLEL keywords are ignored in local mode so certain things you can only test in parallel invironment and slicer seems to be like one of these features.
        Hide
        Christopher Olston added a comment -

        Well, for one thing, I want to test my slicer before I run it on hadoop. In fact, testing is one of the primary reasons for having local mode.

        Show
        Christopher Olston added a comment - Well, for one thing, I want to test my slicer before I run it on hadoop. In fact, testing is one of the primary reasons for having local mode.
        Hide
        Olga Natkovich added a comment -

        I don't think notion of slicer makes sense in local mode and ignoring it might be the right thing. In local mode we have one entity processing the input data so I am not sure why would we want to slice it.

        Show
        Olga Natkovich added a comment - I don't think notion of slicer makes sense in local mode and ignoring it might be the right thing. In local mode we have one entity processing the input data so I am not sure why would we want to slice it.

          People

          • Assignee:
            Unassigned
            Reporter:
            Christopher Olston
          • Votes:
            3 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development