Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.0
    • Component/s: None
    • Labels:
    • Release Note:
      This feature introduces a new keyword - OTHERWISE, and that is not backward compatible - it can break scripts that use it as an alias.

      Description

      "split" statement is better to have a default destination, eg:

      SPLIT A INTO X IF f1<7, Y IF f2==5, Z IF (f3<6 OR f3>6), OTHER otherwise; -- OTHERS has all tuples with f1>=7 && f2!=5 && f3==6
      

      This is a candidate project for Google summer of code 2011. More information about the program can be found at http://wiki.apache.org/pig/GSoc2011

      1. PIG-1904.2.patch
        11 kB
        Gianmarco De Francisci Morales
      2. PIG-1904.1.patch
        7 kB
        Gianmarco De Francisci Morales

        Issue Links

          Activity

          Hide
          Gianmarco De Francisci Morales added a comment -

          From my understanding, this can be done by inserting another LOSplitOutput in the LOSplit plan such that the condition for the output is an AND of all the negated conditions specified on the command line.
          Would this be a reasonable way to proceed?

          Show
          Gianmarco De Francisci Morales added a comment - From my understanding, this can be done by inserting another LOSplitOutput in the LOSplit plan such that the condition for the output is an AND of all the negated conditions specified on the command line. Would this be a reasonable way to proceed?
          Hide
          Daniel Dai added a comment -

          Yes, this is a neat way for it. And this is a big opportunity for LogicalExpressionSimplifier simplifying the "otherwise" expression.

          Show
          Daniel Dai added a comment - Yes, this is a neat way for it. And this is a big opportunity for LogicalExpressionSimplifier simplifying the "otherwise" expression.
          Hide
          Gianmarco De Francisci Morales added a comment -

          Here is an updated proposal for the GSoC that includes this feature as well.

          http://socghop.appspot.com/gsoc/proposal/review/google/gsoc2011/azaroth/1

          Show
          Gianmarco De Francisci Morales added a comment - Here is an updated proposal for the GSoC that includes this feature as well. http://socghop.appspot.com/gsoc/proposal/review/google/gsoc2011/azaroth/1
          Hide
          Gianmarco De Francisci Morales added a comment -

          PIG-1904.1.patch contains the first working implementation of the feature.

          The grammar now recognizes statements like:
          SPLIT a INTO b IF x1 < 0, c OTHERWISE;
          but also like:
          SPLIT a INTO b IF x1 < 0;
          This is a side-effect of making the otherwise branch optional and is a change from past behavior.
          It shouldn't be a problem as the Split maps to a Filter in any case.

          Implemented by copying of the other LOSplitOutput plans, and building a negated disjunction (OR) of the expressions.

          Added unit test for Split-Otherwise

          TODO:
          Disable the feature if the expression contains a @NonDeterministic UDF.
          I plan to do it by spawning a visitor on the expression.
          The visitor will throw an error and explain the reason in the error message.
          Is this a reasonable approach?

          Show
          Gianmarco De Francisci Morales added a comment - PIG-1904 .1.patch contains the first working implementation of the feature. The grammar now recognizes statements like: SPLIT a INTO b IF x1 < 0, c OTHERWISE; but also like: SPLIT a INTO b IF x1 < 0; This is a side-effect of making the otherwise branch optional and is a change from past behavior. It shouldn't be a problem as the Split maps to a Filter in any case. Implemented by copying of the other LOSplitOutput plans, and building a negated disjunction (OR) of the expressions. Added unit test for Split-Otherwise TODO: Disable the feature if the expression contains a @NonDeterministic UDF. I plan to do it by spawning a visitor on the expression. The visitor will throw an error and explain the reason in the error message. Is this a reasonable approach?
          Hide
          Dmitriy V. Ryaboy added a comment -

          Nice catch about @NonDeterministic. Seems like it doesn't work due to the implementation details, the issue isn't fundamental. I'm cool with the partial solution for now, but please file a jira to fix this later.

          Show
          Dmitriy V. Ryaboy added a comment - Nice catch about @NonDeterministic. Seems like it doesn't work due to the implementation details, the issue isn't fundamental. I'm cool with the partial solution for now, but please file a jira to fix this later.
          Hide
          Gianmarco De Francisci Morales added a comment -

          Created PIG-2169 for this.
          Anyway given the benefit/cost ratio I wouldn't try to fix it.
          A Nondeterministic UDF in a Split is probably better expressed as a Sample.
          Anyway I think this simple workaround should work:

          a = LOAD 'a.txt' AS (f1,f2,f3);
          b = FOREACH a GENERATE f1, f2, f3, NonDetUDF(f1,f2,f3) AS f4;
          SPLIT b INTO c IF f4 < 0.5, D OTHERWISE;
          
          Show
          Gianmarco De Francisci Morales added a comment - Created PIG-2169 for this. Anyway given the benefit/cost ratio I wouldn't try to fix it. A Nondeterministic UDF in a Split is probably better expressed as a Sample. Anyway I think this simple workaround should work: a = LOAD 'a.txt' AS (f1,f2,f3); b = FOREACH a GENERATE f1, f2, f3, NonDetUDF(f1,f2,f3) AS f4; SPLIT b INTO c IF f4 < 0.5, D OTHERWISE;
          Hide
          Thejas M Nair added a comment -

          Adding a note in release notes, about how this feature affects backward compatibility.

          Show
          Thejas M Nair added a comment - Adding a note in release notes, about how this feature affects backward compatibility.
          Hide
          Thejas M Nair added a comment -

          The approach you are proposing for @NonDeterministic udf sounds good.

          PIG-1904.1.patch looks good. Some comments -

          I think it is better to retain the restriction that a split needs at least two output aliases. This will prevent split being used instead of filter, and from pig becoming perl .

          Maybe, something like -
          split_clause : SPLIT rel INTO split_branch (COMMA split_branch)* ( COMMA split_branch ) |( COMMA split_otherwise ))

          In LogicalPlanBuilder.java, I think it is better to change the assertion to a if(root == null)

          {throw exception;}

          , as assertions are not enabled by default.

          Show
          Thejas M Nair added a comment - The approach you are proposing for @NonDeterministic udf sounds good. PIG-1904 .1.patch looks good. Some comments - I think it is better to retain the restriction that a split needs at least two output aliases. This will prevent split being used instead of filter, and from pig becoming perl . Maybe, something like - split_clause : SPLIT rel INTO split_branch (COMMA split_branch)* ( COMMA split_branch ) |( COMMA split_otherwise )) In LogicalPlanBuilder.java, I think it is better to change the assertion to a if(root == null) {throw exception;} , as assertions are not enabled by default.
          Hide
          Gianmarco De Francisci Morales added a comment -

          Attaching PIG-1902.2.patch
          Added unit tests for Split-Otherwise
          Added a check for Nondeterministic UDF. There was no need to create my own visitor, I reused the one available in Utils.
          Fixed issue with Split with 1 branch only. The solution proposed by Thejas does not work directly because the '*' is greedy, but I worked around it.

          I think it is ready for review.

          Show
          Gianmarco De Francisci Morales added a comment - Attaching PIG-1902 .2.patch Added unit tests for Split-Otherwise Added a check for Nondeterministic UDF. There was no need to create my own visitor, I reused the one available in Utils. Fixed issue with Split with 1 branch only. The solution proposed by Thejas does not work directly because the '*' is greedy, but I worked around it. I think it is ready for review.
          Hide
          Thejas M Nair added a comment -

          Looks good. +1 . I will commit it after running unit tests. test-patch was successful .

          Show
          Thejas M Nair added a comment - Looks good. +1 . I will commit it after running unit tests. test-patch was successful .
          Hide
          Thejas M Nair added a comment -

          Patch committed to trunk. Thanks Gianmarco!

          Show
          Thejas M Nair added a comment - Patch committed to trunk. Thanks Gianmarco!
          Hide
          Hardik added a comment -

          Hello,

          I got error while apply this.

          grunt> data = load 'atoz.csv' using PigStorage(',');

          grunt> dump data;

          ERROR org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl - Error while trying to run jobs.
          java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected.

          I am using ubuntu 12.02 32bit.
          I installed hadoop2.2.0 and pig 0.12 successfully.
          Haddop is runnig on my system.

          But, when i try to load a file

          so it gives error.
          Please address me where i am wrong.

          ---------------------------------------------------------------------------------------

          grunt> aatoz = load 'atoz.csv' using PigStorage(',');
          grunt> dump aatoz;
          2014-01-23 10:41:44,950 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN
          2014-01-23 10:41:44,968 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
          2014-01-23 10:41:44,969 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
          2014-01-23 10:41:44,969 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
          2014-01-23 10:41:44,971 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
          2014-01-23 10:41:44,972 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
          2014-01-23 10:41:44,972 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
          2014-01-23 10:41:44,984 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
          2014-01-23 10:41:44,998 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
          2014-01-23 10:41:45,000 [Thread-9] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
          2014-01-23 10:41:45,001 [Thread-9] ERROR org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl - Error while trying to run jobs.
          java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected
          at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:225)
          at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:186)
          at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:456)
          at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342)
          at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
          at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.security.auth.Subject.doAs(Subject.java:415)
          at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
          at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
          at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
          at org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:240)
          at java.lang.Thread.run(Thread.java:724)
          at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:260)
          2014-01-23 10:41:45,498 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
          2014-01-23 10:41:45,502 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job null has failed! Stop running all dependent jobs
          2014-01-23 10:41:45,503 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
          2014-01-23 10:41:45,507 [main] ERROR org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to recreate exception from backend error: Unexpected System Error Occured: java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected
          at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:225)
          at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:186)
          at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:456)
          at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342)
          at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
          at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.security.auth.Subject.doAs(Subject.java:415)
          at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
          at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
          at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
          at org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:240)
          at java.lang.Thread.run(Thread.java:724)
          at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:260)

          2014-01-23 10:41:45,507 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
          2014-01-23 10:41:45,507 [main] INFO org.apache.pig.tools.pigstats.SimplePigStats - Detected Local mode. Stats reported below may be incomplete
          2014-01-23 10:41:45,508 [main] INFO org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:

          HadoopVersion PigVersion UserId StartedAt FinishedAt Features
          2.2.0 0.10.1 hardik 2014-01-23 10:41:44 2014-01-23 10:41:45 UNKNOWN

          Failed!

          Failed Jobs:
          JobId Alias Feature Message Outputs
          N/A aatoz MAP_ONLY Message: Unexpected System Error Occured: java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected
          at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:225)
          at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:186)
          at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:456)
          at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342)
          at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
          at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.security.auth.Subject.doAs(Subject.java:415)
          at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
          at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
          at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
          at org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:240)
          at java.lang.Thread.run(Thread.java:724)
          at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:260)
          file:/tmp/temp1979716161/tmp-189979005,

          Input(s):
          Failed to read data from "file:///home/hardik/pig10/bin/input/atoz.csv"

          Output(s):
          Failed to produce result in "file:/tmp/temp1979716161/tmp-189979005"

          Job DAG:
          null

          2014-01-23 10:41:45,509 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
          2014-01-23 10:41:45,510 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1066: Unable to open iterator for alias aatoz
          Details at logfile: /home/hardik/pig10/bin/pig_1390453192689.log

          Show
          Hardik added a comment - Hello, I got error while apply this. grunt> data = load 'atoz.csv' using PigStorage(','); grunt> dump data; ERROR org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl - Error while trying to run jobs. java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected. I am using ubuntu 12.02 32bit. I installed hadoop2.2.0 and pig 0.12 successfully. Haddop is runnig on my system. But, when i try to load a file so it gives error. Please address me where i am wrong. --------------------------------------------------------------------------------------- grunt> aatoz = load 'atoz.csv' using PigStorage(','); grunt> dump aatoz; 2014-01-23 10:41:44,950 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN 2014-01-23 10:41:44,968 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false 2014-01-23 10:41:44,969 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1 2014-01-23 10:41:44,969 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1 2014-01-23 10:41:44,971 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 2014-01-23 10:41:44,972 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job 2014-01-23 10:41:44,972 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3 2014-01-23 10:41:44,984 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job 2014-01-23 10:41:44,998 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission. 2014-01-23 10:41:45,000 [Thread-9] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized 2014-01-23 10:41:45,001 [Thread-9] ERROR org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl - Error while trying to run jobs. java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:225) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:186) at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:456) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265) at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335) at org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:240) at java.lang.Thread.run(Thread.java:724) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:260) 2014-01-23 10:41:45,498 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete 2014-01-23 10:41:45,502 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job null has failed! Stop running all dependent jobs 2014-01-23 10:41:45,503 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete 2014-01-23 10:41:45,507 [main] ERROR org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to recreate exception from backend error: Unexpected System Error Occured: java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:225) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:186) at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:456) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265) at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335) at org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:240) at java.lang.Thread.run(Thread.java:724) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:260) 2014-01-23 10:41:45,507 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed! 2014-01-23 10:41:45,507 [main] INFO org.apache.pig.tools.pigstats.SimplePigStats - Detected Local mode. Stats reported below may be incomplete 2014-01-23 10:41:45,508 [main] INFO org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics: HadoopVersion PigVersion UserId StartedAt FinishedAt Features 2.2.0 0.10.1 hardik 2014-01-23 10:41:44 2014-01-23 10:41:45 UNKNOWN Failed! Failed Jobs: JobId Alias Feature Message Outputs N/A aatoz MAP_ONLY Message: Unexpected System Error Occured: java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:225) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:186) at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:456) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265) at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335) at org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:240) at java.lang.Thread.run(Thread.java:724) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:260) file:/tmp/temp1979716161/tmp-189979005 , Input(s): Failed to read data from "file:///home/hardik/pig10/bin/input/atoz.csv" Output(s): Failed to produce result in "file:/tmp/temp1979716161/tmp-189979005" Job DAG: null 2014-01-23 10:41:45,509 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed! 2014-01-23 10:41:45,510 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1066: Unable to open iterator for alias aatoz Details at logfile: /home/hardik/pig10/bin/pig_1390453192689.log
          Hide
          Daniel Dai added a comment -

          @hdbarot this should not relate to the patch. Are you running Hadoop 2? If so, you need to compile use "-Dhadoopversion=23" flag.

          Show
          Daniel Dai added a comment - @hdbarot this should not relate to the patch. Are you running Hadoop 2? If so, you need to compile use "-Dhadoopversion=23" flag.

            People

            • Assignee:
              Gianmarco De Francisci Morales
              Reporter:
              Daniel Dai
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development