Pig
  1. Pig
  2. PIG-2508

PIG can unpredictably ignore deprecated Hadoop config options

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.9.2, 0.10.0
    • Fix Version/s: 0.10.0, 0.9.3, 0.11
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When deprecated config options are passed to a Pig job, it can unpredictably ignore them and override them with values provided in the defaults due to a "race condition"-like issue.

      This problem was first noticed as part of MAPREDUCE-3665, which was re-filed as HADOOP-7993 so as for it to fall in the right component bucket of the code being fixed. This JIRA fixed the bug on the Hadoop side of the code that caused older deprecated config options to be ignored when they were also specified in the defaults xml file with the newer config name or vice versa.

      However, the problem seemed to persist with Pig jobs and HADOOP-8021 was filed to address the issue.

      A careful step-by-step execution of the code in a debugger reveals an second overlapping bug because of the way PIG is dealing with the configs.

      Not sure how / why this was not seen earlier, but the code in HExecutionEngine.java#recomputeProperties currently mashes together the default Hadoop configs and the user-specified properties into a Properties object. Given that it uses a HashTable to store the properties, if we have a config called "old.config.name" which is now deprecated and replaced by "new.config.name" and if one type is specified in the defaults and another by the user, we get a strange condition in which the repopulated Properties object has [in an unpredictable ordering] the following:

      config1.name=config1.value
      config2.name=config2.value
      ...
      old.config.name=old.config.value
      ...
      new.config.name=new.config.value
      ...
      configx.name=configx.value
      

      When this Properties object gets converted into a Configuration object by the ConfigurationUtil#toConfiguration() routine, the deprecation kicks in and tries to resolve all old configs. Because the ordering is not guaranteed (and because in the case of compress, the hash function consistently gives the new config loaded from the defaults after the old one), the user-specified config is ignored in favor of the default config (which from the point of view of the Hadoop Configuration object is expected standard behavior to replace an earlier specification of a config value with a later one).

      The fix for this is probably straightforward, but will require a re-write of the a chunk of code in HExecutionEngine.java. Instead of mashing together a JobConf object and a Properties object into a Configuration object that is finally re-converted into a JobConf object, the code simply needs to consistently and correctly populate a JobConf / Configuration object that can handle deprecation instead of a "dumb" Java Properties object.

      We recently saw another potential occurrence of this bug where Pig seems to honor only mapreduce.job.queuename parameter for specifying queue name and ignores the parameter mapred.job.queue.name.

      Since this can break a lot of existing jobs that run fine on 0.20, marking this as a blocker.

      1. PIG-2508.patch
        10 kB
        Thomas Weise
      2. PIG-2508.3.patch
        13 kB
        Thomas Weise
      3. PIG-2508.4.patch
        8 kB
        Thomas Weise
      4. PIG-2508.5.patch
        9 kB
        Thomas Weise
      5. PIG-2508-0.10-TestAvroStorage.patch
        0.8 kB
        Thomas Weise

        Issue Links

          Activity

          Hide
          Thomas Weise added a comment -

          Initial patch. With this change deprecated properties, when passed from the command line will be passed through to .23:

          bin/pig -Dmapred.output.compress=true -Dmapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec -x local tst.pig

          Same properties set in the script will still be ignored, more work to be done..

          Show
          Thomas Weise added a comment - Initial patch. With this change deprecated properties, when passed from the command line will be passed through to .23: bin/pig -Dmapred.output.compress=true -Dmapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec -x local tst.pig Same properties set in the script will still be ignored, more work to be done..
          Hide
          Thomas Weise added a comment -

          Updated patch to take care of properties set within the script. This is not a pretty fix - too many references exist to the shared properties object in PigContext.

          test-commit passes

          Show
          Thomas Weise added a comment - Updated patch to take care of properties set within the script. This is not a pretty fix - too many references exist to the shared properties object in PigContext. test-commit passes
          Hide
          Daniel Dai added a comment -

          I will take a look.

          Show
          Daniel Dai added a comment - I will take a look.
          Hide
          Anupam Seth added a comment -

          Tested Thomas' new patch on a 10-node cluster and ee the following:

          On 0.23:
          ========
          In local mode while setting configuration with deprecated name in script:
          -------------------------------------------------------------------------
          Fails with Kerberos exception as follows

          2012-02-07 22:00:08,696 [main] INFO  org.apache.pig.Main - Logging error messages to: /homes/<user>/pig_1328652008690.log
          2012-02-07 22:00:09,010 [main] WARN  org.apache.hadoop.conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
          2012-02-07 22:00:09,011 [main] WARN  org.apache.hadoop.conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
          2012-02-07 22:00:09,011 [main] WARN  org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
          2012-02-07 22:00:09,011 [main] WARN  org.apache.hadoop.conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
          2012-02-07 22:00:09,011 [main] WARN  org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
          2012-02-07 22:00:09,011 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
          2012-02-07 22:00:09,404 [main] WARN  org.apache.hadoop.conf.Configuration - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
          2012-02-07 22:00:09,405 [main] WARN  org.apache.hadoop.conf.Configuration - mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
          2012-02-07 22:00:10,386 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN
          2012-02-07 22:00:10,540 [main] WARN  org.apache.hadoop.conf.Configuration - mapred.textoutputformat.separator is deprecated. Instead, use mapreduce.output.textoutputformat.separator
          2012-02-07 22:00:10,553 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 6000: 
          <file script2-hadoop.pig, line 7, column 0> Output Location Validation Failed for: 'file:///homes/ghimport/script2-hadoop-results More info to follow:
          Can't get Master Kerberos principal for use as renewer
          Details at logfile: /homes/<user>/pig_1328652008690.log
          

          Contents of pig log file

          Pig Stack Trace
          ---------------
          ERROR 6000:
          <file script2-hadoop.pig, line 7, column 0> Output Location Validation Failed for: 'file:///homes/<user>/script2-hadoop-results More info to follow:
          Can't get Master Kerberos principal for use as renewer
          
          org.apache.pig.impl.plan.VisitorException: ERROR 6000:
          <file script2-hadoop.pig, line 7, column 0> Output Location Validation Failed for: 'file:///homes/<user>/script2-hadoop-results More info to follow:
          Can't get Master Kerberos principal for use as renewer
                  at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:95)
                  at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:77)
                  at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
                  at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
                  at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
                  at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
                  at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
                  at org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:45)
                  at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:293)
                  at org.apache.pig.PigServer.compilePp(PigServer.java:1360)
                  at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1297)
                  at org.apache.pig.PigServer.execute(PigServer.java:1289)
                  at org.apache.pig.PigServer.executeBatch(PigServer.java:360)
                  at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:130)
                  at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:191)
                  at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:163)
                  at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
                  at org.apache.pig.Main.run(Main.java:561)
                  at org.apache.pig.Main.main(Main.java:111)
                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                  at java.lang.reflect.Method.invoke(Method.java:597)
                  at org.apache.hadoop.util.RunJar.main(RunJar.java:200)
          Caused by: java.io.IOException: Can't get Master Kerberos principal for use as renewer
                  at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:104)
                  at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:87)
                  at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80)
                  at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:137)
                  at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:80)
                  ... 23 more
          ================================================================================
          

          In cluster mode while setting configuration with deprecated name in script:
          ---------------------------------------------------------------------------
          Passes

          In local mode while setting configuration with new name in script:
          ------------------------------------------------------------------
          Same issue as with local mode above

          In cluster mode while setting configuration with new name in script:
          --------------------------------------------------------------------
          Fails as below

          2012-02-07 22:08:27,164 [main] INFO  org.apache.pig.Main - Logging error messages to: /homes/<user>/pig_1328652507159.log
          2012-02-07 22:08:27,663 [main] WARN  org.apache.hadoop.conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
          2012-02-07 22:08:27,665 [main] WARN  org.apache.hadoop.conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
          2012-02-07 22:08:27,665 [main] WARN  org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
          2012-02-07 22:08:27,665 [main] WARN  org.apache.hadoop.conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
          2012-02-07 22:08:27,665 [main] WARN  org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
          2012-02-07 22:08:27,665 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://<host>
          2012-02-07 22:08:32,052 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN
          2012-02-07 22:08:32,349 [main] WARN  org.apache.hadoop.conf.Configuration - mapred.textoutputformat.separator is deprecated. Instead, use mapreduce.output.textoutputformat.separator
          2012-02-07 22:08:32,360 [main] INFO  org.apache.hadoop.hdfs.DFSClient - Created HDFS_DELEGATION_TOKEN token 28 for <user> on <host>
          2012-02-07 22:08:32,360 [main] INFO  org.apache.hadoop.mapreduce.security.TokenCache - Got dt for hdfs://<host>
          2012-02-07 22:08:32,787 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
          2012-02-07 22:08:32,964 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
          2012-02-07 22:08:32,964 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
          2012-02-07 22:08:33,919 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
          2012-02-07 22:08:33,963 [main] WARN  org.apache.hadoop.conf.Configuration - mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use mapreduce.reduce.markreset.buffer.percent
          2012-02-07 22:08:33,963 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
          2012-02-07 22:08:33,963 [main] WARN  org.apache.hadoop.conf.Configuration - mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use mapreduce.reduce.markreset.buffer.percent
          2012-02-07 22:08:33,963 [main] WARN  org.apache.hadoop.conf.Configuration - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
          2012-02-07 22:08:33,964 [main] WARN  org.apache.hadoop.conf.Configuration - mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
          2012-02-07 22:08:33,974 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 0: 'mapred.output.compress' is set but no value is specified for 'mapred.output.compression.codec'.
          Details at logfile: /homes/<user>/pig_1328652507159.log
          

          Contents of log file:

          Pig Stack Trace
          ---------------
          ERROR 0: 'mapred.output.compress' is set but no value is specified for 'mapred.output.compression.codec'.
          
          org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException: ERROR 0: 'mapred.output.compress' is set but no value is specified for 'mapred.output.compression.codec'.
                  at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:365)
                  at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:258)
                  at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:150)
                  at org.apache.pig.PigServer.launchPlan(PigServer.java:1314)
                  at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1299)
                  at org.apache.pig.PigServer.execute(PigServer.java:1289)
                  at org.apache.pig.PigServer.executeBatch(PigServer.java:360)
                  at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:130)
                  at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:191)
                  at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:163)
                  at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
                  at org.apache.pig.Main.run(Main.java:561)
                  at org.apache.pig.Main.main(Main.java:111)
                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                  at java.lang.reflect.Method.invoke(Method.java:597)
                  at org.apache.hadoop.util.RunJar.main(RunJar.java:200)
          ================================================================================
          

          In local mode while setting configuration with deprecated name from cmd line:
          -----------------------------------------------------------------------------
          Same issue as with local mode above

          In cluster mode while setting configuration with deprecated name from cmd line:
          -------------------------------------------------------------------------------
          Passes

          In local mode while setting configuration with new name from cmd line:
          ----------------------------------------------------------------------
          Same issue as with local mode above

          In cluster mode while setting configuration with new name from cmd line:
          ------------------------------------------------------------------------
          Passes

          On 0.20.2xx:
          ============
          Cannot get it to work at all (tried removing my ivy2 directory, doing ant clean, and then re-compiling the tarball for 0.20 - still, it smells like I have 0.23 libs being referenced somewhere!)

          2012-02-07 23:08:46,237 [main] INFO  org.apache.pig.Main - Logging error messages to: /homes/ghimport/pig_1328656126229.log
          2012-02-07 23:08:46,716 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
          2012-02-07 23:08:47,140 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2998: Unhandled internal error. org/apache/hadoop/mapreduce/task/JobContextImpl
          Details at logfile: /homes/<user>/pig_1328656126229.log
          Exception in thread "main" java.lang.NoClassDefFoundError: Could not initialize class org.apache.pig.tools.pigstats.PigStatsUtil
                  at org.apache.pig.Main.run(Main.java:593)
                  at org.apache.pig.Main.main(Main.java:111)
                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                  at java.lang.reflect.Method.invoke(Method.java:597)
                  at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
          

          Contents of pig log file:

          Pig Stack Trace
          ---------------
          ERROR 2998: Unhandled internal error. org/apache/hadoop/mapreduce/task/JobContextImpl
          
          java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/task/JobContextImpl
                  at org.apache.pig.tools.pigstats.PigStatsUtil.<clinit>(PigStatsUtil.java:54)
                  at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:82)
                  at org.apache.pig.Main.run(Main.java:561)
                  at org.apache.pig.Main.main(Main.java:111)
                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                  at java.lang.reflect.Method.invoke(Method.java:597)
                  at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
          Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.task.JobContextImpl
                  at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
                  at java.security.AccessController.doPrivileged(Native Method)
                  at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
                  at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
                  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
                  at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
                  ... 9 more
          ================================================================================
          
          Show
          Anupam Seth added a comment - Tested Thomas' new patch on a 10-node cluster and ee the following: On 0.23: ======== In local mode while setting configuration with deprecated name in script: ------------------------------------------------------------------------- Fails with Kerberos exception as follows 2012-02-07 22:00:08,696 [main] INFO org.apache.pig.Main - Logging error messages to: /homes/<user>/pig_1328652008690.log 2012-02-07 22:00:09,010 [main] WARN org.apache.hadoop.conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used 2012-02-07 22:00:09,011 [main] WARN org.apache.hadoop.conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address 2012-02-07 22:00:09,011 [main] WARN org.apache.hadoop.conf.Configuration - fs. default .name is deprecated. Instead, use fs.defaultFS 2012-02-07 22:00:09,011 [main] WARN org.apache.hadoop.conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address 2012-02-07 22:00:09,011 [main] WARN org.apache.hadoop.conf.Configuration - fs. default .name is deprecated. Instead, use fs.defaultFS 2012-02-07 22:00:09,011 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file: /// 2012-02-07 22:00:09,404 [main] WARN org.apache.hadoop.conf.Configuration - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress 2012-02-07 22:00:09,405 [main] WARN org.apache.hadoop.conf.Configuration - mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec 2012-02-07 22:00:10,386 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN 2012-02-07 22:00:10,540 [main] WARN org.apache.hadoop.conf.Configuration - mapred.textoutputformat.separator is deprecated. Instead, use mapreduce.output.textoutputformat.separator 2012-02-07 22:00:10,553 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 6000: <file script2-hadoop.pig, line 7, column 0> Output Location Validation Failed for : 'file: ///homes/ghimport/script2-hadoop-results More info to follow: Can't get Master Kerberos principal for use as renewer Details at logfile: /homes/<user>/pig_1328652008690.log Contents of pig log file Pig Stack Trace --------------- ERROR 6000: <file script2-hadoop.pig, line 7, column 0> Output Location Validation Failed for : 'file: ///homes/<user>/script2-hadoop-results More info to follow: Can't get Master Kerberos principal for use as renewer org.apache.pig.impl.plan.VisitorException: ERROR 6000: <file script2-hadoop.pig, line 7, column 0> Output Location Validation Failed for : 'file: ///homes/<user>/script2-hadoop-results More info to follow: Can't get Master Kerberos principal for use as renewer at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:95) at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:77) at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64) at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66) at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53) at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50) at org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:45) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:293) at org.apache.pig.PigServer.compilePp(PigServer.java:1360) at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1297) at org.apache.pig.PigServer.execute(PigServer.java:1289) at org.apache.pig.PigServer.executeBatch(PigServer.java:360) at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:130) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:191) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:163) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84) at org.apache.pig.Main.run(Main.java:561) at org.apache.pig.Main.main(Main.java:111) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:200) Caused by: java.io.IOException: Can't get Master Kerberos principal for use as renewer at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:104) at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:87) at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80) at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:137) at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:80) ... 23 more ================================================================================ In cluster mode while setting configuration with deprecated name in script: --------------------------------------------------------------------------- Passes In local mode while setting configuration with new name in script: ------------------------------------------------------------------ Same issue as with local mode above In cluster mode while setting configuration with new name in script: -------------------------------------------------------------------- Fails as below 2012-02-07 22:08:27,164 [main] INFO org.apache.pig.Main - Logging error messages to: /homes/<user>/pig_1328652507159.log 2012-02-07 22:08:27,663 [main] WARN org.apache.hadoop.conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used 2012-02-07 22:08:27,665 [main] WARN org.apache.hadoop.conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address 2012-02-07 22:08:27,665 [main] WARN org.apache.hadoop.conf.Configuration - fs. default .name is deprecated. Instead, use fs.defaultFS 2012-02-07 22:08:27,665 [main] WARN org.apache.hadoop.conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address 2012-02-07 22:08:27,665 [main] WARN org.apache.hadoop.conf.Configuration - fs. default .name is deprecated. Instead, use fs.defaultFS 2012-02-07 22:08:27,665 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs: //<host> 2012-02-07 22:08:32,052 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN 2012-02-07 22:08:32,349 [main] WARN org.apache.hadoop.conf.Configuration - mapred.textoutputformat.separator is deprecated. Instead, use mapreduce.output.textoutputformat.separator 2012-02-07 22:08:32,360 [main] INFO org.apache.hadoop.hdfs.DFSClient - Created HDFS_DELEGATION_TOKEN token 28 for <user> on <host> 2012-02-07 22:08:32,360 [main] INFO org.apache.hadoop.mapreduce.security.TokenCache - Got dt for hdfs: //<host> 2012-02-07 22:08:32,787 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false 2012-02-07 22:08:32,964 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1 2012-02-07 22:08:32,964 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1 2012-02-07 22:08:33,919 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job 2012-02-07 22:08:33,963 [main] WARN org.apache.hadoop.conf.Configuration - mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use mapreduce.reduce.markreset.buffer.percent 2012-02-07 22:08:33,963 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3 2012-02-07 22:08:33,963 [main] WARN org.apache.hadoop.conf.Configuration - mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use mapreduce.reduce.markreset.buffer.percent 2012-02-07 22:08:33,963 [main] WARN org.apache.hadoop.conf.Configuration - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress 2012-02-07 22:08:33,964 [main] WARN org.apache.hadoop.conf.Configuration - mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec 2012-02-07 22:08:33,974 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 0: 'mapred.output.compress' is set but no value is specified for 'mapred.output.compression.codec'. Details at logfile: /homes/<user>/pig_1328652507159.log Contents of log file: Pig Stack Trace --------------- ERROR 0: 'mapred.output.compress' is set but no value is specified for 'mapred.output.compression.codec'. org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException: ERROR 0: 'mapred.output.compress' is set but no value is specified for 'mapred.output.compression.codec'. at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:365) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:258) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:150) at org.apache.pig.PigServer.launchPlan(PigServer.java:1314) at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1299) at org.apache.pig.PigServer.execute(PigServer.java:1289) at org.apache.pig.PigServer.executeBatch(PigServer.java:360) at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:130) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:191) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:163) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84) at org.apache.pig.Main.run(Main.java:561) at org.apache.pig.Main.main(Main.java:111) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:200) ================================================================================ In local mode while setting configuration with deprecated name from cmd line: ----------------------------------------------------------------------------- Same issue as with local mode above In cluster mode while setting configuration with deprecated name from cmd line: ------------------------------------------------------------------------------- Passes In local mode while setting configuration with new name from cmd line: ---------------------------------------------------------------------- Same issue as with local mode above In cluster mode while setting configuration with new name from cmd line: ------------------------------------------------------------------------ Passes On 0.20.2xx: ============ Cannot get it to work at all (tried removing my ivy2 directory, doing ant clean, and then re-compiling the tarball for 0.20 - still, it smells like I have 0.23 libs being referenced somewhere!) 2012-02-07 23:08:46,237 [main] INFO org.apache.pig.Main - Logging error messages to: /homes/ghimport/pig_1328656126229.log 2012-02-07 23:08:46,716 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file: /// 2012-02-07 23:08:47,140 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2998: Unhandled internal error. org/apache/hadoop/mapreduce/task/JobContextImpl Details at logfile: /homes/<user>/pig_1328656126229.log Exception in thread "main" java.lang.NoClassDefFoundError: Could not initialize class org.apache.pig.tools.pigstats.PigStatsUtil at org.apache.pig.Main.run(Main.java:593) at org.apache.pig.Main.main(Main.java:111) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Contents of pig log file: Pig Stack Trace --------------- ERROR 2998: Unhandled internal error. org/apache/hadoop/mapreduce/task/JobContextImpl java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/task/JobContextImpl at org.apache.pig.tools.pigstats.PigStatsUtil.<clinit>(PigStatsUtil.java:54) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:82) at org.apache.pig.Main.run(Main.java:561) at org.apache.pig.Main.main(Main.java:111) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.task.JobContextImpl at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang. ClassLoader .loadClass( ClassLoader .java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang. ClassLoader .loadClass( ClassLoader .java:248) ... 9 more ================================================================================
          Hide
          Thomas Weise added a comment -

          The failures in local mode look more like a .23 Hadoop cluster environment issue (I see the same running on my cluster with 0.9 head w/ and w/o patch).

          In cluster mode, passing the compression properties from command line and from script pass.

          Script:

          set mapred.output.compress true
          set mapred.output.compression.codec org.apache.hadoop.io.compress.BZip2Codec
          rmf t1-results
          raw = LOAD 't1.txt' USING PigStorage('\t') AS (user, time);
          STORE raw INTO 't1-results' USING PigStorage();
          

          Command line (w/o set in script):

          bin/pig -Dmapred.output.compress=true -Dmapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec tst.pig
          
          Show
          Thomas Weise added a comment - The failures in local mode look more like a .23 Hadoop cluster environment issue (I see the same running on my cluster with 0.9 head w/ and w/o patch). In cluster mode, passing the compression properties from command line and from script pass. Script: set mapred.output.compress true set mapred.output.compression.codec org.apache.hadoop.io.compress.BZip2Codec rmf t1-results raw = LOAD 't1.txt' USING PigStorage('\t') AS (user, time); STORE raw INTO 't1-results' USING PigStorage(); Command line (w/o set in script): bin/pig -Dmapred.output.compress= true -Dmapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec tst.pig
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/3805/
          -----------------------------------------------------------

          Review request for pig.

          Summary
          -------

          See PIG-2508

          This addresses bug PIG-2508.
          https://issues.apache.org/jira/browse/PIG-2508

          Diffs


          trunk/src/org/apache/pig/backend/datastorage/DataStorage.java 1241308
          trunk/src/org/apache/pig/backend/hadoop/datastorage/HDataStorage.java 1241308
          trunk/src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java 1241308
          trunk/src/org/apache/pig/tools/grunt/GruntParser.java 1241308
          trunk/test/org/apache/pig/test/TestInputOutputFileValidator.java 1241308

          Diff: https://reviews.apache.org/r/3805/diff

          Testing
          -------

          Thanks,

          Daniel

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3805/ ----------------------------------------------------------- Review request for pig. Summary ------- See PIG-2508 This addresses bug PIG-2508 . https://issues.apache.org/jira/browse/PIG-2508 Diffs trunk/src/org/apache/pig/backend/datastorage/DataStorage.java 1241308 trunk/src/org/apache/pig/backend/hadoop/datastorage/HDataStorage.java 1241308 trunk/src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java 1241308 trunk/src/org/apache/pig/tools/grunt/GruntParser.java 1241308 trunk/test/org/apache/pig/test/TestInputOutputFileValidator.java 1241308 Diff: https://reviews.apache.org/r/3805/diff Testing ------- Thanks, Daniel
          Hide
          Thomas Weise added a comment -

          Current patch causes failure in TestHBaseStorage

          Show
          Thomas Weise added a comment - Current patch causes failure in TestHBaseStorage
          Hide
          Thomas Weise added a comment -

          New patch for 0.9: Before using jobConf to set new values from script, set all current properties from pigContext.

          Show
          Thomas Weise added a comment - New patch for 0.9: Before using jobConf to set new values from script, set all current properties from pigContext.
          Hide
          Thomas Weise added a comment -

          TestHBaseStorage passes with this patch. Will run full tests again.

          Show
          Thomas Weise added a comment - TestHBaseStorage passes with this patch. Will run full tests again.
          Hide
          Daniel Dai added a comment -

          PIG-2508.4.patch is much better. Now new property will not add to pigcontext.properties directly, instead, it injects into jobconf and then read back to pigcontext.properties. Jobconf will have a chance to process depredation key. It is cumbersome but should work. I am fine with it for 0.9 cuz it requires less change. For 0.10/trunk, I would like to make things more cleaner.

          Show
          Daniel Dai added a comment - PIG-2508 .4.patch is much better. Now new property will not add to pigcontext.properties directly, instead, it injects into jobconf and then read back to pigcontext.properties. Jobconf will have a chance to process depredation key. It is cumbersome but should work. I am fine with it for 0.9 cuz it requires less change. For 0.10/trunk, I would like to make things more cleaner.
          Hide
          Thomas Weise added a comment -

          All unit tests pass with .4 patch.

          @Daniel: Key is to set properties through an instance of JobConf, as this is the only way to apply deprecation logic. The interface contracts of java.util.Properties and JobConf variant of Configuration are not compatible. What we put into JobConf is not necessarily what we get back for a given key. A cleaner solution could be to change all relate code to work with Configuration instead of Properties and use a JobConf instance as prototype.

          Show
          Thomas Weise added a comment - All unit tests pass with .4 patch. @Daniel: Key is to set properties through an instance of JobConf, as this is the only way to apply deprecation logic. The interface contracts of java.util.Properties and JobConf variant of Configuration are not compatible. What we put into JobConf is not necessarily what we get back for a given key. A cleaner solution could be to change all relate code to work with Configuration instead of Properties and use a JobConf instance as prototype.
          Hide
          Thomas Weise added a comment -

          Zebra unit tests fail with the following error:

          
          Unable to open iterator for alias records
          org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias records
                  at org.apache.pig.PigServer.openIterator(PigServer.java:901)
                  at org.apache.hadoop.zebra.pig.TestMapTableLoader.testReader(TestMapTableLoader.java:136)
          Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias records
                  at org.apache.pig.PigServer.storeEx(PigServer.java:1000)
                  at org.apache.pig.PigServer.store(PigServer.java:963)
                  at org.apache.pig.PigServer.openIterator(PigServer.java:876)
          Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2043: Unexpected error during execution.
                  at org.apache.pig.PigServer.launchPlan(PigServer.java:1325)
                  at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1299)
                  at org.apache.pig.PigServer.storeEx(PigServer.java:996)
          Caused by: java.lang.IllegalStateException: Variable substitution depth too large: 20 ${fs.default.name}
                  at org.apache.hadoop.conf.Configuration.substituteVars(Configuration.java:399)
                  at org.apache.hadoop.conf.Configuration.get(Configuration.java:469)
                  at org.apache.hadoop.fs.FileSystem.getDefaultUri(FileSystem.java:131)
                  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:242)
                  at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:225)
                  at org.apache.hadoop.mapred.LocalJobRunner.<init>(LocalJobRunner.java:418)
                  at org.apache.hadoop.mapred.JobClient.init(JobClient.java:472)
                  at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:457)
                  at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:125)
                  at org.apache.pig.PigServer.launchPlan(PigServer.java:1314)
          
          
          
          Show
          Thomas Weise added a comment - Zebra unit tests fail with the following error: Unable to open iterator for alias records org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias records at org.apache.pig.PigServer.openIterator(PigServer.java:901) at org.apache.hadoop.zebra.pig.TestMapTableLoader.testReader(TestMapTableLoader.java:136) Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias records at org.apache.pig.PigServer.storeEx(PigServer.java:1000) at org.apache.pig.PigServer.store(PigServer.java:963) at org.apache.pig.PigServer.openIterator(PigServer.java:876) Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2043: Unexpected error during execution. at org.apache.pig.PigServer.launchPlan(PigServer.java:1325) at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1299) at org.apache.pig.PigServer.storeEx(PigServer.java:996) Caused by: java.lang.IllegalStateException: Variable substitution depth too large: 20 ${fs. default .name} at org.apache.hadoop.conf.Configuration.substituteVars(Configuration.java:399) at org.apache.hadoop.conf.Configuration.get(Configuration.java:469) at org.apache.hadoop.fs.FileSystem.getDefaultUri(FileSystem.java:131) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:242) at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:225) at org.apache.hadoop.mapred.LocalJobRunner.<init>(LocalJobRunner.java:418) at org.apache.hadoop.mapred.JobClient.init(JobClient.java:472) at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:457) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:125) at org.apache.pig.PigServer.launchPlan(PigServer.java:1314)
          Hide
          Thomas Weise added a comment -

          Adding fix to zebra build.xml to not pass placeholders for undefined properties into unit test.

          Show
          Thomas Weise added a comment - Adding fix to zebra build.xml to not pass placeholders for undefined properties into unit test.
          Hide
          Daniel Dai added a comment -

          +1 for 0.9 branch. I will commit it shortly.

          Show
          Daniel Dai added a comment - +1 for 0.9 branch. I will commit it shortly.
          Hide
          Daniel Dai added a comment -

          test-patch:
          [exec] -1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
          [exec] Please justify why no tests are needed for this patch.
          [exec]
          [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
          [exec]
          [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

          Patch committed to 0.9 branch. For 0.10/trunk, we might use a different approach. I will open a separate Jira to track.

          Thanks Thomas!

          Show
          Daniel Dai added a comment - test-patch: [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no tests are needed for this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. Patch committed to 0.9 branch. For 0.10/trunk, we might use a different approach. I will open a separate Jira to track. Thanks Thomas!
          Hide
          Thomas Weise added a comment -

          Fix needed for 0.10 until better solution can be developed.

          Show
          Thomas Weise added a comment - Fix needed for 0.10 until better solution can be developed.
          Hide
          Thomas Weise added a comment -

          Unit tests pass on 0.10

          Show
          Thomas Weise added a comment - Unit tests pass on 0.10
          Hide
          Daniel Dai added a comment -

          Committed to 0.10 as Thomas requested. I need couple of more days to work on a better fix, which will go to trunk.

          Show
          Daniel Dai added a comment - Committed to 0.10 as Thomas requested. I need couple of more days to work on a better fix, which will go to trunk.
          Hide
          Thomas Weise added a comment -

          0.10 also needs a change to TestAvroStorage, which otherwise fails on 0.23 (patch attached). See PIG-2319

          
          Testcase: testArrayWithSnappyCompression took 0.033 sec
                  Caused an ERROR
          'mapred.output.compress' is set but no value is specified for 'mapred.output.compression.codec'.
          org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException: ERROR 0: 'mapred.output.compress' is set but no value is specified for 'mapred.output.compression.codec'.
                  at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:366)
          
          Show
          Thomas Weise added a comment - 0.10 also needs a change to TestAvroStorage, which otherwise fails on 0.23 (patch attached). See PIG-2319 Testcase: testArrayWithSnappyCompression took 0.033 sec Caused an ERROR 'mapred.output.compress' is set but no value is specified for 'mapred.output.compression.codec'. org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException: ERROR 0: 'mapred.output.compress' is set but no value is specified for 'mapred.output.compression.codec'. at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:366)
          Hide
          Daniel Dai added a comment -

          PIG-2508-0.10-TestAvroStorage.patch committed to 0.10/trunk.

          Show
          Daniel Dai added a comment - PIG-2508 -0.10-TestAvroStorage.patch committed to 0.10/trunk.
          Hide
          Daniel Dai added a comment -

          Fixed in 0.9/0.10. For trunk, I would like to address it in PIG-2552.

          Show
          Daniel Dai added a comment - Fixed in 0.9/0.10. For trunk, I would like to address it in PIG-2552 .
          Hide
          Julien Le Dem added a comment -

          Hi Daniel,
          As it seems PIG-2552 has not made progress so far, can we commit this patch in trunk to avoid a regression when using trunk?
          PIG-2552 can still happen when it's ready.

          Show
          Julien Le Dem added a comment - Hi Daniel, As it seems PIG-2552 has not made progress so far, can we commit this patch in trunk to avoid a regression when using trunk? PIG-2552 can still happen when it's ready.
          Hide
          Daniel Dai added a comment -

          That's fine for me.

          Show
          Daniel Dai added a comment - That's fine for me.
          Hide
          Dmitriy V. Ryaboy added a comment -

          Committed to trunk.

          Show
          Dmitriy V. Ryaboy added a comment - Committed to trunk.

            People

            • Assignee:
              Thomas Weise
              Reporter:
              Anupam Seth
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development