Pig
  1. Pig
  2. PIG-2738

pig.exec.reducers.max has no default value

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 0.10.0
    • Fix Version/s: None
    • Component/s: impl
    • Labels:
      None
    • Environment:

      Kubuntu 12.04 64Bit

      Description

      setDefaultsIfUnset in org/apache/pig/impl/util/PropertiesUtil.java does not set pig.exec.reducers.max to 999 as documented. As a consequence testDefaultPigProperties in org.apache.pig.test.TestPigServer fails with a NullPointerException accessing the property.

        Issue Links

          Activity

          Hide
          Johannes Schwenk added a comment -

          Attached proposed patch.

          Show
          Johannes Schwenk added a comment - Attached proposed patch.
          Hide
          Prashant Kommireddi added a comment -

          Hi Johannes,

          Is pig-default.properties on your classpath? Max reducers is set in this file and should be loaded into Properties object by default.

              public PigServer(ExecType execType) throws ExecException {
                  this(execType, PropertiesUtil.loadDefaultProperties());
              }
          

          loadDefaultProperties looks for pig-default.properties

              public static void loadDefaultProperties(Properties properties) {
                  loadPropertiesFromFile(properties,
                          System.getProperty("user.home") + "/.pigrc");
                  loadPropertiesFromClasspath(properties, DEFAULT_PROPERTIES_FILE);
          

          This should be present in your PIG_HOME/conf directory?

          Show
          Prashant Kommireddi added a comment - Hi Johannes, Is pig-default.properties on your classpath? Max reducers is set in this file and should be loaded into Properties object by default. public PigServer(ExecType execType) throws ExecException { this (execType, PropertiesUtil.loadDefaultProperties()); } loadDefaultProperties looks for pig-default.properties public static void loadDefaultProperties(Properties properties) { loadPropertiesFromFile(properties, System .getProperty( "user.home" ) + "/.pigrc" ); loadPropertiesFromClasspath(properties, DEFAULT_PROPERTIES_FILE); This should be present in your PIG_HOME/conf directory?
          Hide
          Johannes Schwenk added a comment -

          Hi Prashant!

          Well I see, how this should work now... All default properties in the pig namespace are going into pig-default.properties and all other defaults are getting set by setDefaultsIfUnset ?!

          I still am wondering why I have no pig-default.properties in my classpath. It seems like the prebuilt jars in the 0.10.0 release are containing the pig-default.properties file, but it is missing from the PIG_HOME/conf dir, so that my custom built jars ended up missing the file and thus the tests failed. Is this known or should I file a bug?

          Thanks again for clearing this up!

          Show
          Johannes Schwenk added a comment - Hi Prashant! Well I see, how this should work now... All default properties in the pig namespace are going into pig-default.properties and all other defaults are getting set by setDefaultsIfUnset ?! I still am wondering why I have no pig-default.properties in my classpath. It seems like the prebuilt jars in the 0.10.0 release are containing the pig-default.properties file, but it is missing from the PIG_HOME/conf dir, so that my custom built jars ended up missing the file and thus the tests failed. Is this known or should I file a bug? Thanks again for clearing this up!
          Hide
          Johannes Schwenk added a comment -

          As Arvind Prabhakar has discovered there are differences between the svn version tagges release-0.10.0 and the tar.gz version that is advertised as release version 0.10.0 for download from apache.org... I use the tar.gz which does not contain the PIG_HOME/conf/pig-default.properties !

          That said maybe there could be a warning if the file is missing during ant build? That way unsuspecting developers would not spend hours on debugging this.

          Show
          Johannes Schwenk added a comment - As Arvind Prabhakar has discovered there are differences between the svn version tagges release-0.10.0 and the tar.gz version that is advertised as release version 0.10.0 for download from apache.org... I use the tar.gz which does not contain the PIG_HOME/conf/pig-default.properties ! That said maybe there could be a warning if the file is missing during ant build? That way unsuspecting developers would not spend hours on debugging this.
          Hide
          Daniel Dai added a comment -

          The release tar ball does not contains pig-default.properties. It is bundled into pig-withouthadoop.jar. However, if you rebuild the release, the generated pig-withouthadoop.jar does not contain pig-default.properties. It is a known issue and is tracked in PIG-2669.

          Show
          Daniel Dai added a comment - The release tar ball does not contains pig-default.properties. It is bundled into pig-withouthadoop.jar. However, if you rebuild the release, the generated pig-withouthadoop.jar does not contain pig-default.properties. It is a known issue and is tracked in PIG-2669 .
          Hide
          Johannes Schwenk added a comment -

          Apparently this does not need a patch (removed proposed patch) and is already beeing tracked. So I marked this as a clone of PIG-2669.

          Show
          Johannes Schwenk added a comment - Apparently this does not need a patch (removed proposed patch) and is already beeing tracked. So I marked this as a clone of PIG-2669 .

            People

            • Assignee:
              Unassigned
              Reporter:
              Johannes Schwenk
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development