Hadoop Common
  1. Hadoop Common
  2. HADOOP-7287

Configuration deprecation mechanism doesn't work properly for GenericOptionsParser/Tools

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.22.0, 0.23.0
    • Fix Version/s: 0.22.0, 0.23.0
    • Component/s: conf
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      For example, you can't use -D options on the "hadoop fs" command line in order to specify the deprecated names of configuration options. The issue is that the ordering is:

      • JVM starts
      • GenericOptionsParser creates a Configuration object and calls set() for each of the options specified on command line
      • DistributedFileSystem or other class eventually instantiates HdfsConfiguration which adds the deprecations
      • Some class calls conf.get("new key") and sees the default instead of the version set on the command line
      1. hadoop-7287-trunk.2.patch
        6 kB
        Aaron T. Myers
      2. hadoop-7287-trunk.1.patch
        5 kB
        Aaron T. Myers
      3. hadoop-7287-still-broken.txt
        5 kB
        Todd Lipcon
      4. hadoop-7287-trunk.0.patch
        3 kB
        Aaron T. Myers
      5. hadoop-7287-testcase.txt
        0.7 kB
        Todd Lipcon

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #698 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk/698/)
          HADOOP-7287. Configuration deprecation mechanism doesn't work properly for GenericOptionsParser and Tools. Contributed by Aaron T. Myers.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1126719
          Files :

          • /hadoop/common/trunk/CHANGES.txt
          • /hadoop/common/trunk/src/java/org/apache/hadoop/conf/Configuration.java
          • /hadoop/common/trunk/src/test/core/org/apache/hadoop/conf/TestConfigurationDeprecation.java
          • /hadoop/common/trunk/src/test/test-fake-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #698 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk/698/ ) HADOOP-7287 . Configuration deprecation mechanism doesn't work properly for GenericOptionsParser and Tools. Contributed by Aaron T. Myers. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1126719 Files : /hadoop/common/trunk/CHANGES.txt /hadoop/common/trunk/src/java/org/apache/hadoop/conf/Configuration.java /hadoop/common/trunk/src/test/core/org/apache/hadoop/conf/TestConfigurationDeprecation.java /hadoop/common/trunk/src/test/test-fake-default.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-22-branch #56 (See https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/56/)
          HADOOP-7287. Configuration deprecation mechanism doesn't work properly for GenericOptionsParser and Tools. Contributed by Aaron T. Myers.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1126724
          Files :

          • /hadoop/common/branches/branch-0.22/src/test/core/org/apache/hadoop/conf/TestConfigurationDeprecation.java
          • /hadoop/common/branches/branch-0.22/src/test/test-fake-default.xml
          • /hadoop/common/branches/branch-0.22/CHANGES.txt
          • /hadoop/common/branches/branch-0.22/src/java/org/apache/hadoop/conf/Configuration.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-22-branch #56 (See https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/56/ ) HADOOP-7287 . Configuration deprecation mechanism doesn't work properly for GenericOptionsParser and Tools. Contributed by Aaron T. Myers. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1126724 Files : /hadoop/common/branches/branch-0.22/src/test/core/org/apache/hadoop/conf/TestConfigurationDeprecation.java /hadoop/common/branches/branch-0.22/src/test/test-fake-default.xml /hadoop/common/branches/branch-0.22/CHANGES.txt /hadoop/common/branches/branch-0.22/src/java/org/apache/hadoop/conf/Configuration.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #616 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/616/)
          HADOOP-7287. Configuration deprecation mechanism doesn't work properly for GenericOptionsParser and Tools. Contributed by Aaron T. Myers.

          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1126719
          Files :

          • /hadoop/common/trunk/CHANGES.txt
          • /hadoop/common/trunk/src/java/org/apache/hadoop/conf/Configuration.java
          • /hadoop/common/trunk/src/test/core/org/apache/hadoop/conf/TestConfigurationDeprecation.java
          • /hadoop/common/trunk/src/test/test-fake-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #616 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/616/ ) HADOOP-7287 . Configuration deprecation mechanism doesn't work properly for GenericOptionsParser and Tools. Contributed by Aaron T. Myers. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1126719 Files : /hadoop/common/trunk/CHANGES.txt /hadoop/common/trunk/src/java/org/apache/hadoop/conf/Configuration.java /hadoop/common/trunk/src/test/core/org/apache/hadoop/conf/TestConfigurationDeprecation.java /hadoop/common/trunk/src/test/test-fake-default.xml
          Hide
          Todd Lipcon added a comment -

          Committed to trunk and branch 22. Thanks, Aaron!

          Show
          Todd Lipcon added a comment - Committed to trunk and branch 22. Thanks, Aaron!
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12479976/hadoop-7287-trunk.2.patch
          against trunk revision 1125221.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/496//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/496//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/496//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479976/hadoop-7287-trunk.2.patch against trunk revision 1125221. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/496//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/496//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/496//console This message is automatically generated.
          Hide
          Aaron T. Myers added a comment -

          Updated patch which prints a warning message when the deprecated key is accessed.

          Show
          Aaron T. Myers added a comment - Updated patch which prints a warning message when the deprecated key is accessed.
          Hide
          Todd Lipcon added a comment -

          With HDFS-1932 committed the above manual test now passes.

          But, one more issue I think before we can commit this: in this test case it doesn't print any warning about use of the deprecated flag. Would that be easy to address or is it better to do in another jira?

          Show
          Todd Lipcon added a comment - With HDFS-1932 committed the above manual test now passes. But, one more issue I think before we can commit this: in this test case it doesn't print any warning about use of the deprecated flag. Would that be easy to address or is it better to do in another jira?
          Hide
          Todd Lipcon added a comment -

          I tried the manual test case described above in this JIRA – using ./bin/hadoop fs -Ddfs.blocksize=4096 -touchz f against a local HDFS cluster. The config does not appear to take effect. So, apparently the test cases above are not good enough representations of what really happens for this use case.

          Show
          Todd Lipcon added a comment - I tried the manual test case described above in this JIRA – using ./bin/hadoop fs -Ddfs.blocksize=4096 -touchz f against a local HDFS cluster. The config does not appear to take effect. So, apparently the test cases above are not good enough representations of what really happens for this use case.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12479736/hadoop-7287-trunk.1.patch
          against trunk revision 1125051.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/484//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/484//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/484//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479736/hadoop-7287-trunk.1.patch against trunk revision 1125051. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/484//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/484//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/484//console This message is automatically generated.
          Hide
          Aaron T. Myers added a comment -

          Just finished running the full test suite. All of the M/R tests passed. The following HDFS tests failed, all of which are known to be failing or flaky.

          org.apache.hadoop.tools.TestJMXGet
          org.apache.hadoop.hdfs.TestFileConcurrentReader
          org.apache.hadoop.hdfs.TestDFSStorageStateRecovery

          Show
          Aaron T. Myers added a comment - Just finished running the full test suite. All of the M/R tests passed. The following HDFS tests failed, all of which are known to be failing or flaky. org.apache.hadoop.tools.TestJMXGet org.apache.hadoop.hdfs.TestFileConcurrentReader org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
          Hide
          Aaron T. Myers added a comment -

          Great catch, Todd, and thanks for providing the test case. I've attached an updated patch which addresses this issue.

          Show
          Aaron T. Myers added a comment - Great catch, Todd, and thanks for providing the test case. I've attached an updated patch which addresses this issue.
          Hide
          Todd Lipcon added a comment -

          Though this fixes the test case, it doesn't deal with the case where the "new" key is being set as a default in one of the -default.xml files.

          This patch includes another test case that shows the issue.

          Show
          Todd Lipcon added a comment - Though this fixes the test case, it doesn't deal with the case where the "new" key is being set as a default in one of the -default.xml files. This patch includes another test case that shows the issue.
          Hide
          Aaron T. Myers added a comment -

          All of the M/R core tests passed.

          Show
          Aaron T. Myers added a comment - All of the M/R core tests passed.
          Hide
          Aaron T. Myers added a comment -

          HDFS tests which failed:

          org.apache.hadoop.tools.TestJMXGet
          org.apache.hadoop.hdfs.server.namenode.TestBlocksWithNotEnoughRacks
          org.apache.hadoop.hdfs.TestDecommission
          org.apache.hadoop.hdfs.TestDatanodeBlockScanner
          org.apache.hadoop.hdfs.TestHDFSTrash

          All of which, I believe, are known to be flaky or failing. I was able to run TestBlocksWithNotEnoughRacks, TestDecommission, and TestDatanodeBlockScanner individually, and they passed.

          Running the M/R tests now.

          Show
          Aaron T. Myers added a comment - HDFS tests which failed: org.apache.hadoop.tools.TestJMXGet org.apache.hadoop.hdfs.server.namenode.TestBlocksWithNotEnoughRacks org.apache.hadoop.hdfs.TestDecommission org.apache.hadoop.hdfs.TestDatanodeBlockScanner org.apache.hadoop.hdfs.TestHDFSTrash All of which, I believe, are known to be flaky or failing. I was able to run TestBlocksWithNotEnoughRacks, TestDecommission, and TestDatanodeBlockScanner individually, and they passed. Running the M/R tests now.
          Hide
          Aaron T. Myers added a comment -

          I think since this is such a central part of the code, we should make sure to run the HDFS and MR tests against the patched Common jar. Would you mind doing that and posting results?

          Happy to. This is presently underway.

          Show
          Aaron T. Myers added a comment - I think since this is such a central part of the code, we should make sure to run the HDFS and MR tests against the patched Common jar. Would you mind doing that and posting results? Happy to. This is presently underway.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12479630/hadoop-7287-trunk.0.patch
          against trunk revision 1124368.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/470//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/470//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/470//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479630/hadoop-7287-trunk.0.patch against trunk revision 1124368. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/470//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/470//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/470//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          This seems fairly reasonable. I think since this is such a central part of the code, we should make sure to run the HDFS and MR tests against the patched Common jar. Would you mind doing that and posting results?

          Show
          Todd Lipcon added a comment - This seems fairly reasonable. I think since this is such a central part of the code, we should make sure to run the HDFS and MR tests against the patched Common jar. Would you mind doing that and posting results?
          Hide
          Aaron T. Myers added a comment -

          Patch for trunk which addresses the issue.

          Show
          Aaron T. Myers added a comment - Patch for trunk which addresses the issue.
          Hide
          Todd Lipcon added a comment -

          Here's a test case which illustrates why this problem happens. You can easily reproduce from the command line by trying something like:

          ./bin/hadoop fs -Ddfs.block.size=4096 -touchz f1
          ./bin/hadoop fs -stat "%o" f1

          and you'll see that the deprecated option was ignored and there was no warning.

          Show
          Todd Lipcon added a comment - Here's a test case which illustrates why this problem happens. You can easily reproduce from the command line by trying something like: ./bin/hadoop fs -Ddfs.block.size=4096 -touchz f1 ./bin/hadoop fs -stat "%o" f1 and you'll see that the deprecated option was ignored and there was no warning.

            People

            • Assignee:
              Aaron T. Myers
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development