Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-671

Documentation change for updated configuration keys.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HDFS-531, HADOOP-6233 and HDFS-631 have resulted in changes in several config keys. The hadoop documentation needs to be updated to reflect those changes.

        Issue Links

          Activity

          Hide
          Nigel Daley added a comment -

          Seems like a blocker for 0.22.

          Show
          Nigel Daley added a comment - Seems like a blocker for 0.22.
          Hide
          Tom White added a comment -

          Here's a patch which fixes the deprecated configuration keys in the forrest docs.

          Show
          Tom White added a comment - Here's a patch which fixes the deprecated configuration keys in the forrest docs.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12477462/HDFS-671.patch
          against trunk revision 1096974.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.hdfs.server.namenode.TestNameNodeResourceChecker
          org.apache.hadoop.hdfs.TestFileConcurrentReader

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/419//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/419//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/419//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12477462/HDFS-671.patch against trunk revision 1096974. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.server.namenode.TestNameNodeResourceChecker org.apache.hadoop.hdfs.TestFileConcurrentReader +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/419//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/419//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/419//console This message is automatically generated.
          Hide
          Eli Collins added a comment -

          This needs to go in trunk as well right? The docs on trunk still refer to fs.default.name rather than fs.defaultFS.

          CommonConfigurationKeysPublic.java defines FS_DEFAULT_NAME_KEY as "fs.defaultFS" and FsConfig.java defines FS_DEFAULT_NAME_KEY = "fs.default.name". Why are both needed?

          Show
          Eli Collins added a comment - This needs to go in trunk as well right? The docs on trunk still refer to fs.default.name rather than fs.defaultFS. CommonConfigurationKeysPublic.java defines FS_DEFAULT_NAME_KEY as "fs.defaultFS" and FsConfig.java defines FS_DEFAULT_NAME_KEY = "fs.default.name". Why are both needed?
          Hide
          Tom White added a comment -

          > This needs to go in trunk as well right?

          Correct.

          > CommonConfigurationKeysPublic.java defines FS_DEFAULT_NAME_KEY as "fs.defaultFS" and FsConfig.java defines FS_DEFAULT_NAME_KEY = "fs.default.name". Why are both needed?

          This looks like an inconsistency. FsConfig is used by FileContext, and looks like its keys need updating to be consistent with (or use) CommonConfigurationKeysPublic. This can be tackled in another JIRA.

          Show
          Tom White added a comment - > This needs to go in trunk as well right? Correct. > CommonConfigurationKeysPublic.java defines FS_DEFAULT_NAME_KEY as "fs.defaultFS" and FsConfig.java defines FS_DEFAULT_NAME_KEY = "fs.default.name". Why are both needed? This looks like an inconsistency. FsConfig is used by FileContext, and looks like its keys need updating to be consistent with (or use) CommonConfigurationKeysPublic. This can be tackled in another JIRA.
          Hide
          Eli Collins added a comment -

          This looks like an inconsistency. FsConfig is used by FileContext, and looks like its keys need updating to be consistent with (or use) CommonConfigurationKeysPublic. This can be tackled in another JIRA.

          Doesn't that impact this change? Eg I'm not sure we should be switching the docs to use defaultFS.

          Show
          Eli Collins added a comment - This looks like an inconsistency. FsConfig is used by FileContext, and looks like its keys need updating to be consistent with (or use) CommonConfigurationKeysPublic. This can be tackled in another JIRA. Doesn't that impact this change? Eg I'm not sure we should be switching the docs to use defaultFS.
          Hide
          Tom White added a comment -

          "fs.default.name" has been deprecated in favour of "fs.defaultFS" and I checked that FileContext works with "fs.defaultFS". So I think that this change and the one in HADOOP-7244 should stand. I've opened HADOOP-7245 to address moving the keys used by FsConfig to be the new ones.

          Show
          Tom White added a comment - "fs.default.name" has been deprecated in favour of "fs.defaultFS" and I checked that FileContext works with "fs.defaultFS". So I think that this change and the one in HADOOP-7244 should stand. I've opened HADOOP-7245 to address moving the keys used by FsConfig to be the new ones.
          Hide
          Eli Collins added a comment -

          Makes sense. I reviewed those jiras, and filed HDFS-1900 since the difference in block size keys names across common and HDFS doesn't make sense to me.

          +1 this patch lgtm.

          Show
          Eli Collins added a comment - Makes sense. I reviewed those jiras, and filed HDFS-1900 since the difference in block size keys names across common and HDFS doesn't make sense to me. +1 this patch lgtm.
          Hide
          Eli Collins added a comment -

          I've committed this to trunk and branch 22. Thanks Tom!

          Show
          Eli Collins added a comment - I've committed this to trunk and branch 22. Thanks Tom!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-22-branch #41 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-22-branch/41/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-22-branch #41 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-22-branch/41/ )
          Hide
          Dmitriy V. Ryaboy added a comment -

          Guys,

          While working on getting Pig to play with 0.22, I was getting 35 errors for one of the test cases (TestGrunt). In the process of debugging, I noticed a lot of deprecation warnings concerning "fs.default.name" and changed the references across the Pig codebase to use fs.defaultFS, just to silence the noise. No other code changes were made between compilations. Suddenly the errors dropped to 13.

          I believe the cause is that Pig plays pretty loose with switching between Conf and using Properties directly, and was using fs.default.name all over the place. Seems like the deprecation warning and docs should user stronger language – using the old string anywhere but in the xml file is likely to cause problems, as illustrated by this.

          Show
          Dmitriy V. Ryaboy added a comment - Guys, While working on getting Pig to play with 0.22, I was getting 35 errors for one of the test cases (TestGrunt). In the process of debugging, I noticed a lot of deprecation warnings concerning "fs.default.name" and changed the references across the Pig codebase to use fs.defaultFS, just to silence the noise. No other code changes were made between compilations. Suddenly the errors dropped to 13. I believe the cause is that Pig plays pretty loose with switching between Conf and using Properties directly, and was using fs.default.name all over the place. Seems like the deprecation warning and docs should user stronger language – using the old string anywhere but in the xml file is likely to cause problems, as illustrated by this.
          Hide
          Eli Collins added a comment -

          @Dmitriy - have you seen HADOOP-7287? Perhaps the deprecated name was being set in such a way that was causing it to be ignored?

          Show
          Eli Collins added a comment - @Dmitriy - have you seen HADOOP-7287 ? Perhaps the deprecated name was being set in such a way that was causing it to be ignored?
          Hide
          Dmitriy V. Ryaboy added a comment -

          Eli, looks likely to be the case – though I did see quite a few warnings, too, so there might be a mix of both going on.

          Show
          Dmitriy V. Ryaboy added a comment - Eli, looks likely to be the case – though I did see quite a few warnings, too, so there might be a mix of both going on.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #673 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/673/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #673 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/673/ )

            People

            • Assignee:
              Tom White
              Reporter:
              Jitendra Nath Pandey
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development