Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.13.0
    • Fix Version/s: 0.18.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      Hide
      Withdrew the upgrade-to-CRC facility. HDFS will no longer support upgrades from versions without CRCs for block data. Users upgrading from version 0.13 or earlier must first upgrade to an intermediate (0.14, 0.15, 0.16, 0.17) version before doing upgrade to version 0.18 or later.
      Show
      Withdrew the upgrade-to-CRC facility. HDFS will no longer support upgrades from versions without CRCs for block data. Users upgrading from version 0.13 or earlier must first upgrade to an intermediate (0.14, 0.15, 0.16, 0.17) version before doing upgrade to version 0.18 or later.

      Description

      HDFS will no longer support upgrades from versions without CRCs for block data. Users upgrading from version 0.13 or earlier must first upgrade to an intermediate (0.14, 0.15, 0.16, 0.17) version before upgrade to 0.18 or later.

      1. HADOOP-2797.patch
        95 kB
        Raghu Angadi
      2. HADOOP-2797.patch
        94 kB
        Raghu Angadi
      3. HADOOP-2797.patch
        99 kB
        Raghu Angadi
      4. HADOOP-2797.patch
        106 kB
        Raghu Angadi
      5. hadoop-14-dfs-dir.tgz
        149 kB
        Raghu Angadi

        Issue Links

          Activity

          Hide
          Raghu Angadi added a comment -

          This is more than just removing BlockLevelCrcs. Hadoop needs a 'LAST_UPGRADABLE_LAYOUT_VERSION' which is considered early during "version upgrade" (as opposed to distributed upgrade) because when user tries to upgrade from very old version, NameNode and DataNode should exit before making any changes to the existing diretories.

          Show
          Raghu Angadi added a comment - This is more than just removing BlockLevelCrcs. Hadoop needs a 'LAST_UPGRADABLE_LAYOUT_VERSION' which is considered early during "version upgrade" (as opposed to distributed upgrade) because when user tries to upgrade from very old version, NameNode and DataNode should exit before making any changes to the existing diretories.
          Hide
          Raghu Angadi added a comment -

          Suggested patch is attached. Note that that this is the first time Hadoop has a notion of versions that are not supported for upgraded. So there are a few non-trivial changes in "version upgrade" layer..

          This does everything except changing hadoop-12-dfs-dir.tgz. This image will be upgraded to hadoop-14-dfs-dir.tgz. So TestDFSUpgradeFromImage fails.

          Show
          Raghu Angadi added a comment - Suggested patch is attached. Note that that this is the first time Hadoop has a notion of versions that are not supported for upgraded. So there are a few non-trivial changes in "version upgrade" layer.. This does everything except changing hadoop-12-dfs-dir.tgz . This image will be upgraded to hadoop-14-dfs-dir.tgz . So TestDFSUpgradeFromImage fails.
          Hide
          Raghu Angadi added a comment -

          Updated patch for trunk.

          Show
          Raghu Angadi added a comment - Updated patch for trunk.
          Hide
          dhruba borthakur added a comment -

          +1. Code looks good.

          One option would be to keep the existing test and ensure that it fails. This will test that if anybody tries to upgrade an old version, then the upgrade will fail.

          Show
          dhruba borthakur added a comment - +1. Code looks good. One option would be to keep the existing test and ensure that it fails . This will test that if anybody tries to upgrade an old version, then the upgrade will fail.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Is it true that some codes in FSImage.loadFSImage(File) can also be removed?

          Show
          Tsz Wo Nicholas Sze added a comment - Is it true that some codes in FSImage.loadFSImage(File) can also be removed?
          Hide
          Raghu Angadi added a comment -

          Yes. It is true.. probably 2-3 checks.

          Show
          Raghu Angadi added a comment - Yes. It is true.. probably 2-3 checks.
          Hide
          Raghu Angadi added a comment -

          Updated patch for Hudson. This has all the changes except for the changes to hadoop-12-dfs-dir.txt and hadoop-12-dfs-dir.tgz required by TestDFSUpgradeFromImage. All the Hudson tests except TestDFSUpgradeFromImage should pass.

          > One option would be to keep the existing test and ensure that it fails. This will test that if anybody tries to upgrade an old version, then the upgrade will fail.

          In stead a new test case is added to TestDFSUpgrade.java that tests upgrade from older version.

          Once Hudson passes (except one test), the following extra steps are required to commit :

          • rename hadoop-12-dfs-dir.txt to hadoop-dfs-dir.txt
          • delete hadoop-12-dfs-dir.tgz and add hadoop-dfs-dir.tgz (to be attached).
          • apply the updated patch.
          Show
          Raghu Angadi added a comment - Updated patch for Hudson. This has all the changes except for the changes to hadoop-12-dfs-dir.txt and hadoop-12-dfs-dir.tgz required by TestDFSUpgradeFromImage. All the Hudson tests except TestDFSUpgradeFromImage should pass. > One option would be to keep the existing test and ensure that it fails. This will test that if anybody tries to upgrade an old version, then the upgrade will fail. In stead a new test case is added to TestDFSUpgrade.java that tests upgrade from older version. Once Hudson passes (except one test), the following extra steps are required to commit : rename hadoop-12-dfs-dir.txt to hadoop-dfs-dir.txt delete hadoop-12-dfs-dir.tgz and add hadoop-dfs-dir.tgz (to be attached). apply the updated patch.
          Hide
          Raghu Angadi added a comment -

          > All the Hudson tests except TestDFSUpgradeFromImage should pass.
          One more test 'TestDistributedUpgrade' will fail too since it depends on the same DFS image.

          Show
          Raghu Angadi added a comment - > All the Hudson tests except TestDFSUpgradeFromImage should pass. One more test 'TestDistributedUpgrade' will fail too since it depends on the same DFS image.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12381061/HADOOP-2797.patch
          against trunk revision 645773.

          @author +1. The patch does not contain any @author tags.

          tests included +1. The patch appears to include 10 new or modified tests.

          javadoc +1. The javadoc tool did not generate any warning messages.

          javac +1. The applied patch does not generate any new javac compiler warnings.

          release audit +1. The applied patch does not generate any new release audit warnings.

          findbugs -1. The patch appears to introduce 1 new Findbugs warnings.

          core tests -1. The patch failed core unit tests.

          contrib tests +1. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2338/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2338/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2338/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2338/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12381061/HADOOP-2797.patch against trunk revision 645773. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 10 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs -1. The patch appears to introduce 1 new Findbugs warnings. core tests -1. The patch failed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2338/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2338/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2338/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2338/console This message is automatically generated.
          Hide
          Raghu Angadi added a comment -

          More complete patch (but won't work with hudson). FindBugs warning is also fixed. I will commit this with required svn changes.

          Show
          Raghu Angadi added a comment - More complete patch (but won't work with hudson). FindBugs warning is also fixed. I will commit this with required svn changes.
          Hide
          Raghu Angadi added a comment -

          hadoop-14-dfs-dir.tgz will replace hadoop-12-dfs-dir.tgz in the trunk.

          Show
          Raghu Angadi added a comment - hadoop-14-dfs-dir.tgz will replace hadoop-12-dfs-dir.tgz in the trunk.
          Hide
          Raghu Angadi added a comment -

          I just committed this.

          Show
          Raghu Angadi added a comment - I just committed this.
          Hide
          Hudson added a comment -
          Show
          Hudson added a comment - Integrated in Hadoop-trunk #475 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/475/ )
          Hide
          Tanping Wang added a comment -

          The test file images used , hadoop-14-dfs-dir.tgz as for now are generated and uploaded only. Based on the comments in hadoop-1629, these FSimages suppose to contain the various categories

          • zero length files
          • file with replication set higher than number of datanodes
          • file with no .crc file
          • file with corrupt .crc file
          • file with multiple blocks (will need to set dfs.block.size to a small value)
          • file with multiple checksum blocks
          • empty directory
          • all of the above again but with a different io.bytes.per.checksum setting

          As FSImage structure could be changed in future releases, developers need to generate new FS images. There should be at least test program that can generate these FSImages contained in version control. Otherwise, this test is not very sustainable.

          Show
          Tanping Wang added a comment - The test file images used , hadoop-14-dfs-dir.tgz as for now are generated and uploaded only. Based on the comments in hadoop-1629, these FSimages suppose to contain the various categories zero length files file with replication set higher than number of datanodes file with no .crc file file with corrupt .crc file file with multiple blocks (will need to set dfs.block.size to a small value) file with multiple checksum blocks empty directory all of the above again but with a different io.bytes.per.checksum setting As FSImage structure could be changed in future releases, developers need to generate new FS images. There should be at least test program that can generate these FSImages contained in version control. Otherwise, this test is not very sustainable.

            People

            • Assignee:
              Raghu Angadi
              Reporter:
              Robert Chansler
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development