Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.13.0
    • Fix Version/s: 0.18.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      Hide
      Withdrew the upgrade-to-CRC facility. HDFS will no longer support upgrades from versions without CRCs for block data. Users upgrading from version 0.13 or earlier must first upgrade to an intermediate (0.14, 0.15, 0.16, 0.17) version before doing upgrade to version 0.18 or later.
      Show
      Withdrew the upgrade-to-CRC facility. HDFS will no longer support upgrades from versions without CRCs for block data. Users upgrading from version 0.13 or earlier must first upgrade to an intermediate (0.14, 0.15, 0.16, 0.17) version before doing upgrade to version 0.18 or later.

      Description

      HDFS will no longer support upgrades from versions without CRCs for block data. Users upgrading from version 0.13 or earlier must first upgrade to an intermediate (0.14, 0.15, 0.16, 0.17) version before upgrade to 0.18 or later.

      1. HADOOP-2797.patch
        95 kB
        Raghu Angadi
      2. HADOOP-2797.patch
        94 kB
        Raghu Angadi
      3. HADOOP-2797.patch
        99 kB
        Raghu Angadi
      4. HADOOP-2797.patch
        106 kB
        Raghu Angadi
      5. hadoop-14-dfs-dir.tgz
        149 kB
        Raghu Angadi

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          15h 59m 1 Raghu Angadi 29/Apr/08 15:49
          Open Open Patch Available Patch Available
          81d 2h 2 Raghu Angadi 29/Apr/08 15:49
          Patch Available Patch Available Resolved Resolved
          9h 43m 1 Raghu Angadi 30/Apr/08 01:32
          Resolved Resolved Closed Closed
          114d 18h 17m 1 Nigel Daley 22/Aug/08 19:50
          Gavin made changes -
          Link This issue is depended upon by HDFS-282 [ HDFS-282 ]
          Gavin made changes -
          Link This issue blocks HDFS-282 [ HDFS-282 ]
          Gavin made changes -
          Link This issue is depended upon by MAPREDUCE-557 [ MAPREDUCE-557 ]
          Gavin made changes -
          Link This issue blocks MAPREDUCE-557 [ MAPREDUCE-557 ]
          Hide
          Tanping Wang added a comment -

          The test file images used , hadoop-14-dfs-dir.tgz as for now are generated and uploaded only. Based on the comments in hadoop-1629, these FSimages suppose to contain the various categories

          • zero length files
          • file with replication set higher than number of datanodes
          • file with no .crc file
          • file with corrupt .crc file
          • file with multiple blocks (will need to set dfs.block.size to a small value)
          • file with multiple checksum blocks
          • empty directory
          • all of the above again but with a different io.bytes.per.checksum setting

          As FSImage structure could be changed in future releases, developers need to generate new FS images. There should be at least test program that can generate these FSImages contained in version control. Otherwise, this test is not very sustainable.

          Show
          Tanping Wang added a comment - The test file images used , hadoop-14-dfs-dir.tgz as for now are generated and uploaded only. Based on the comments in hadoop-1629, these FSimages suppose to contain the various categories zero length files file with replication set higher than number of datanodes file with no .crc file file with corrupt .crc file file with multiple blocks (will need to set dfs.block.size to a small value) file with multiple checksum blocks empty directory all of the above again but with a different io.bytes.per.checksum setting As FSImage structure could be changed in future releases, developers need to generate new FS images. There should be at least test program that can generate these FSImages contained in version control. Otherwise, this test is not very sustainable.
          Owen O'Malley made changes -
          Component/s dfs [ 12310710 ]
          Nigel Daley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Robert Chansler made changes -
          Release Note HDFS will no longer support upgrades from versions without CRCs for block data. Users upgrading from version 0.13 or earlier must first upgrade to an intermediate (0.14, 0.15, 0.16, 0.17) version before upgrade to 0.18 or later. Withdrew the upgrade-to-CRC facility. HDFS will no longer support upgrades from versions without CRCs for block data. Users upgrading from version 0.13 or earlier must first upgrade to an intermediate (0.14, 0.15, 0.16, 0.17) version before doing upgrade to version 0.18 or later.
          Konstantin Shvachko made changes -
          Link This issue relates to HADOOP-3573 [ HADOOP-3573 ]
          Nigel Daley made changes -
          Fix Version/s 0.18.0 [ 12312972 ]
          Hide
          Hudson added a comment -
          Show
          Hudson added a comment - Integrated in Hadoop-trunk #475 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/475/ )
          Raghu Angadi made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Raghu Angadi made changes -
          Release Note HDFS will no longer support upgrades from versions without CRCs for block data. Users upgrading from version 0.13 or earlier must first upgrade to an intermediate (0.14, 0.15, 0.16, 0.17) version before upgrade to 0.18 or later.
          Hide
          Raghu Angadi added a comment -

          I just committed this.

          Show
          Raghu Angadi added a comment - I just committed this.
          Raghu Angadi made changes -
          Attachment hadoop-14-dfs-dir.tgz [ 12381158 ]
          Hide
          Raghu Angadi added a comment -

          hadoop-14-dfs-dir.tgz will replace hadoop-12-dfs-dir.tgz in the trunk.

          Show
          Raghu Angadi added a comment - hadoop-14-dfs-dir.tgz will replace hadoop-12-dfs-dir.tgz in the trunk.
          Raghu Angadi made changes -
          Attachment HADOOP-2797.patch [ 12381156 ]
          Hide
          Raghu Angadi added a comment -

          More complete patch (but won't work with hudson). FindBugs warning is also fixed. I will commit this with required svn changes.

          Show
          Raghu Angadi added a comment - More complete patch (but won't work with hudson). FindBugs warning is also fixed. I will commit this with required svn changes.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12381061/HADOOP-2797.patch
          against trunk revision 645773.

          @author +1. The patch does not contain any @author tags.

          tests included +1. The patch appears to include 10 new or modified tests.

          javadoc +1. The javadoc tool did not generate any warning messages.

          javac +1. The applied patch does not generate any new javac compiler warnings.

          release audit +1. The applied patch does not generate any new release audit warnings.

          findbugs -1. The patch appears to introduce 1 new Findbugs warnings.

          core tests -1. The patch failed core unit tests.

          contrib tests +1. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2338/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2338/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2338/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2338/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12381061/HADOOP-2797.patch against trunk revision 645773. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 10 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs -1. The patch appears to introduce 1 new Findbugs warnings. core tests -1. The patch failed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2338/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2338/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2338/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2338/console This message is automatically generated.
          Raghu Angadi made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Raghu Angadi made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Nigel Daley made changes -
          Description HDFS will no longer support upgrades from versions without CRCs for block data. Users upgrading from version 0.13 or earlier must first upgrade to an intermediate (0.14, 0.15, 0.16) version before upgrade to 0.17 or later. HDFS will no longer support upgrades from versions without CRCs for block data. Users upgrading from version 0.13 or earlier must first upgrade to an intermediate (0.14, 0.15, 0.16, 0.17) version before upgrade to 0.18 or later.
          Hide
          Raghu Angadi added a comment -

          > All the Hudson tests except TestDFSUpgradeFromImage should pass.
          One more test 'TestDistributedUpgrade' will fail too since it depends on the same DFS image.

          Show
          Raghu Angadi added a comment - > All the Hudson tests except TestDFSUpgradeFromImage should pass. One more test 'TestDistributedUpgrade' will fail too since it depends on the same DFS image.
          Raghu Angadi made changes -
          Hadoop Flags [Incompatible change]
          Status Open [ 1 ] Patch Available [ 10002 ]
          Raghu Angadi made changes -
          Attachment HADOOP-2797.patch [ 12381061 ]
          Hide
          Raghu Angadi added a comment -

          Updated patch for Hudson. This has all the changes except for the changes to hadoop-12-dfs-dir.txt and hadoop-12-dfs-dir.tgz required by TestDFSUpgradeFromImage. All the Hudson tests except TestDFSUpgradeFromImage should pass.

          > One option would be to keep the existing test and ensure that it fails. This will test that if anybody tries to upgrade an old version, then the upgrade will fail.

          In stead a new test case is added to TestDFSUpgrade.java that tests upgrade from older version.

          Once Hudson passes (except one test), the following extra steps are required to commit :

          • rename hadoop-12-dfs-dir.txt to hadoop-dfs-dir.txt
          • delete hadoop-12-dfs-dir.tgz and add hadoop-dfs-dir.tgz (to be attached).
          • apply the updated patch.
          Show
          Raghu Angadi added a comment - Updated patch for Hudson. This has all the changes except for the changes to hadoop-12-dfs-dir.txt and hadoop-12-dfs-dir.tgz required by TestDFSUpgradeFromImage. All the Hudson tests except TestDFSUpgradeFromImage should pass. > One option would be to keep the existing test and ensure that it fails. This will test that if anybody tries to upgrade an old version, then the upgrade will fail. In stead a new test case is added to TestDFSUpgrade.java that tests upgrade from older version. Once Hudson passes (except one test), the following extra steps are required to commit : rename hadoop-12-dfs-dir.txt to hadoop-dfs-dir.txt delete hadoop-12-dfs-dir.tgz and add hadoop-dfs-dir.tgz (to be attached). apply the updated patch.
          Hide
          Raghu Angadi added a comment -

          Yes. It is true.. probably 2-3 checks.

          Show
          Raghu Angadi added a comment - Yes. It is true.. probably 2-3 checks.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Is it true that some codes in FSImage.loadFSImage(File) can also be removed?

          Show
          Tsz Wo Nicholas Sze added a comment - Is it true that some codes in FSImage.loadFSImage(File) can also be removed?
          Hide
          dhruba borthakur added a comment -

          +1. Code looks good.

          One option would be to keep the existing test and ensure that it fails. This will test that if anybody tries to upgrade an old version, then the upgrade will fail.

          Show
          dhruba borthakur added a comment - +1. Code looks good. One option would be to keep the existing test and ensure that it fails . This will test that if anybody tries to upgrade an old version, then the upgrade will fail.
          Raghu Angadi made changes -
          Attachment HADOOP-2797.patch [ 12381045 ]
          Hide
          Raghu Angadi added a comment -

          Updated patch for trunk.

          Show
          Raghu Angadi added a comment - Updated patch for trunk.
          Tsz Wo Nicholas Sze made changes -
          Link This issue blocks HADOOP-3316 [ HADOOP-3316 ]
          Raghu Angadi made changes -
          Link This issue blocks HADOOP-2885 [ HADOOP-2885 ]
          Robert Chansler made changes -
          Fix Version/s 0.17.0 [ 12312913 ]
          Raghu Angadi made changes -
          Attachment HADOOP-2797.patch [ 12376162 ]
          Hide
          Raghu Angadi added a comment -

          Suggested patch is attached. Note that that this is the first time Hadoop has a notion of versions that are not supported for upgraded. So there are a few non-trivial changes in "version upgrade" layer..

          This does everything except changing hadoop-12-dfs-dir.tgz. This image will be upgraded to hadoop-14-dfs-dir.tgz. So TestDFSUpgradeFromImage fails.

          Show
          Raghu Angadi added a comment - Suggested patch is attached. Note that that this is the first time Hadoop has a notion of versions that are not supported for upgraded. So there are a few non-trivial changes in "version upgrade" layer.. This does everything except changing hadoop-12-dfs-dir.tgz . This image will be upgraded to hadoop-14-dfs-dir.tgz . So TestDFSUpgradeFromImage fails.
          Hide
          Raghu Angadi added a comment -

          This is more than just removing BlockLevelCrcs. Hadoop needs a 'LAST_UPGRADABLE_LAYOUT_VERSION' which is considered early during "version upgrade" (as opposed to distributed upgrade) because when user tries to upgrade from very old version, NameNode and DataNode should exit before making any changes to the existing diretories.

          Show
          Raghu Angadi added a comment - This is more than just removing BlockLevelCrcs. Hadoop needs a 'LAST_UPGRADABLE_LAYOUT_VERSION' which is considered early during "version upgrade" (as opposed to distributed upgrade) because when user tries to upgrade from very old version, NameNode and DataNode should exit before making any changes to the existing diretories.
          Raghu Angadi made changes -
          Field Original Value New Value
          Assignee Raghu Angadi [ rangadi ]
          Robert Chansler created issue -

            People

            • Assignee:
              Raghu Angadi
              Reporter:
              Robert Chansler
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development