Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3177

Allow DFSClient to find out and use the CRC type being used for a file.

    Details

    • Hadoop Flags:
      Reviewed

      Description

      To support HADOOP-8060, DFSClient should be able to find out the checksum type being used for files in hdfs.
      In my prototype, DataTransferProtocol was extended to include the checksum type in the blockChecksum() response. DFSClient uses it in getFileChecksum() to determin the checksum type. Also append() can be configured to use the existing checksum type instead of the configured one.

      1. hdfs-3177.patch
        16 kB
        Kihwal Lee
      2. hdfs-3177-after-hadoop-8239.patch.txt
        29 kB
        Kihwal Lee
      3. hdfs-3177-after-hadoop-8239-8240.patch.txt
        27 kB
        Kihwal Lee
      4. hdfs-3177-branch2-trunk.patch.txt
        31 kB
        Kihwal Lee
      5. hdfs-3177-branch2-trunk.patch.txt
        29 kB
        Kihwal Lee
      6. hdfs-3177-branch2-trunk.patch.txt
        28 kB
        Kihwal Lee
      7. hdfs-3177-branch2-trunk.patch.txt
        26 kB
        Kihwal Lee
      8. hdfs-3177-branch2-trunk.patch.txt
        23 kB
        Kihwal Lee
      9. hdfs-3177-branch2-trunk.patch.txt
        23 kB
        Kihwal Lee
      10. hdfs-3177-branch2-trunk.patch.txt
        27 kB
        Kihwal Lee
      11. hdfs-3177-minimum-without-append-concat-branch-0.23.patch.txt
        7 kB
        Kihwal Lee
      12. hdfs-3177-minimum-without-append-concat-trunk-branch2.patch.txt
        6 kB
        Kihwal Lee
      13. hdfs-3177-with-hadoop-8239.patch.txt
        38 kB
        Kihwal Lee
      14. hdfs-3177-with-hadoop-8239-8240.patch.txt
        89 kB
        Kihwal Lee
      15. hdfs-3177-with-hadoop-8239-8240.patch.txt
        90 kB
        Kihwal Lee
      16. hdfs-3177-with-hadoop-8239-8240.patch.txt
        85 kB
        Kihwal Lee

        Issue Links

          Activity

          Hide
          Kihwal Lee added a comment -

          Posting the initial patch for feedback. I will add unit tests and do more testing.

          Show
          Kihwal Lee added a comment - Posting the initial patch for feedback. I will add unit tests and do more testing.
          Hide
          Kihwal Lee added a comment -

          Posting a patch for feedback. I am doing more testing and there will be new test cases.
          So far, I verified that FileSystem.getFileChecksum() works across branch-1 and trunk with this patch via hftp. httpfs should work fine. webhdfs will break without HDFS-3176 after the common portions of HADOOP-8060 are committed.

          Show
          Kihwal Lee added a comment - Posting a patch for feedback. I am doing more testing and there will be new test cases. So far, I verified that FileSystem.getFileChecksum() works across branch-1 and trunk with this patch via hftp. httpfs should work fine. webhdfs will break without HDFS-3176 after the common portions of HADOOP-8060 are committed.
          Hide
          Kihwal Lee added a comment -

          The new patch includes support for configurable mixed checksum mode for append (off by default) and new/modified test cases.

          Show
          Kihwal Lee added a comment - The new patch includes support for configurable mixed checksum mode for append (off by default) and new/modified test cases.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12541213/hdfs-3177-with-hadoop-8239-8240.patch.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:

          org.apache.hadoop.hdfs.TestReplication
          org.apache.hadoop.hdfs.TestFileAppend2
          org.apache.hadoop.hdfs.TestReplaceDatanodeOnFailure

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3023//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3023//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3023//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541213/hdfs-3177-with-hadoop-8239-8240.patch.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: org.apache.hadoop.hdfs.TestReplication org.apache.hadoop.hdfs.TestFileAppend2 org.apache.hadoop.hdfs.TestReplaceDatanodeOnFailure +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3023//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3023//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3023//console This message is automatically generated.
          Hide
          Kihwal Lee added a comment -

          Corrected a bug due to a typo. Corrected a test case to have correct permissions set on files.

          Show
          Kihwal Lee added a comment - Corrected a bug due to a typo. Corrected a test case to have correct permissions set on files.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12541245/hdfs-3177-with-hadoop-8239-8240.patch.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 7 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:

          org.apache.hadoop.hdfs.TestEncryptedTransfer
          org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3028//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3028//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3028//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541245/hdfs-3177-with-hadoop-8239-8240.patch.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: org.apache.hadoop.hdfs.TestEncryptedTransfer org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3028//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3028//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3028//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12541267/hdfs-3177-with-hadoop-8239-8240.patch.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 7 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:

          org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
          org.apache.hadoop.hdfs.TestDatanodeBlockScanner

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3030//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3030//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541267/hdfs-3177-with-hadoop-8239-8240.patch.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks org.apache.hadoop.hdfs.TestDatanodeBlockScanner +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3030//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3030//console This message is automatically generated.
          Hide
          Kihwal Lee added a comment -

          The failed tests run fine on my machine.
          There are open JIRAs for these test failures. I will investigate a bit more and file JIRAs if they were failing differently.

          -------------------------------------------------------
           T E S T S
          -------------------------------------------------------
          Running org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
          Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 59.817 sec
          Running org.apache.hadoop.hdfs.TestDatanodeBlockScanner
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 74.002 sec
          
          Show
          Kihwal Lee added a comment - The failed tests run fine on my machine. There are open JIRAs for these test failures. I will investigate a bit more and file JIRAs if they were failing differently. ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 59.817 sec Running org.apache.hadoop.hdfs.TestDatanodeBlockScanner Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 74.002 sec
          Hide
          Kihwal Lee added a comment -

          The new patch based on the new HADOOP-8239 patch has been uploaded.

          Show
          Kihwal Lee added a comment - The new patch based on the new HADOOP-8239 patch has been uploaded.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12541654/hdfs-3177-with-hadoop-8239.patch.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.ha.TestZKFailoverController
          org.apache.hadoop.hdfs.TestPersistBlocks

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3048//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3048//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541654/hdfs-3177-with-hadoop-8239.patch.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.ha.TestZKFailoverController org.apache.hadoop.hdfs.TestPersistBlocks +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3048//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3048//console This message is automatically generated.
          Hide
          Kihwal Lee added a comment -

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.ha.TestZKFailoverController
          org.apache.hadoop.hdfs.TestPersistBlocks

          These are known issues. HADOOP-8591 and HDFS-3811

          Show
          Kihwal Lee added a comment - -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.ha.TestZKFailoverController org.apache.hadoop.hdfs.TestPersistBlocks These are known issues. HADOOP-8591 and HDFS-3811
          Hide
          Kihwal Lee added a comment -

          The new patch removes the JsonUtil change done in HADOOP-8239 to apply cleanly.

          Show
          Kihwal Lee added a comment - The new patch removes the JsonUtil change done in HADOOP-8239 to apply cleanly.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12541744/hdfs-3177-branch2-trunk.patch.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.namenode.TestBackupNode
          org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3051//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3051//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541744/hdfs-3177-branch2-trunk.patch.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.TestBackupNode org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3051//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3051//console This message is automatically generated.
          Hide
          Kihwal Lee added a comment -
           -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:
          
          org.apache.hadoop.hdfs.server.namenode.TestBackupNode
          org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          

          None of test failures was caused by this patch

          • TestBackupNode: faild due to "Port in use: 0.0.0.0:50105" in the initial cluster startup even before trying restart
          • TestDataNodeVolumeFailureReporting: The test worked okay but org.apache.hadoop.util.ExitUtil$ExitException was thrown during the cluster shutdown. It failed the same way before. (e.g. Aug 04, https://builds.apache.org/job/PreCommit-HADOOP-Build/1248//testReport/)

          I will investigate a bit more and file jiras if needed.

          Show
          Kihwal Lee added a comment - -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.TestBackupNode org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting None of test failures was caused by this patch TestBackupNode: faild due to "Port in use: 0.0.0.0:50105" in the initial cluster startup even before trying restart TestDataNodeVolumeFailureReporting: The test worked okay but org.apache.hadoop.util.ExitUtil$ExitException was thrown during the cluster shutdown. It failed the same way before. (e.g. Aug 04, https://builds.apache.org/job/PreCommit-HADOOP-Build/1248//testReport/ ) I will investigate a bit more and file jiras if needed.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I do not understand the design. Some questions:

          • For append, it makes a lot of sense to keep using the existing checksum type. What is the use case for using a different checksum type?
          • Suppose the last block is half written with CRC32 in a close file. Then, the file is re-opened for append with CRC32C. Would the block has two checksum types, i.e. first half is CRC32 and the second half is CRC32C?
          • Suppose a close file is already using more than one checksum type. Then, the file is re-opened for append with dfs.client.append.allow-different-checksum == false. Which checksum should it use? Or should it fail?
          Show
          Tsz Wo Nicholas Sze added a comment - I do not understand the design. Some questions: For append, it makes a lot of sense to keep using the existing checksum type. What is the use case for using a different checksum type? Suppose the last block is half written with CRC32 in a close file. Then, the file is re-opened for append with CRC32C. Would the block has two checksum types, i.e. first half is CRC32 and the second half is CRC32C? Suppose a close file is already using more than one checksum type. Then, the file is re-opened for append with dfs.client.append.allow-different-checksum == false. Which checksum should it use? Or should it fail?
          Hide
          Kihwal Lee added a comment -

          For append, it makes a lot of sense to keep using the existing checksum type. What is the use case for using a different checksum type?

          I don't think it makes sense either, but that was the design decision made in HDFS-2130. There might have been some use cases for this, so I tried to support it while making the default to not allow it. If you feel that this should be the behavior with no configurable option, I will be happy to update the patch accordingly.

          What do you think we should do for concat()? It is supposed to be quick namenode only operation, so I don't feel comfortable about inserting code to check the checksums of input files.

          Suppose the last block is half written with CRC32 in a close file. Then, the file is re-opened for append with CRC32C. Would the block has two checksum types, i.e. first half is CRC32 and the second half is CRC32C?

          No. Datanode will continue to use the same checksum parameters of the existing partial block for writing, independent of what client is sending with data. Input data integrity check is still done, of course.

          Suppose a close file is already using more than one checksum type. Then, the file is re-opened for append with dfs.client.append.allow-different-checksum == false. Which checksum should it use? Or should it fail?

          I don't think we can do much for existing files. Users can detect it with getFileChecksum(), which will show DataChecksum.Type.MIXED as its checksum type. For these files, checksum will still be used for block -level integrity check and nothing will break until something like distcp tries to compare FileChecksums after copying.

          Show
          Kihwal Lee added a comment - For append, it makes a lot of sense to keep using the existing checksum type. What is the use case for using a different checksum type? I don't think it makes sense either, but that was the design decision made in HDFS-2130 . There might have been some use cases for this, so I tried to support it while making the default to not allow it. If you feel that this should be the behavior with no configurable option, I will be happy to update the patch accordingly. What do you think we should do for concat()? It is supposed to be quick namenode only operation, so I don't feel comfortable about inserting code to check the checksums of input files. Suppose the last block is half written with CRC32 in a close file. Then, the file is re-opened for append with CRC32C. Would the block has two checksum types, i.e. first half is CRC32 and the second half is CRC32C? No. Datanode will continue to use the same checksum parameters of the existing partial block for writing, independent of what client is sending with data. Input data integrity check is still done, of course. Suppose a close file is already using more than one checksum type. Then, the file is re-opened for append with dfs.client.append.allow-different-checksum == false. Which checksum should it use? Or should it fail? I don't think we can do much for existing files. Users can detect it with getFileChecksum(), which will show DataChecksum.Type.MIXED as its checksum type. For these files, checksum will still be used for block -level integrity check and nothing will break until something like distcp tries to compare FileChecksums after copying.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Hi Kihwal, thank you for the answers. Let's disallow append with different checksum types and remove the conf property for the moment. We could add it if we find it useful later.

          concat() is tricky. It seems that we have to allow concat with different checksum types since checksum is invisible to users. Fortunately, this should not a common case: concat is usually used with distcp and all blocks should be created with the same checksum type.

          Show
          Tsz Wo Nicholas Sze added a comment - Hi Kihwal, thank you for the answers. Let's disallow append with different checksum types and remove the conf property for the moment. We could add it if we find it useful later. concat() is tricky. It seems that we have to allow concat with different checksum types since checksum is invisible to users. Fortunately, this should not a common case: concat is usually used with distcp and all blocks should be created with the same checksum type.
          Hide
          Kihwal Lee added a comment -

          Th new patch addresses the above comment. append() now enforces the same checksum as long as the existing target file doesn't already have mixed checksum types.

          User can detect the files with mixed checksum types by checking the result of getFileChecksum(). The checksum type will be set to "MIXED".

          Show
          Kihwal Lee added a comment - Th new patch addresses the above comment. append() now enforces the same checksum as long as the existing target file doesn't already have mixed checksum types. User can detect the files with mixed checksum types by checking the result of getFileChecksum(). The checksum type will be set to "MIXED".
          Hide
          Kihwal Lee added a comment -

          PreCommit-HDFS-Build is stuck. It has been showing to run "TestHASafeMode" for more than 3 hours now. It's not timing out. I think someone needs to bounce it.

          So I ran test patch on my own and all looked fine.

          Show
          Kihwal Lee added a comment - PreCommit-HDFS-Build is stuck. It has been showing to run "TestHASafeMode" for more than 3 hours now. It's not timing out. I think someone needs to bounce it. So I ran test patch on my own and all looked fine.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12542065/hdfs-3177-branch2-trunk.patch.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestPersistBlocks

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3074//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3074//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3074//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542065/hdfs-3177-branch2-trunk.patch.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestPersistBlocks +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3074//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3074//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3074//console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -
          • The patch adds invoking callGetBlockLocations(..) in append. As a result, append additionally requires read permission. I think it is an unacceptable incompatible change. We need to think about this carefully. I suggest work on the append change in a separated JIRA.
          • For the changes in getFileChecksum(..), most of the changes is for refactoring callBlockChecksum(..). If we remove the append change here, let's also defer the refactoring so that it is easier to review. BTW, there is an unused "retry" variable in callBlockChecksum(..).
          Show
          Tsz Wo Nicholas Sze added a comment - The patch adds invoking callGetBlockLocations(..) in append. As a result, append additionally requires read permission. I think it is an unacceptable incompatible change. We need to think about this carefully. I suggest work on the append change in a separated JIRA. For the changes in getFileChecksum(..), most of the changes is for refactoring callBlockChecksum(..). If we remove the append change here, let's also defer the refactoring so that it is easier to review. BTW, there is an unused "retry" variable in callBlockChecksum(..).
          Hide
          Kihwal Lee added a comment -

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

          They are not caused by this patch. Previous builds have the indentical warnings. HDFS-3835 mentions another jora fixing one of the warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:
          org.apache.hadoop.hdfs.TestPersistBlocks

          Not caused by this patch. There already is a jira for this: HDFS-3811

          Show
          Kihwal Lee added a comment - -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. They are not caused by this patch. Previous builds have the indentical warnings. HDFS-3835 mentions another jora fixing one of the warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestPersistBlocks Not caused by this patch. There already is a jira for this: HDFS-3811
          Hide
          Kihwal Lee added a comment -

          append additionally requires read permission. I think it is an unacceptable incompatible change.

          How about allowing getBlockLocations() for both read and write? The block tokens will contain permission (in mode) so the permission won't be violated on DN. I would rather get it done in this jira.

          Show
          Kihwal Lee added a comment - append additionally requires read permission. I think it is an unacceptable incompatible change. How about allowing getBlockLocations() for both read and write? The block tokens will contain permission (in mode) so the permission won't be violated on DN. I would rather get it done in this jira.
          Hide
          Kihwal Lee added a comment -
          • changed getBlockLocations() so it can be served with either read or write permission.
          • reverted TestFileAppend2 and it passes. (append on the file with permission 0200)
          • got rid of the unused variable.
          Show
          Kihwal Lee added a comment - changed getBlockLocations() so it can be served with either read or write permission. reverted TestFileAppend2 and it passes. (append on the file with permission 0200) got rid of the unused variable.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Hi Kihwal, ClientProtocol.append(..) currently returns the last block locations. Could we use it instead of getting the first block?

          Show
          Tsz Wo Nicholas Sze added a comment - Hi Kihwal, ClientProtocol.append(..) currently returns the last block locations. Could we use it instead of getting the first block?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12542113/hdfs-3177-branch2-trunk.patch.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestDFSPermission

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3075//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3075//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3075//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542113/hdfs-3177-branch2-trunk.patch.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestDFSPermission +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3075//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3075//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3075//console This message is automatically generated.
          Hide
          Kihwal Lee added a comment -

          Hi Kihwal, ClientProtocol.append(..) currently returns the last block locations. Could we use it instead of getting the first block?

          No. It can return an empty list, if the end of file coincides with the block boundary.

          Show
          Kihwal Lee added a comment - Hi Kihwal, ClientProtocol.append(..) currently returns the last block locations. Could we use it instead of getting the first block? No. It can return an empty list, if the end of file coincides with the block boundary.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          How about we change append to return the last block anyway and use DFSClient to detect if it is a full block?

          Show
          Tsz Wo Nicholas Sze added a comment - How about we change append to return the last block anyway and use DFSClient to detect if it is a full block?
          Hide
          Kihwal Lee added a comment -

          That can work too. I will update the patch in a moment.

          Show
          Kihwal Lee added a comment - That can work too. I will update the patch in a moment.
          Hide
          Kihwal Lee added a comment -

          Main changes:

          • Updated BlockManager#convertLastBlockToUnderConstruction() to return the last block even if it's full, but won't turn that to under construction. Verified that nothing else is depending on the old behavior.
          • Updated DFSOutputStream so that new DataStreamer() is called to add a new block if {lastBlock}

            is full.

          Show
          Kihwal Lee added a comment - Main changes: Updated BlockManager#convertLastBlockToUnderConstruction() to return the last block even if it's full, but won't turn that to under construction. Verified that nothing else is depending on the old behavior. Updated DFSOutputStream so that new DataStreamer() is called to add a new block if {lastBlock} is full.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12542155/hdfs-3177-branch2-trunk.patch.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestEncryptedTransfer
          org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3077//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3077//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3077//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542155/hdfs-3177-branch2-trunk.patch.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestEncryptedTransfer org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3077//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3077//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3077//console This message is automatically generated.
          Hide
          Kihwal Lee added a comment -

          Fixed the cause of the test breakages. blockChecksum() denied access when the token is for write. I think reading block checksum can be allowed, since it does not reveal the data content.

          Show
          Kihwal Lee added a comment - Fixed the cause of the test breakages. blockChecksum() denied access when the token is for write. I think reading block checksum can be allowed, since it does not reveal the data content.
          Hide
          Sanjay Radia added a comment -

          Mixed checksums in a should not be allowed.
          If we don't allow a different checksum for append then mixed checksums can occur only due to concat. Is the checksum type also stored on NN? If so we can prevent the concact of such files.

          Show
          Sanjay Radia added a comment - Mixed checksums in a should not be allowed. If we don't allow a different checksum for append then mixed checksums can occur only due to concat. Is the checksum type also stored on NN? If so we can prevent the concact of such files.
          Hide
          Sanjay Radia added a comment -

          typo: Mixed checksums in a /file/ should not be allowed.

          Show
          Sanjay Radia added a comment - typo: Mixed checksums in a /file/ should not be allowed.
          Hide
          Kihwal Lee added a comment -

          checksum type is only stored with blocks on dn. The only way to find out the type is to access the block.

          Show
          Kihwal Lee added a comment - checksum type is only stored with blocks on dn. The only way to find out the type is to access the block.
          Hide
          Kihwal Lee added a comment -

          I think consistent checksum in concat() can be supported by inserting checks. It hits datanodes, but won't be too bad since it's reading only checksum file and sending MD5 of it.

          I tested this and it passes TestHDFSConcat.

            public void concat(String trg, String [] srcs) throws IOException {
              checkOpen();
              try {
          +      // check the checksum consistency
          +      MD5MD5CRC32FileChecksum csum = null;
          +      String src = "";
          +      for (String s : srcs) {
          +        MD5MD5CRC32FileChecksum csumToCompare = getFileChecksum(s);
          +        if (csumToCompare.getChecksumOpt().getChecksumType() ==
          +            DataChecksum.Type.MIXED) {
          +          throw new IOException("Mixed checksum type detected in " +
          +              s + ". This is not supported in concat()");
          +        }
          +        if (csum == null) {
          +          csum = csumToCompare;
          +          src = s;
          +          continue;
          +        }
          +        if (csum.getChecksumOpt().getChecksumType() !=
          +            csumToCompare.getChecksumOpt().getChecksumType()) {
          +          throw new IOException("Checksum types are different between " + s
          +              + " and  " + src);
          +        }
          +      }
                namenode.concat(trg, srcs);
              } catch(RemoteException re) {
                throw re.unwrapRemoteException(AccessControlException.class,
                                               UnresolvedPathException.class);
              }
            }
          
          Show
          Kihwal Lee added a comment - I think consistent checksum in concat() can be supported by inserting checks. It hits datanodes, but won't be too bad since it's reading only checksum file and sending MD5 of it. I tested this and it passes TestHDFSConcat. public void concat( String trg, String [] srcs) throws IOException { checkOpen(); try { + // check the checksum consistency + MD5MD5CRC32FileChecksum csum = null ; + String src = ""; + for ( String s : srcs) { + MD5MD5CRC32FileChecksum csumToCompare = getFileChecksum(s); + if (csumToCompare.getChecksumOpt().getChecksumType() == + DataChecksum.Type.MIXED) { + throw new IOException( "Mixed checksum type detected in " + + s + ". This is not supported in concat()" ); + } + if (csum == null ) { + csum = csumToCompare; + src = s; + continue ; + } + if (csum.getChecksumOpt().getChecksumType() != + csumToCompare.getChecksumOpt().getChecksumType()) { + throw new IOException( "Checksum types are different between " + s + + " and " + src); + } + } namenode.concat(trg, srcs); } catch (RemoteException re) { throw re.unwrapRemoteException(AccessControlException.class, UnresolvedPathException.class); } }
          Hide
          Kihwal Lee added a comment -

          The new patch includes the concat() change mentioned above.

          Show
          Kihwal Lee added a comment - The new patch includes the concat() change mentioned above.
          Hide
          Kihwal Lee added a comment -

          Added a simple test case in TestHDFSConcat to make sure concat() fails when sources have different checksums.

          Show
          Kihwal Lee added a comment - Added a simple test case in TestHDFSConcat to make sure concat() fails when sources have different checksums.
          Hide
          Kihwal Lee added a comment -

          The new patch is a minimum one without anything related to consistent checksum. No append(), no concat() changes.

          This should be independent of how we do file-level consistent checksum in the future.

          Show
          Kihwal Lee added a comment - The new patch is a minimum one without anything related to consistent checksum. No append(), no concat() changes. This should be independent of how we do file-level consistent checksum in the future.
          Hide
          Kihwal Lee added a comment -

          For the changes in getFileChecksum(..), most of the changes is for refactoring callBlockChecksum(..). If we remove the append change here, let's also defer the refactoring so that it is easier to review. BTW, there is an unused "retry" variable in callBlockChecksum(..).

          This is what the minimum patch is doing. Please make a final decision soon and pick either the full patch or the minimum one.

          Show
          Kihwal Lee added a comment - For the changes in getFileChecksum(..), most of the changes is for refactoring callBlockChecksum(..). If we remove the append change here, let's also defer the refactoring so that it is easier to review. BTW, there is an unused "retry" variable in callBlockChecksum(..). This is what the minimum patch is doing. Please make a final decision soon and pick either the full patch or the minimum one.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12542214/hdfs-3177-branch2-trunk.patch.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestSafeMode
          org.apache.hadoop.hdfs.TestPersistBlocks

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3081//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3081//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3081//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542214/hdfs-3177-branch2-trunk.patch.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestSafeMode org.apache.hadoop.hdfs.TestPersistBlocks +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3081//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3081//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3081//console This message is automatically generated.
          Hide
          Kihwal Lee added a comment -

          The findbugs warning and test failures have nothing to do with the patch.

          Show
          Kihwal Lee added a comment - The findbugs warning and test failures have nothing to do with the patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12542223/hdfs-3177-minimum-without-append-concat-trunk-branch2.patch.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
          org.apache.hadoop.hdfs.TestHftpDelegationToken

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3083//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3083//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3083//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542223/hdfs-3177-minimum-without-append-concat-trunk-branch2.patch.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks org.apache.hadoop.hdfs.TestHftpDelegationToken +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3083//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3083//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3083//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12542223/hdfs-3177-minimum-without-append-concat-trunk-branch2.patch.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestDatanodeBlockScanner
          org.apache.hadoop.hdfs.TestPersistBlocks

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3084//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3084//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3084//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542223/hdfs-3177-minimum-without-append-concat-trunk-branch2.patch.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestDatanodeBlockScanner org.apache.hadoop.hdfs.TestPersistBlocks +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3084//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3084//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3084//console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 the latest patch looks good.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 the latest patch looks good.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2633 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2633/)
          HDFS-3177. Update DFSClient and DataXceiver to handle different checkum types in file checksum computation. Contributed by Kihwal Lee (Revision 1376928)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376928
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsProtoUtil.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2633 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2633/ ) HDFS-3177 . Update DFSClient and DataXceiver to handle different checkum types in file checksum computation. Contributed by Kihwal Lee (Revision 1376928) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376928 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsProtoUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this. Kihwal, thank you for all the great works!

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this. Kihwal, thank you for all the great works!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2697 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2697/)
          HDFS-3177. Update DFSClient and DataXceiver to handle different checkum types in file checksum computation. Contributed by Kihwal Lee (Revision 1376928)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376928
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsProtoUtil.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2697 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2697/ ) HDFS-3177 . Update DFSClient and DataXceiver to handle different checkum types in file checksum computation. Contributed by Kihwal Lee (Revision 1376928) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376928 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsProtoUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
          Hide
          Daryn Sharp added a comment -

          I have committed to branch-23.

          Show
          Daryn Sharp added a comment - I have committed to branch-23.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #354 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/354/)
          HDFS-3177. Update DFSClient and DataXceiver to handle different checkum types in file checksum computation. (Kihwal Lee via daryn) (Revision 1376955)

          Result = UNSTABLE
          daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376955
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsProtoUtil.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #354 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/354/ ) HDFS-3177 . Update DFSClient and DataXceiver to handle different checkum types in file checksum computation. (Kihwal Lee via daryn) (Revision 1376955) Result = UNSTABLE daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376955 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsProtoUtil.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1145 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1145/)
          HDFS-3177. Update DFSClient and DataXceiver to handle different checkum types in file checksum computation. Contributed by Kihwal Lee (Revision 1376928)

          Result = FAILURE
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376928
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsProtoUtil.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1145 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1145/ ) HDFS-3177 . Update DFSClient and DataXceiver to handle different checkum types in file checksum computation. Contributed by Kihwal Lee (Revision 1376928) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376928 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsProtoUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1176 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1176/)
          HDFS-3177. Update DFSClient and DataXceiver to handle different checkum types in file checksum computation. Contributed by Kihwal Lee (Revision 1376928)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376928
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsProtoUtil.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1176 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1176/ ) HDFS-3177 . Update DFSClient and DataXceiver to handle different checkum types in file checksum computation. Contributed by Kihwal Lee (Revision 1376928) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376928 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsProtoUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/datatransfer.proto /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java

            People

            • Assignee:
              Kihwal Lee
              Reporter:
              Kihwal Lee
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development