Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12442

WebHdfsFileSystem#getFileBlockLocations will always return BlockLocation#corrupt as false

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 2.9.0, 3.0.0-alpha2
    • Fix Version/s: None
    • Component/s: webhdfs
    • Labels:
      None

      Description

      Was going through JsonUtilClient#toBlockLocation code.
      Below is the relevant code snippet.

      JsonUtilClient.java
       /** Convert a Json map to BlockLocation. **/
        static BlockLocation toBlockLocation(Map<?, ?> m)
            throws IOException{
          ...
          ...  
          boolean corrupt = Boolean.
              getBoolean(m.get("corrupt").toString());
          ...
          ...
        }
      

      According to java docs for Boolean#getBoolean

      Returns true if and only if the system property named by the argument exists and is equal to the string "true". 
      

      I assume, the map value for key corrupt will be populated with either true or false.
      On the client side, Boolean#getBoolean will look for system property for true or false.
      So it will always return false unless the system property is set for true or false.

      1. HDFS-12442-1.patch
        3 kB
        Rushabh S Shah

        Issue Links

          Activity

          Hide
          shahrs87 Rushabh S Shah added a comment -

          Attaching a simple patch with test case.

          Show
          shahrs87 Rushabh S Shah added a comment - Attaching a simple patch with test case.
          Hide
          shahrs87 Rushabh S Shah added a comment -

          Weiwei Yang: can you please review.

          Show
          shahrs87 Rushabh S Shah added a comment - Weiwei Yang : can you please review.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 8s Maven dependency ordering for branch
          +1 mvninstall 13m 57s trunk passed
          +1 compile 1m 27s trunk passed
          +1 checkstyle 0m 41s trunk passed
          +1 mvnsite 1m 28s trunk passed
          +1 findbugs 3m 15s trunk passed
          +1 javadoc 1m 1s trunk passed
                Patch Compile Tests
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 1m 25s the patch passed
          +1 compile 1m 30s the patch passed
          +1 javac 1m 30s the patch passed
          +1 checkstyle 0m 40s the patch passed
          +1 mvnsite 1m 25s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 21s the patch passed
          +1 javadoc 0m 58s the patch passed
                Other Tests
          +1 unit 1m 14s hadoop-hdfs-client in the patch passed.
          -1 unit 92m 23s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          127m 3s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestLeaseRecoveryStriped
            hadoop.hdfs.qjournal.server.TestJournalNodeSync
            hadoop.hdfs.server.namenode.TestReencryption
            hadoop.hdfs.TestReplication
          Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:71bbb86
          JIRA Issue HDFS-12442
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12886983/HDFS-12442-1.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 3a07dd3f39a4 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / bb34ae9
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/21127/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21127/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21127/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests 0 mvndep 0m 8s Maven dependency ordering for branch +1 mvninstall 13m 57s trunk passed +1 compile 1m 27s trunk passed +1 checkstyle 0m 41s trunk passed +1 mvnsite 1m 28s trunk passed +1 findbugs 3m 15s trunk passed +1 javadoc 1m 1s trunk passed       Patch Compile Tests 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 25s the patch passed +1 compile 1m 30s the patch passed +1 javac 1m 30s the patch passed +1 checkstyle 0m 40s the patch passed +1 mvnsite 1m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 21s the patch passed +1 javadoc 0m 58s the patch passed       Other Tests +1 unit 1m 14s hadoop-hdfs-client in the patch passed. -1 unit 92m 23s hadoop-hdfs in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 127m 3s Reason Tests Failed junit tests hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.qjournal.server.TestJournalNodeSync   hadoop.hdfs.server.namenode.TestReencryption   hadoop.hdfs.TestReplication Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue HDFS-12442 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12886983/HDFS-12442-1.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3a07dd3f39a4 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / bb34ae9 Default Java 1.8.0_144 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-HDFS-Build/21127/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21127/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21127/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cheersyang Weiwei Yang added a comment -

          Hi Rushabh S Shah

          Thank you for catching this up and the fix. That was a mis-use of the API like you pointed out, can we replace that with Boolean.parseBoolean()? Other seems good to me. But I am not a committer yet, guess you need some one else to help review and the commit, thank you.

          Show
          cheersyang Weiwei Yang added a comment - Hi Rushabh S Shah Thank you for catching this up and the fix. That was a mis-use of the API like you pointed out, can we replace that with Boolean.parseBoolean() ? Other seems good to me. But I am not a committer yet, guess you need some one else to help review and the commit, thank you.
          Hide
          shahrs87 Rushabh S Shah added a comment - - edited

          Thanks Weiwei Yang for the review.
          I don't know whether we would need this jira judging by the developments going on in HDFS-11156.
          If we still need this patch, I will attach a new patch to address review comment.
          Lets wait and watch.

          Show
          shahrs87 Rushabh S Shah added a comment - - edited Thanks Weiwei Yang for the review. I don't know whether we would need this jira judging by the developments going on in HDFS-11156 . If we still need this patch, I will attach a new patch to address review comment. Lets wait and watch.

            People

            • Assignee:
              shahrs87 Rushabh S Shah
              Reporter:
              shahrs87 Rushabh S Shah
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:

                Development