Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2722

HttpFs shouldn't be using an int for block size

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.1
    • Component/s: hdfs-client
    • Labels:
      None

      Description

      ./hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java: blockSize = fs.getConf().getInt("dfs.block.size", 67108864);

      Should instead be using dfs.blocksize and should instead be long.

      I'll post a patch for this after HDFS-1314 is resolved – which changes the internal behavior a bit (should be getLongBytes, and not just getLong, to gain formatting advantages).

        Issue Links

          Activity

          Hide
          Harsh J added a comment -

          We should use fs.getDefaultBlockSize(); here.

          Show
          Harsh J added a comment - We should use fs.getDefaultBlockSize(); here.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12508602/HDFS-2722.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javadoc. The javadoc tool appears to have generated 20 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings).

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1738//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1738//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1738//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1738//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12508602/HDFS-2722.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 20 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings). +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1738//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1738//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1738//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1738//console This message is automatically generated.
          Hide
          Harsh J added a comment -

          The findbugs reported here is really odd. Does appear unrelated to this config wrapping patch.

          Class org.apache.hadoop.hdfs.tools.GetConf$Command defines non-transient non-serializable instance field handler

          This Serializable class defines a non-primitive instance field which is neither transient, Serializable, or java.lang.Object, and does not appear to implement the Externalizable interface or the readObject() and writeObject() methods. Objects of this class will not be deserialized correctly if a non-Serializable object is stored in this field.

          What makes this odd is that GetConf.java was not touched since September 2011 per viewvc. Why would this appear now?

          Show
          Harsh J added a comment - The findbugs reported here is really odd. Does appear unrelated to this config wrapping patch. Class org.apache.hadoop.hdfs.tools.GetConf$Command defines non-transient non-serializable instance field handler This Serializable class defines a non-primitive instance field which is neither transient, Serializable, or java.lang.Object, and does not appear to implement the Externalizable interface or the readObject() and writeObject() methods. Objects of this class will not be deserialized correctly if a non-Serializable object is stored in this field. What makes this odd is that GetConf.java was not touched since September 2011 per viewvc . Why would this appear now?
          Hide
          Alejandro Abdelnur added a comment -

          +1

          Show
          Alejandro Abdelnur added a comment - +1
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1558 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1558/)
          HDFS-2722. HttpFs should not be using an int for block size. (harsh)

          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226861
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1558 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1558/ ) HDFS-2722 . HttpFs should not be using an int for block size. (harsh) harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226861 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1486 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1486/)
          HDFS-2722. HttpFs should not be using an int for block size. (harsh)

          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226861
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1486 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1486/ ) HDFS-2722 . HttpFs should not be using an int for block size. (harsh) harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226861 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          Harsh J added a comment -

          Committed to branch-0.23 and trunk. Thanks for the review Alejandro!

          Show
          Harsh J added a comment - Committed to branch-0.23 and trunk. Thanks for the review Alejandro!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Commit #320 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/320/)
          merge HDFS-2722 (harsh)

          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226880
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #320 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/320/ ) merge HDFS-2722 (harsh) harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226880 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-0.23-Commit #331 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/331/)
          merge HDFS-2722 (harsh)

          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226880
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #331 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/331/ ) merge HDFS-2722 (harsh) harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226880 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1507 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1507/)
          HDFS-2722. HttpFs should not be using an int for block size. (harsh)

          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226861
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1507 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1507/ ) HDFS-2722 . HttpFs should not be using an int for block size. (harsh) harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226861 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Commit #342 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/342/)
          merge HDFS-2722 (harsh)

          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226880
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #342 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/342/ ) merge HDFS-2722 (harsh) harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226880 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #915 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/915/)
          HDFS-2722. HttpFs should not be using an int for block size. (harsh)

          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226861
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #915 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/915/ ) HDFS-2722 . HttpFs should not be using an int for block size. (harsh) harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226861 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #128 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/128/)
          merge HDFS-2722 (harsh)

          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226880
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #128 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/128/ ) merge HDFS-2722 (harsh) harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226880 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Build #150 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/150/)
          merge HDFS-2722 (harsh)

          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226880
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #150 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/150/ ) merge HDFS-2722 (harsh) harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226880 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #948 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/948/)
          HDFS-2722. HttpFs should not be using an int for block size. (harsh)

          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226861
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #948 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/948/ ) HDFS-2722 . HttpFs should not be using an int for block size. (harsh) harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226861 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

            People

            • Assignee:
              Harsh J
              Reporter:
              Harsh J
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development