Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2722

HttpFs shouldn't be using an int for block size

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.1
    • Component/s: hdfs-client
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      ./hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java: blockSize = fs.getConf().getInt("dfs.block.size", 67108864);

      Should instead be using dfs.blocksize and should instead be long.

      I'll post a patch for this after HDFS-1314 is resolved – which changes the internal behavior a bit (should be getLongBytes, and not just getLong, to gain formatting advantages).

        Issue Links

          Activity

          Harsh J created issue -
          Harsh J made changes -
          Field Original Value New Value
          Description {{./hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java: blockSize = fs.getConf().getInt("dfs.block.size", 67108864);}}

          Should instead be using dfs.blocksize.

          I'll post a patch for this after HDFS-1314 is resolved -- which changes the internal behavior a bit (should be getLongBytes, and not just getLong, to gain formatting advantages).
          {{./hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java: blockSize = fs.getConf().getInt("dfs.block.size", 67108864);}}

          Should instead be using dfs.blocksize and should instead be long.

          I'll post a patch for this after HDFS-1314 is resolved -- which changes the internal behavior a bit (should be getLongBytes, and not just getLong, to gain formatting advantages).
          Harsh J made changes -
          Assignee Harsh J [ qwertymaniac ]
          Harsh J made changes -
          Assignee Harsh J [ qwertymaniac ] Sho Shimauchi [ sho.shimauchi ]
          Harsh J made changes -
          Link This issue is part of HDFS-1314 [ HDFS-1314 ]
          Hide
          Harsh J added a comment -

          We should use fs.getDefaultBlockSize(); here.

          Show
          Harsh J added a comment - We should use fs.getDefaultBlockSize(); here.
          Harsh J made changes -
          Assignee Sho Shimauchi [ sho.shimauchi ] Harsh J [ qwertymaniac ]
          Harsh J made changes -
          Attachment HDFS-2722.patch [ 12508602 ]
          Harsh J made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 0.24.0, 0.23.1, 1.1.0 [ 12317653, 12318885, 12317959 ]
          Harsh J made changes -
          Target Version/s 1.1.0, 0.23.1, 0.24.0 [ 12317959, 12318885, 12317653 ] 0.24.0, 0.23.1 [ 12317653, 12318885 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12508602/HDFS-2722.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javadoc. The javadoc tool appears to have generated 20 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings).

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1738//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1738//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1738//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1738//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12508602/HDFS-2722.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 20 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings). +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1738//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1738//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1738//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1738//console This message is automatically generated.
          Hide
          Harsh J added a comment -

          The findbugs reported here is really odd. Does appear unrelated to this config wrapping patch.

          Class org.apache.hadoop.hdfs.tools.GetConf$Command defines non-transient non-serializable instance field handler

          This Serializable class defines a non-primitive instance field which is neither transient, Serializable, or java.lang.Object, and does not appear to implement the Externalizable interface or the readObject() and writeObject() methods. Objects of this class will not be deserialized correctly if a non-Serializable object is stored in this field.

          What makes this odd is that GetConf.java was not touched since September 2011 per viewvc. Why would this appear now?

          Show
          Harsh J added a comment - The findbugs reported here is really odd. Does appear unrelated to this config wrapping patch. Class org.apache.hadoop.hdfs.tools.GetConf$Command defines non-transient non-serializable instance field handler This Serializable class defines a non-primitive instance field which is neither transient, Serializable, or java.lang.Object, and does not appear to implement the Externalizable interface or the readObject() and writeObject() methods. Objects of this class will not be deserialized correctly if a non-Serializable object is stored in this field. What makes this odd is that GetConf.java was not touched since September 2011 per viewvc . Why would this appear now?
          Hide
          Alejandro Abdelnur added a comment -

          +1

          Show
          Alejandro Abdelnur added a comment - +1
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1558 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1558/)
          HDFS-2722. HttpFs should not be using an int for block size. (harsh)

          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226861
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1558 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1558/ ) HDFS-2722 . HttpFs should not be using an int for block size. (harsh) harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226861 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1486 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1486/)
          HDFS-2722. HttpFs should not be using an int for block size. (harsh)

          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226861
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1486 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1486/ ) HDFS-2722 . HttpFs should not be using an int for block size. (harsh) harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226861 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          Harsh J added a comment -

          Committed to branch-0.23 and trunk. Thanks for the review Alejandro!

          Show
          Harsh J added a comment - Committed to branch-0.23 and trunk. Thanks for the review Alejandro!
          Harsh J made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Target Version/s 0.23.1, 0.24.0 [ 12318885, 12317653 ] 0.24.0, 0.23.1 [ 12317653, 12318885 ]
          Fix Version/s 0.23.1 [ 12318885 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Commit #320 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/320/)
          merge HDFS-2722 (harsh)

          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226880
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #320 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/320/ ) merge HDFS-2722 (harsh) harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226880 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-0.23-Commit #331 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/331/)
          merge HDFS-2722 (harsh)

          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226880
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #331 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/331/ ) merge HDFS-2722 (harsh) harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226880 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1507 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1507/)
          HDFS-2722. HttpFs should not be using an int for block size. (harsh)

          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226861
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1507 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1507/ ) HDFS-2722 . HttpFs should not be using an int for block size. (harsh) harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226861 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Commit #342 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/342/)
          merge HDFS-2722 (harsh)

          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226880
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #342 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/342/ ) merge HDFS-2722 (harsh) harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226880 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #915 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/915/)
          HDFS-2722. HttpFs should not be using an int for block size. (harsh)

          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226861
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #915 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/915/ ) HDFS-2722 . HttpFs should not be using an int for block size. (harsh) harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226861 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #128 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/128/)
          merge HDFS-2722 (harsh)

          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226880
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #128 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/128/ ) merge HDFS-2722 (harsh) harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226880 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Build #150 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/150/)
          merge HDFS-2722 (harsh)

          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226880
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #150 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/150/ ) merge HDFS-2722 (harsh) harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226880 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #948 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/948/)
          HDFS-2722. HttpFs should not be using an int for block size. (harsh)

          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226861
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #948 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/948/ ) HDFS-2722 . HttpFs should not be using an int for block size. (harsh) harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1226861 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          22h 53m 1 Harsh J 25/Dec/11 04:50
          Patch Available Patch Available Resolved Resolved
          9d 12h 32m 1 Harsh J 03/Jan/12 17:23
          Resolved Resolved Closed Closed
          61d 9h 26m 1 Arun C Murthy 05/Mar/12 02:49

            People

            • Assignee:
              Harsh J
              Reporter:
              Harsh J
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development