Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2727

libhdfs should get the default block size from the server

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.23.0, 1.0.0, 2.0.0-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: libhdfs
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      libhdfs now uses the server block size configuration rather than the deprecated dfs.block.size client configuration.

      Description

      libhdfs uses the deprecated property dfs.block.size. It would be better for libhdfs to use the FileSystem#getDefaultBlockSize API instead.

      This API has two advantages:
      1. It uses the new, non-deprecated dfs.blocksize property
      2. It handles string escapes like 'M' for 'megabytes' etc. which can appear in dfs.blocksize and dfs.block.size

      1. HDFS-2727.002.patch
        14 kB
        Colin Patrick McCabe
      2. HDFS-2727.001.patch
        9 kB
        Colin Patrick McCabe

        Activity

        Hide
        Harsh J added a comment -

        It should not rely on properties for dfs.blocksize and dfs.replication, and instead fetch those from the jFS object itself, via the getDefaultBlockSize and getDefaultReplication API calls. This will help avoid maintenance in future

        Show
        Harsh J added a comment - It should not rely on properties for dfs.blocksize and dfs.replication, and instead fetch those from the jFS object itself, via the getDefaultBlockSize and getDefaultReplication API calls. This will help avoid maintenance in future
        Hide
        Colin Patrick McCabe added a comment -

        This patch uses FileSystem.getDefaultBlockSize() to avoid re-implementing the parsing logic for dfs.blocksize / dfs.block.size / whatever.

        This fixes a bug where setting dfs.blocksize="128M" would cause libhdfs to not work.

        Also implement hdfsGetDefaultBlockSizeAtPath, and add a unit test for same.

        Show
        Colin Patrick McCabe added a comment - This patch uses FileSystem.getDefaultBlockSize() to avoid re-implementing the parsing logic for dfs.blocksize / dfs.block.size / whatever. This fixes a bug where setting dfs.blocksize="128M" would cause libhdfs to not work. Also implement hdfsGetDefaultBlockSizeAtPath, and add a unit test for same.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12541291/HDFS-2727.001.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3035//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3035//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541291/HDFS-2727.001.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3035//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3035//console This message is automatically generated.
        Hide
        Eli Collins added a comment -

        Looks great. Two nits:

        • The java equivalent is fs.getDefaultBlockSize (no "AtPath")
        • "The path to find the default blocksize at." might better be written as "The given path will be used to locate the actual filesystem. The full path does not have to exist." to match the FileSystem docs.

        Please update the jira description with the problem / approach here.

        Also since the new tests don't cover this please test that libhdfs works w/ a non-default server side value (eg 128mb).

        Show
        Eli Collins added a comment - Looks great. Two nits: The java equivalent is fs.getDefaultBlockSize (no "AtPath") "The path to find the default blocksize at." might better be written as "The given path will be used to locate the actual filesystem. The full path does not have to exist." to match the FileSystem docs. Please update the jira description with the problem / approach here. Also since the new tests don't cover this please test that libhdfs works w/ a non-default server side value (eg 128mb).
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12541686/HDFS-2727.002.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3050//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3050//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541686/HDFS-2727.002.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3050//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3050//console This message is automatically generated.
        Hide
        Eli Collins added a comment -

        +1

        Nit: the diff to jni_helper.c is a nop (removes the EOL), I'll revert it

        Show
        Eli Collins added a comment - +1 Nit: the diff to jni_helper.c is a nop (removes the EOL), I'll revert it
        Hide
        Eli Collins added a comment -

        I've committed this and merged to branch-2. Thanks Colin!

        Show
        Eli Collins added a comment - I've committed this and merged to branch-2. Thanks Colin!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2674 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2674/)
        HDFS-2727. libhdfs should get the default block size from the server. Contributed by Colin Patrick McCabe (Revision 1375383)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1375383
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test_libhdfs_threaded.c
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2674 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2674/ ) HDFS-2727 . libhdfs should get the default block size from the server. Contributed by Colin Patrick McCabe (Revision 1375383) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1375383 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test_libhdfs_threaded.c
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2610 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2610/)
        HDFS-2727. libhdfs should get the default block size from the server. Contributed by Colin Patrick McCabe (Revision 1375383)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1375383
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test_libhdfs_threaded.c
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2610 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2610/ ) HDFS-2727 . libhdfs should get the default block size from the server. Contributed by Colin Patrick McCabe (Revision 1375383) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1375383 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test_libhdfs_threaded.c
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2639 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2639/)
        HDFS-2727. libhdfs should get the default block size from the server. Contributed by Colin Patrick McCabe (Revision 1375383)

        Result = FAILURE
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1375383
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test_libhdfs_threaded.c
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2639 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2639/ ) HDFS-2727 . libhdfs should get the default block size from the server. Contributed by Colin Patrick McCabe (Revision 1375383) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1375383 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test_libhdfs_threaded.c
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1141 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1141/)
        HDFS-2727. libhdfs should get the default block size from the server. Contributed by Colin Patrick McCabe (Revision 1375383)

        Result = FAILURE
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1375383
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test_libhdfs_threaded.c
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1141 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1141/ ) HDFS-2727 . libhdfs should get the default block size from the server. Contributed by Colin Patrick McCabe (Revision 1375383) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1375383 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test_libhdfs_threaded.c
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1173 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1173/)
        HDFS-2727. libhdfs should get the default block size from the server. Contributed by Colin Patrick McCabe (Revision 1375383)

        Result = FAILURE
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1375383
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test_libhdfs_threaded.c
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1173 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1173/ ) HDFS-2727 . libhdfs should get the default block size from the server. Contributed by Colin Patrick McCabe (Revision 1375383) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1375383 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.h /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test_libhdfs_threaded.c

          People

          • Assignee:
            Colin Patrick McCabe
            Reporter:
            Sho Shimauchi
          • Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development