Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-578

Support for using server default values for blockSize and replication when creating a file

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: hdfs-client, namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      New FileSystem.getServerDefaults() reports the server's default file creation parameters.

      Description

      This is a sub-task of HADOOP-4952. This improvement makes it possible for a client to specify that it wants to use the server default values for blockSize and replication params when creating a file.

      1. h578-16.patch
        11 kB
        Kan Zhang
      2. h578-14.patch
        11 kB
        Kan Zhang
      3. h578-13.patch
        16 kB
        Kan Zhang

        Issue Links

          Activity

          Hide
          Robert Chansler added a comment -

          Editorial pass over all release notes prior to publication of 0.21.

          Show
          Robert Chansler added a comment - Editorial pass over all release notes prior to publication of 0.21.
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #21 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/21/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #21 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/21/ )
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #5 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/5/)
          . Add support for new FileSystem method for clients to get server defaults. Contributed by Kan Zhang.

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #5 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/5/ ) . Add support for new FileSystem method for clients to get server defaults. Contributed by Kan Zhang.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #21 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/21/)
          . Add support for new FileSystem method for clients to get server defaults. Contributed by Kan Zhang.

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #21 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/21/ ) . Add support for new FileSystem method for clients to get server defaults. Contributed by Kan Zhang.
          Hide
          Kan Zhang added a comment -

          Attached a new patch, which merges cleanly with current trunk. Also modified the name of a test method to be more appropriate. No functional change. Also ran test and test-patch on the new patch and they were successful.

          Show
          Kan Zhang added a comment - Attached a new patch, which merges cleanly with current trunk. Also modified the name of a test method to be more appropriate. No functional change. Also ran test and test-patch on the new patch and they were successful.
          Hide
          Jakob Homan added a comment -

          +1 Changed to incompatible change since it increments the client protocol version.

          Show
          Jakob Homan added a comment - +1 Changed to incompatible change since it increments the client protocol version.
          Hide
          Kan Zhang added a comment -

          Ran ant test and ant test-patch. They were both successful.

          Show
          Kan Zhang added a comment - Ran ant test and ant test-patch. They were both successful.
          Hide
          Kan Zhang added a comment -

          Per request from Sanjay, this patch will only add method getServerDefaults() to DFSClient and will not otherwise change any existing behavior of DFSClient. Allowing user specifying -1 for server defaults will be done in HADOOP-4952. New patch attached.

          Show
          Kan Zhang added a comment - Per request from Sanjay, this patch will only add method getServerDefaults() to DFSClient and will not otherwise change any existing behavior of DFSClient. Allowing user specifying -1 for server defaults will be done in HADOOP-4952 . New patch attached.
          Hide
          Kan Zhang added a comment -

          Thanks for pointing it out. I ran "ant test" just now and it was successful.

          Show
          Kan Zhang added a comment - Thanks for pointing it out. I ran "ant test" just now and it was successful.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > I ran run-test-hdfs ...
          Hi Kan, you have to run "ant test", which includes all the tests. "run-test-hdfs" does not include the fault injection tests.

          Show
          Tsz Wo Nicholas Sze added a comment - > I ran run-test-hdfs ... Hi Kan, you have to run "ant test", which includes all the tests. "run-test-hdfs" does not include the fault injection tests.
          Hide
          Kan Zhang added a comment -

          I ran run-test-hdfs and it passed. Here is the test-patch result.

          [exec] +1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] +1 tests included. The patch appears to include 6 new or modified tests.
          [exec]
          [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
          [exec]
          [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

          Show
          Kan Zhang added a comment - I ran run-test-hdfs and it passed. Here is the test-patch result. [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 6 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
          Hide
          Kan Zhang added a comment -

          I meant cannot.

          Show
          Kan Zhang added a comment - I meant cannot.
          Hide
          Kan Zhang added a comment -

          Ignore the above Hadoop QA comment. This patch can be run without HADOOP-6235 being committed.

          Show
          Kan Zhang added a comment - Ignore the above Hadoop QA comment. This patch can be run without HADOOP-6235 being committed.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12418460/h578-13.patch
          against trunk revision 810631.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The patch appears to cause tar ant target to fail.

          -1 findbugs. The patch appears to cause Findbugs to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/1/testReport/
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/1/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/1/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12418460/h578-13.patch against trunk revision 810631. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 findbugs. The patch appears to cause Findbugs to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/1/testReport/ Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/1/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/1/console This message is automatically generated.
          Hide
          Kan Zhang added a comment -

          Attached a patch that allows users to use server default values when creating a file (by specifying -1). The 4 params (blockSize, bytesPerChecksum, writePacketSize and io.file.buffer.size) are bundled together. If the user chooses to use server default for blockSize, all 4 params will use server defaults. DFSClient caches server default values for 1 hour and fetches them from server only when necessary.

          On a separate note, client can choose io.file.buffer.size and pass it to create() call, but it is actually ignored in the current implementation.

          Show
          Kan Zhang added a comment - Attached a patch that allows users to use server default values when creating a file (by specifying -1). The 4 params (blockSize, bytesPerChecksum, writePacketSize and io.file.buffer.size) are bundled together. If the user chooses to use server default for blockSize, all 4 params will use server defaults. DFSClient caches server default values for 1 hour and fetches them from server only when necessary. On a separate note, client can choose io.file.buffer.size and pass it to create() call, but it is actually ignored in the current implementation.
          Hide
          Kan Zhang added a comment -

          I suggest we bundle the 3 params (blockSize, bytesPerChecksum and writePacketSize) together. If a client decides to use server default for blockSize, then all 3 will use server defaults. If a (sophisticated) client decides not to use server default for blockSize, then the client has to provide valid values for all 3 params. In particular, the validation that blockSize matches bytesPerChecksum is done at the client side since currently bytesPerChecksum is not sent to the namenode on the create() call. but I'm open to adding this param to the create() call so that it can be validated on the namenode. Thoughts?

          Show
          Kan Zhang added a comment - I suggest we bundle the 3 params (blockSize, bytesPerChecksum and writePacketSize) together. If a client decides to use server default for blockSize, then all 3 will use server defaults. If a (sophisticated) client decides not to use server default for blockSize, then the client has to provide valid values for all 3 params. In particular, the validation that blockSize matches bytesPerChecksum is done at the client side since currently bytesPerChecksum is not sent to the namenode on the create() call. but I'm open to adding this param to the create() call so that it can be validated on the namenode. Thoughts?
          Hide
          Sanjay Radia added a comment -

          This is closely tied to HADOOP-4952 (so please refer to that) which states that we support options 1 and 2 but not 3:

          1. Specify the desired values as explict parameters in the methods such as create. [as in current system]
          2. Use SS defaults - different FS and their deployments can have their own defaults and also frees the admin from distributing the default config to all client nodes. [New]
          3. Use client side defaults derived from the config. [ is in current system]

          It also proposes that there are no per FileSystem defaults, only per deployment SS defaults (see HADOOP-4952 for details).

          What it means is that one does not need fs config variables except for the default fs in the config.
          This works well for everything except the bytesPerChecksum, because the create call does not have parameter called bytesPerChecksum.
          If application configurable bytesPerChecksum is important then there are two options: (a) add it as a parameter to the create call (most folks
          will get confused by what to set) or (b) support options 3. If we support option 3 then to be consistent we should probably support it for all parameters (blkSize, repFac, buffer size, etc).

          Show
          Sanjay Radia added a comment - This is closely tied to HADOOP-4952 (so please refer to that) which states that we support options 1 and 2 but not 3: Specify the desired values as explict parameters in the methods such as create. [as in current system] Use SS defaults - different FS and their deployments can have their own defaults and also frees the admin from distributing the default config to all client nodes. [New] Use client side defaults derived from the config. [ is in current system] It also proposes that there are no per FileSystem defaults, only per deployment SS defaults (see HADOOP-4952 for details). What it means is that one does not need fs config variables except for the default fs in the config. This works well for everything except the bytesPerChecksum, because the create call does not have parameter called bytesPerChecksum. If application configurable bytesPerChecksum is important then there are two options: (a) add it as a parameter to the create call (most folks will get confused by what to set) or (b) support options 3. If we support option 3 then to be consistent we should probably support it for all parameters (blkSize, repFac, buffer size, etc).
          Hide
          dhruba borthakur added a comment -

          > Do you expect a significant performance difference here?

          Good point. I am not really aware of any objective benchmark in this regard. No sure what performance impact this might have.

          > was proposing that we remove that flexibility

          What is the simplification if we remove this flexibility, can you pl explain? thanks.

          Show
          dhruba borthakur added a comment - > Do you expect a significant performance difference here? Good point. I am not really aware of any objective benchmark in this regard. No sure what performance impact this might have. > was proposing that we remove that flexibility What is the simplification if we remove this flexibility, can you pl explain? thanks.
          Hide
          Sanjay Radia added a comment -

          I was proposing that we remove that flexibility. How many other file systems let you set bytesPerChecksum?
          Do you expect a significant performance difference here?

          Show
          Sanjay Radia added a comment - I was proposing that we remove that flexibility. How many other file systems let you set bytesPerChecksum? Do you expect a significant performance difference here?
          Hide
          dhruba borthakur added a comment -

          > My current thinking is if the client chooses to use server default for blockSize, it should use server defaults for io.bytes.per.checksum and dfs.write.packet.size at the same time. What do you think?

          Sounds good to me.

          > The bytesPerChecksum and and packetSize are always SS

          The default value of 512 is suitable for random reads, isn't it? if an application knows that it does not need random read support for a file, it can specify the bytesperChecksum to be larger that the default. Don't we want to allow that flexibility?

          Show
          dhruba borthakur added a comment - > My current thinking is if the client chooses to use server default for blockSize, it should use server defaults for io.bytes.per.checksum and dfs.write.packet.size at the same time. What do you think? Sounds good to me. > The bytesPerChecksum and and packetSize are always SS The default value of 512 is suitable for random reads, isn't it? if an application knows that it does not need random read support for a file, it can specify the bytesperChecksum to be larger that the default. Don't we want to allow that flexibility?
          Hide
          Sanjay Radia added a comment -

          Proposal: BlockSize(BS) and RepF (RF)are either obtained from server side (SS) or application specifies it in his call to create.
          The bytesPerChecksum and and packetSize are always SS (ie cannot be specified on client side); ie remove this flexibility from the current system.

          Show
          Sanjay Radia added a comment - Proposal: BlockSize(BS) and RepF (RF)are either obtained from server side (SS) or application specifies it in his call to create. The bytesPerChecksum and and packetSize are always SS (ie cannot be specified on client side); ie remove this flexibility from the current system.
          Hide
          Kan Zhang added a comment -

          Yes, io.file.buffer.size is passed in to DFSOutputStream ctor, but it doesn't seem to be used in anyway? However, there are 2 other values used by DFSOutputStream that are chosen by the client, i.e., io.bytes.per.checksum and dfs.write.packet.size. My current thinking is if the client chooses to use server default for blockSize, it should use server defaults for io.bytes.per.checksum and dfs.write.packet.size at the same time. What do you think?

          Show
          Kan Zhang added a comment - Yes, io.file.buffer.size is passed in to DFSOutputStream ctor, but it doesn't seem to be used in anyway? However, there are 2 other values used by DFSOutputStream that are chosen by the client, i.e., io.bytes.per.checksum and dfs.write.packet.size. My current thinking is if the client chooses to use server default for blockSize, it should use server defaults for io.bytes.per.checksum and dfs.write.packet.size at the same time. What do you think?
          Hide
          dhruba borthakur added a comment -

          There are many other parameters that the client can default to the server settings. For example, io.file.buffer.size. What if we allow an option for the client to fetch a subset of the configuration from the Namenode and then use that and then overlay the client-side hdfs-site.xml?

          Show
          dhruba borthakur added a comment - There are many other parameters that the client can default to the server settings. For example, io.file.buffer.size. What if we allow an option for the client to fetch a subset of the configuration from the Namenode and then use that and then overlay the client-side hdfs-site.xml?

            People

            • Assignee:
              Kan Zhang
              Reporter:
              Kan Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development