Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.2.0
    • Fix Version/s: 0.3.0
    • Component/s: None
    • Labels:
      None

      Description

      Trying to change the DFS block size, led the realization that the 32,000,000 was hard coded into the source code. I propose:
      1. Change the default block size to 64 * 1024 * 1024.
      2. Add the config variable dfs.block.size that sets the default block size.
      3. Add a parameter to the FileSystem, DFSClient, and ClientProtocol create method that let's the user control the block size.
      4. Rename the FileSystem.getBlockSize to getDefaultBlockSize.
      5. Add a new method to FileSytem.getBlockSize that takes a pathname.
      6. Use long for the block size in the API, which is what was used before. However, the implementation will not work if block size is set bigger than 2**31.
      7. Have the InputFormatBase use the blocksize of each file to determine the split size.

      Thoughts?

      1. dfs-blocksize.patch
        35 kB
        Owen O'Malley
      2. TEST-org.apache.hadoop.fs.TestCopyFiles.txt
        48 kB
        Doug Cutting
      3. dfs-blocksize-2.patch
        35 kB
        Owen O'Malley

        Activity

        Owen O'Malley created issue -
        Owen O'Malley made changes -
        Field Original Value New Value
        Attachment dfs-blocksize.patch [ 12331921 ]
        Doug Cutting made changes -
        Owen O'Malley made changes -
        Attachment dfs-blocksize-2.patch [ 12331933 ]
        Doug Cutting made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Doug Cutting made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Doug Cutting made changes -
        Workflow jira [ 12363064 ] no reopen closed [ 12373123 ]
        Doug Cutting made changes -
        Workflow no reopen closed [ 12373123 ] no-reopen-closed [ 12373459 ]
        Doug Cutting made changes -
        Workflow no-reopen-closed [ 12373459 ] no-reopen-closed, patch-avail [ 12377772 ]
        Owen O'Malley made changes -
        Component/s dfs [ 12310710 ]

          People

          • Assignee:
            Owen O'Malley
            Reporter:
            Owen O'Malley
          • Votes:
            1 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development