Hadoop Common
  1. Hadoop Common
  2. HADOOP-3805

improve fuse-dfs write performance which is 33% slower than hadoop dfs -copyFromLocal

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.19.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      on hadoop17 running on the namenode/fuse mount point, fuse is 33% slower for writing a 1GB file. I don't know right now how to improve this but thought I should open up a JIRA for it.

      Note - this benchmark is with no tuning or special fuse config params.

      1. hadoop-3805.txt
        5 kB
        Pete Wyckoff

        Activity

        Pete Wyckoff created issue -
        Hide
        Pete Wyckoff added a comment -

        Discussion of options and kernels needed for better write performance in fuse:

        http://www.nabble.com/fuse-performance-td18271595.html

        basically need to use -o big_writes and have a kernel > 2.6.26-rc*

        I don't currently have access to such a kernel, but it's probably worth while just adding the -obig_writes option by default and having a conifg vars no_big_writes use_big_writes.

        I will do up a patch.

        Show
        Pete Wyckoff added a comment - Discussion of options and kernels needed for better write performance in fuse: http://www.nabble.com/fuse-performance-td18271595.html basically need to use -o big_writes and have a kernel > 2.6.26-rc* I don't currently have access to such a kernel, but it's probably worth while just adding the -obig_writes option by default and having a conifg vars no_big_writes use_big_writes. I will do up a patch.
        Hide
        Pete Wyckoff added a comment -

        I should also mention that hdfs_write utility build on libhdfs is no slower than the shell copy, so the problem is likely not in libhdfs. (based on very simple tests and Owen also confirmed that he thought libhdfs wouldn't have any performance issues)

        Show
        Pete Wyckoff added a comment - I should also mention that hdfs_write utility build on libhdfs is no slower than the shell copy, so the problem is likely not in libhdfs. (based on very simple tests and Owen also confirmed that he thought libhdfs wouldn't have any performance issues)
        Hide
        Pete Wyckoff added a comment -

        I made -obig_writes opt in so it wouldn't cause a backwards compat problem. I also #ifdef the actual use of the flag because it's only supported in fuse 2.8.0 + and 2.7.4 is the current stable release.

        Show
        Pete Wyckoff added a comment - I made -obig_writes opt in so it wouldn't cause a backwards compat problem. I also #ifdef the actual use of the flag because it's only supported in fuse 2.8.0 + and 2.7.4 is the current stable release.
        Pete Wyckoff made changes -
        Field Original Value New Value
        Attachment hadoop-3805.txt [ 12387356 ]
        Pete Wyckoff made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12387356/hadoop-3805.txt
        against trunk revision 682978.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3004/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3004/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3004/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3004/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12387356/hadoop-3805.txt against trunk revision 682978. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3004/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3004/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3004/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3004/console This message is automatically generated.
        Hide
        dhruba borthakur added a comment -

        +1 Code looks good.

        Show
        dhruba borthakur added a comment - +1 Code looks good.
        Hide
        Zheng Shao added a comment -

        Committed.

        Show
        Zheng Shao added a comment - Committed.
        Zheng Shao made changes -
        Fix Version/s 0.19.0 [ 12313211 ]
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hadoop Flags [Reviewed]
        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Hadoop-trunk #581 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/581/ )
        Nigel Daley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Owen O'Malley made changes -
        Component/s contrib/fuse-dfs [ 12312376 ]

          People

          • Assignee:
            Unassigned
            Reporter:
            Pete Wyckoff
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development