Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.2
    • Fix Version/s: 0.23.0
    • Component/s: fuse-dfs
    • Labels:
      None
    • Environment:

      Fedora core 10, x86_64, 2.6.27.7-134.fc10.x86_64 #1 SMP (AMD 64), gcc 4.3.2, java 1.6.0 (IcedTea6 1.4 (fedora-7.b12.fc10-x86_64) Runtime Environment (build 1.6.0_0-b12) OpenJDK 64-Bit Server VM (build 10.0-b19, mixed mode)

    • Hadoop Flags:
      Reviewed

      Description

      Fuse-dfs should cache fs handles on a per-user basis. This significantly increases performance, and has the side effect of fixing the current code which leaks fs handles.

      The original bug description follows:

      I run the following test:

      1. Run hadoop DFS in single node mode
      2. start up fuse_dfs
      3. copy my source tree, about 250 megs, into the DFS
      cp -av * /mnt/hdfs/

      in /var/log/messages I keep seeing:

      Dec 22 09:02:08 bodum fuse_dfs: ERROR: hdfs trying to utime /bar/backend-trunk2/src/machinery/hadoop/output/2008/11/19 to 1229385138/1229963739

      and then eventually

      Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1333
      Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1333
      Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037
      Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1333
      Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037
      Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1333
      Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1209
      Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037
      Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037
      Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037
      Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037
      Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037
      Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037
      Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037
      Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037
      Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037
      Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1209
      Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037
      Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037
      Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037
      Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1333
      Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1209
      Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037

      and the file system hangs. hadoop is still running and I don't see any errors in it's logs. I have to unmount the dfs and restart fuse_dfs and then everything is fine again. At some point I see the following messages in the /var/log/messages:

      ERROR: dfs problem - could not close file_handle(139677114350528) for /bar/backend-trunk2/src/machinery/hadoop/input/2008/12/14/actionrecordlog-8339-93825052368848-1229278807.log fuse_dfs.c:1464
      Dec 22 09:04:49 bodum fuse_dfs: ERROR: dfs problem - could not close file_handle(139676770220176) for /bar/backend-trunk2/src/machinery/hadoop/input/2008/12/14/actionrecordlog-8140-93825025883216-1229278759.log fuse_dfs.c:1464
      Dec 22 09:05:13 bodum fuse_dfs: ERROR: dfs problem - could not close file_handle(139677114812832) for /bar/backend-trunk2/src/machinery/hadoop/input/2008/12/14/actionrecordlog-8138-93825070138960-1229251587.log fuse_dfs.c:1464

      Is this a known issue? Am I just flooding the system too much. All of this is being performed on a single, dual core, machine.

      Thanks!
      ttyl
      Dima

      1. fuse_dfs_020_memleaks.patch
        25 kB
        Brian Bockelman
      2. fuse_dfs_020_memleaks_v3.patch
        20 kB
        Brian Bockelman
      3. fuse_dfs_020_memleaks_v8.patch
        24 kB
        Brian Bockelman
      4. hdfs-420-1.patch
        41 kB
        Eli Collins
      5. hdfs-420-2.patch
        40 kB
        Eli Collins
      6. hdfs-420-3.patch
        42 kB
        Eli Collins

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #702 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/702/)
          HDFS-420. Fuse-dfs should cache fs handles. Contributed by Brian Bockelman and Eli Collins

          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1137675
          Files :

          • /hadoop/common/trunk/hdfs/src/contrib/build-contrib.xml
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_unlink.c
          • /hadoop/common/trunk/hdfs/CHANGES.txt
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_getattr.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_release.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_utimens.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_options.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_stat_struct.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_dfs_wrapper.sh
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_rename.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_mkdir.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_statfs.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_rmdir.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/build.xml
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_users.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_init.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_access.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/configure.ac
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_truncate.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_connect.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_readdir.c
          • /hadoop/common/trunk/hdfs/src/contrib/build.xml
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_open.c
          • /hadoop/common/trunk/hdfs/src/c++/libhdfs/hdfs.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_connect.h
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_dfs.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_chmod.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_chown.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_context_handle.h
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_dfs.h
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #702 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/702/ ) HDFS-420 . Fuse-dfs should cache fs handles. Contributed by Brian Bockelman and Eli Collins eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1137675 Files : /hadoop/common/trunk/hdfs/src/contrib/build-contrib.xml /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_unlink.c /hadoop/common/trunk/hdfs/CHANGES.txt /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_getattr.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_release.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_utimens.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_options.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_stat_struct.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_dfs_wrapper.sh /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_rename.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_mkdir.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_statfs.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_rmdir.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/build.xml /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_users.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_init.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_access.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/configure.ac /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_truncate.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_connect.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_readdir.c /hadoop/common/trunk/hdfs/src/contrib/build.xml /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_open.c /hadoop/common/trunk/hdfs/src/c++/libhdfs/hdfs.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_connect.h /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_dfs.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_chmod.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_chown.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_context_handle.h /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_dfs.h
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #751 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/751/)
          HDFS-420. Fuse-dfs should cache fs handles. Contributed by Brian Bockelman and Eli Collins

          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1137675
          Files :

          • /hadoop/common/trunk/hdfs/src/contrib/build-contrib.xml
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_unlink.c
          • /hadoop/common/trunk/hdfs/CHANGES.txt
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_getattr.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_release.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_utimens.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_options.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_stat_struct.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_dfs_wrapper.sh
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_rename.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_mkdir.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_statfs.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_rmdir.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/build.xml
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_users.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_init.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_access.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/configure.ac
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_truncate.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_connect.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_readdir.c
          • /hadoop/common/trunk/hdfs/src/contrib/build.xml
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_open.c
          • /hadoop/common/trunk/hdfs/src/c++/libhdfs/hdfs.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_connect.h
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_dfs.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_chmod.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_chown.c
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_context_handle.h
          • /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_dfs.h
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #751 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/751/ ) HDFS-420 . Fuse-dfs should cache fs handles. Contributed by Brian Bockelman and Eli Collins eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1137675 Files : /hadoop/common/trunk/hdfs/src/contrib/build-contrib.xml /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_unlink.c /hadoop/common/trunk/hdfs/CHANGES.txt /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_getattr.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_release.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_utimens.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_options.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_stat_struct.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_dfs_wrapper.sh /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_rename.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_mkdir.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_statfs.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_rmdir.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/build.xml /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_users.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_init.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_access.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/configure.ac /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_truncate.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_connect.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_readdir.c /hadoop/common/trunk/hdfs/src/contrib/build.xml /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_open.c /hadoop/common/trunk/hdfs/src/c++/libhdfs/hdfs.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_connect.h /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_dfs.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_chmod.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_chown.c /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_context_handle.h /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_dfs.h
          Hide
          Eli Collins added a comment -

          I've committed this. Thanks Brian and Todd.

          Show
          Eli Collins added a comment - I've committed this. Thanks Brian and Todd.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12483012/hdfs-420-3.patch
          against trunk revision 1136975.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 7 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/794//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/794//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/794//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12483012/hdfs-420-3.patch against trunk revision 1136975. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/794//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/794//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/794//console This message is automatically generated.
          Hide
          Eli Collins added a comment -

          Rebased on trunk. Needed to update build-contrib.xml to work after the build/webapps reorg.

          Show
          Eli Collins added a comment - Rebased on trunk. Needed to update build-contrib.xml to work after the build/webapps reorg.
          Hide
          Todd Lipcon added a comment -

          +1, looks good to me. One typo you can fix when you commit: "allocate a has table" instead of "hash table"

          Show
          Todd Lipcon added a comment - +1, looks good to me. One typo you can fix when you commit: "allocate a has table" instead of "hash table"
          Hide
          Eli Collins added a comment -

          I'm hoping future versions of libhdfs will help out here, so I'd like to keep things factored out into doConnect and doDisconnect. That way we can revisit ref-counting in the future if we'd like.

          Agree - I left all the connect/disconnect logic as is. I removed the ref counting code that I added in the previous patch.

          Lemme know if you hit any snags. I've tested this a bit and it's working well for me.

          Show
          Eli Collins added a comment - I'm hoping future versions of libhdfs will help out here, so I'd like to keep things factored out into doConnect and doDisconnect. That way we can revisit ref-counting in the future if we'd like. Agree - I left all the connect/disconnect logic as is. I removed the ref counting code that I added in the previous patch. Lemme know if you hit any snags. I've tested this a bit and it's working well for me.
          Hide
          Brian Bockelman added a comment -

          Hi,

          I added the ref-counting in there originally because I thought I could get a refcounting scheme to work. I really couldn't come up with a way to safely disconnect the filesystem. OTOH, there really seems to be no reason to disconnect the filesystem: we've had mounts lasting for a month.

          I'm hoping future versions of libhdfs will help out here, so I'd like to keep things factored out into doConnect and doDisconnect. That way we can revisit ref-counting in the future if we'd like.

          When I get back from vacation, I'll put Eli's final patch through the paces locally.

          Brian

          Show
          Brian Bockelman added a comment - Hi, I added the ref-counting in there originally because I thought I could get a refcounting scheme to work. I really couldn't come up with a way to safely disconnect the filesystem. OTOH, there really seems to be no reason to disconnect the filesystem: we've had mounts lasting for a month. I'm hoping future versions of libhdfs will help out here, so I'd like to keep things factored out into doConnect and doDisconnect. That way we can revisit ref-counting in the future if we'd like. When I get back from vacation, I'll put Eli's final patch through the paces locally. Brian
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12480755/hdfs-420-2.patch
          against trunk revision 1128542.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/657//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/657//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/657//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480755/hdfs-420-2.patch against trunk revision 1128542. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/657//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/657//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/657//console This message is automatically generated.
          Hide
          Eli Collins added a comment -

          Updated patch, rebased on trunk.

          Show
          Eli Collins added a comment - Updated patch, rebased on trunk.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12480753/hdfs-420-2.patch
          against trunk revision 1128542.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/656//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480753/hdfs-420-2.patch against trunk revision 1128542. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/656//console This message is automatically generated.
          Hide
          Eli Collins added a comment -

          If caching handles we don't need the ref counting, I left the code in to show how alloc/free'ing handles could work, but since we're good not using that I'll remove it entirely. This also simplifies insertFs since we don't need to worry about checking for an entry with a free'd handles.

          Since we only create a handle on the first access by a user and it takes ~500ms to create a handle I think it's OK to block searching for an existing handle on creating a new handle. Currently different users will serialize on the table lock when searching for their cached handle. We could use a rw lock to get around that but I don't think it will be an issue in practice.

          Updated patch attach.

          Show
          Eli Collins added a comment - If caching handles we don't need the ref counting, I left the code in to show how alloc/free'ing handles could work, but since we're good not using that I'll remove it entirely. This also simplifies insertFs since we don't need to worry about checking for an entry with a free'd handles. Since we only create a handle on the first access by a user and it takes ~500ms to create a handle I think it's OK to block searching for an existing handle on creating a new handle. Currently different users will serialize on the table lock when searching for their cached handle. We could use a rw lock to get around that but I don't think it will be an issue in practice. Updated patch attach.
          Hide
          Todd Lipcon added a comment -

          A little confused here – the doConnectAsUser function does reference counting in the hashtable entries, but doDisconnect has ifdeffed out the code that decrements the reference counts and disconnects. So, it seems from the code and the discussion that the code is attempting to keep a connection around for the lifetime of the fuse mount for each user.

          Given that, I think it's better to remove the refcounting code entirely, or ifdef it throughout with a define like #define USE_FS_REFCOUNTING.

          In terms of locking granularity, do we care that one user's connection process may hold up another user who is already cached?

          Show
          Todd Lipcon added a comment - A little confused here – the doConnectAsUser function does reference counting in the hashtable entries, but doDisconnect has ifdeffed out the code that decrements the reference counts and disconnects. So, it seems from the code and the discussion that the code is attempting to keep a connection around for the lifetime of the fuse mount for each user. Given that, I think it's better to remove the refcounting code entirely, or ifdef it throughout with a define like #define USE_FS_REFCOUNTING. In terms of locking granularity, do we care that one user's connection process may hold up another user who is already cached?
          Hide
          Eli Collins added a comment -

          Test failure is unrelated. TestFuseDFS doesn't run by default, but I ran it on my host where it passes.

          Show
          Eli Collins added a comment - Test failure is unrelated. TestFuseDFS doesn't run by default, but I ran it on my host where it passes.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12480494/hdfs-420-1.patch
          against trunk revision 1127759.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.hdfs.TestDFSStorageStateRecovery

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/633//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/633//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/633//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480494/hdfs-420-1.patch against trunk revision 1127759. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.TestDFSStorageStateRecovery +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/633//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/633//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/633//console This message is automatically generated.
          Hide
          Eli Collins added a comment -

          Latest patch looks good. I agree we should cache fs handles in fuse-dfs. In the updated patch attached I refactored out functions for creating the fs handle table, finding and inserting references into it. I also implemented doDisconnect, it works (I bet the segv you saw was from trying to free an entry in the table when the ref-count drops to zero) however the performance is much worse. I did some measurements comparing hdfsConnectAsUserNewInstance vs hdfsConnectAsUser and they perform similarly. The initial call is always slow because it creates a JVM, but after that it looks like the bulk of the time is spent loading configuration, the FileSystem cache in the Java client doesn't make a difference (even when the client and NN are a hop away). So it turns out the big performance gain comes from caching the fs handle in fuse-dfs vs getting a handle from libHdfs. Since eg an ls can result in many fuse-dfs operations it really pays to cache the handle.

          The updated patch also contains the following additional changes:

          • Improved the fix for HDFS-1757.
          • Modifed doConnectAsUser to return the fs rather than use a IN/OUT arg.
          • Made some the ERRORs in fuse_options.c INFOs since they're not error cases
          • Removed the dead code in fuse_impls_access.c. This is HDFS-428.
          • Removed the non-PERMS code since this no longer compiles, is not used.
          • Removed the auto-connect in dfs_init since connection testing is already done in main.
          • Minor update fuse_dfs_wrapper.sh to reflect the new ivy dir names.
          • Removed doConnect, it's only used once and we don't want to encourage more uses.

          I verified TestFuseDFS passes. Also did some stress testing of a fuse-dfs mount using multiple clients/users on a pseudo-cluster running from a build of trunk.

          Mind taking a look at the latest code?

          Show
          Eli Collins added a comment - Latest patch looks good. I agree we should cache fs handles in fuse-dfs. In the updated patch attached I refactored out functions for creating the fs handle table, finding and inserting references into it. I also implemented doDisconnect, it works (I bet the segv you saw was from trying to free an entry in the table when the ref-count drops to zero) however the performance is much worse. I did some measurements comparing hdfsConnectAsUserNewInstance vs hdfsConnectAsUser and they perform similarly. The initial call is always slow because it creates a JVM, but after that it looks like the bulk of the time is spent loading configuration, the FileSystem cache in the Java client doesn't make a difference (even when the client and NN are a hop away). So it turns out the big performance gain comes from caching the fs handle in fuse-dfs vs getting a handle from libHdfs. Since eg an ls can result in many fuse-dfs operations it really pays to cache the handle. The updated patch also contains the following additional changes: Improved the fix for HDFS-1757 . Modifed doConnectAsUser to return the fs rather than use a IN/OUT arg. Made some the ERRORs in fuse_options.c INFOs since they're not error cases Removed the dead code in fuse_impls_access.c. This is HDFS-428 . Removed the non-PERMS code since this no longer compiles, is not used. Removed the auto-connect in dfs_init since connection testing is already done in main. Minor update fuse_dfs_wrapper.sh to reflect the new ivy dir names. Removed doConnect, it's only used once and we don't want to encourage more uses. I verified TestFuseDFS passes. Also did some stress testing of a fuse-dfs mount using multiple clients/users on a pseudo-cluster running from a build of trunk. Mind taking a look at the latest code?
          Hide
          Brock Noland added a comment -

          Looks like hudson could not apply the patch due to the directory structure.

          I cannot speak to the patches correctness, but I manually applied and saw great improvement in terms memory. Setting -Xmx128m caused a repeatable OOM in about 5 minutes reading about 1GB of data from 800 files. With the patch, my test has been running for an hour without an OOM.

          Show
          Brock Noland added a comment - Looks like hudson could not apply the patch due to the directory structure. I cannot speak to the patches correctness, but I manually applied and saw great improvement in terms memory. Setting -Xmx128m caused a repeatable OOM in about 5 minutes reading about 1GB of data from 800 files. With the patch, my test has been running for an hour without an OOM.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12479821/fuse_dfs_020_memleaks_v8.patch
          against trunk revision 1125057.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/589//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479821/fuse_dfs_020_memleaks_v8.patch against trunk revision 1125057. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/589//console This message is automatically generated.
          Hide
          Brian Bockelman added a comment -

          Ok, I tested this one for awhile prior to posting it. We have been running this on our ~2PB cluster with 250 machines for around 2-3 weeks.

          No crashes have been reported. No memory leaks are observed. Unit tests pass. Site admins report they are much happier with FUSE

          Show
          Brian Bockelman added a comment - Ok, I tested this one for awhile prior to posting it. We have been running this on our ~2PB cluster with 250 machines for around 2-3 weeks. No crashes have been reported. No memory leaks are observed. Unit tests pass. Site admins report they are much happier with FUSE
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12473592/fuse_dfs_020_memleaks_v3.patch
          against trunk revision 1094748.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/389//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12473592/fuse_dfs_020_memleaks_v3.patch against trunk revision 1094748. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/389//console This message is automatically generated.
          Hide
          Brian Bockelman added a comment -

          There seems to be threading issues in libhdfs when disconnecting a filesystem - if I create two filesystem handles, disconnect one, then try to do operations with the second, the JVM itself sigsegv's and dies. Probably needs investigation on another ticket.

          Hence, I created this new patch. It creates a global hashtable based upon username and only has one filesystem object per user. This will permanently hold filesystem objects, but no longer creates a FS per operation, which caused the original issue. It also addresses the major slowdown issues with creating/releasing filesystem objects per operation. Doing a "ls -lhR" is more reasonable.

          I believe Eli's other comments are addressed.

          Show
          Brian Bockelman added a comment - There seems to be threading issues in libhdfs when disconnecting a filesystem - if I create two filesystem handles, disconnect one, then try to do operations with the second, the JVM itself sigsegv's and dies. Probably needs investigation on another ticket. Hence, I created this new patch. It creates a global hashtable based upon username and only has one filesystem object per user. This will permanently hold filesystem objects, but no longer creates a FS per operation, which caused the original issue. It also addresses the major slowdown issues with creating/releasing filesystem objects per operation. Doing a "ls -lhR" is more reasonable. I believe Eli's other comments are addressed.
          Hide
          Eli Collins added a comment -

          Hey Brian,

          Good catch. The patch does two things which are probably better implemented as separate patches, let me know what you think.

          1. Removes the ability to compile-out perms with the libhdfs.noperms option. Should be orthogonal to #2, think we should do this, but in a separate patch. IIUC the code is currently optional (though compiled in by default) so people can disable it for performance. I think we can always have perms enabled in fuse-dfs but a better way to do this is use the reentrant versions of the relevant functions (and beware of HADOOP-7156) rather than introduce new locks. Btw if we're removing PERM we also need to remove the code that sets it in src/contrib/fuse-dfs/build.xml.
          2. Makes each fuse-dfs operation get (a new) and release FS handles. While the change introduces some additional overhead (each operation now needs a new client) we need to do it for correctness (can later make FS handle caching work in another jira).

          How about this jira just covers the patch for #2?

          Some additional comments from looking at the code:

          • In fuse_imples_getattr.c line 37 it looks like this change is still in progress, eg why does it connect unconditionally? You can remove the commented out code on the following line.
          • In fuse_impls_chown.c line 60 we should not reconenct here right?
          • Nit on line 76 of the same file, you don't need to assign null here.

          Thanks,
          Eli

          Show
          Eli Collins added a comment - Hey Brian, Good catch. The patch does two things which are probably better implemented as separate patches, let me know what you think. Removes the ability to compile-out perms with the libhdfs.noperms option. Should be orthogonal to #2, think we should do this, but in a separate patch. IIUC the code is currently optional (though compiled in by default) so people can disable it for performance. I think we can always have perms enabled in fuse-dfs but a better way to do this is use the reentrant versions of the relevant functions (and beware of HADOOP-7156 ) rather than introduce new locks. Btw if we're removing PERM we also need to remove the code that sets it in src/contrib/fuse-dfs/build.xml. Makes each fuse-dfs operation get (a new) and release FS handles. While the change introduces some additional overhead (each operation now needs a new client) we need to do it for correctness (can later make FS handle caching work in another jira). How about this jira just covers the patch for #2? Some additional comments from looking at the code: In fuse_imples_getattr.c line 37 it looks like this change is still in progress, eg why does it connect unconditionally? You can remove the commented out code on the following line. In fuse_impls_chown.c line 60 we should not reconenct here right? Nit on line 76 of the same file, you don't need to assign null here. Thanks, Eli
          Hide
          Brian Bockelman added a comment -

          Attaching a new version of the memleaks patch; previous one had some thread safety issues.

          Hammering it on our local cluster; we'll have some feedback in a few days.

          Going to hit submit patch to verify everything build Apache-side, and so someone can start to review it.

          Show
          Brian Bockelman added a comment - Attaching a new version of the memleaks patch; previous one had some thread safety issues. Hammering it on our local cluster; we'll have some feedback in a few days. Going to hit submit patch to verify everything build Apache-side, and so someone can start to review it.
          Hide
          Brian Bockelman added a comment -

          The attached patch changes all uses of hdfsConnect to hdfsConnectNewInstance and hdfsConnectAsUser to hdfsConnectAsUserNewInstance.

          Thus, since we own the FileSystem instance, we can call hdfsDisconnect and no longer leak the handles.

          This requires either 0.21 or HDFS-1000.

          Show
          Brian Bockelman added a comment - The attached patch changes all uses of hdfsConnect to hdfsConnectNewInstance and hdfsConnectAsUser to hdfsConnectAsUserNewInstance. Thus, since we own the FileSystem instance, we can call hdfsDisconnect and no longer leak the handles. This requires either 0.21 or HDFS-1000 .
          Hide
          Brian Bockelman added a comment -

          Marking this as blocked by HDFS-422. In 0.20 (sadly, I fixed this in my local copy of 0.19 and never invested in a clean patch), we leak one FileSystem handle per file open. This is the most likely source of your issue.

          Show
          Brian Bockelman added a comment - Marking this as blocked by HDFS-422 . In 0.20 (sadly, I fixed this in my local copy of 0.19 and never invested in a clean patch), we leak one FileSystem handle per file open. This is the most likely source of your issue.
          Hide
          Brian Bockelman added a comment -

          Hey Dima,

          First, try applying HDFS-464 to alleviate some memory leaks present in libhdfs.

          Then, try the attached file; this releases the reference to the FileSystem.

          With this, we have been able to run FUSE-DFS stably for weeks at a time. I've taken a heap dump of the resulting system and not found any more obvious leaks (with these two patches, albeit on Hadoop-0.19.x versions of them, I was able to set the heap size to 4MB and create tens of thousands of files).

          To debug better, repeat your test with fuse_dfs in a second terminal with the "-d" option to make it stay in the foreground. In this case, you will be able to capture the error messages Hadoop spits out. Alternately, you can muck around with the log4j settings and get these to spit out to a log file.

          Brian

          Show
          Brian Bockelman added a comment - Hey Dima, First, try applying HDFS-464 to alleviate some memory leaks present in libhdfs. Then, try the attached file; this releases the reference to the FileSystem. With this, we have been able to run FUSE-DFS stably for weeks at a time. I've taken a heap dump of the resulting system and not found any more obvious leaks (with these two patches, albeit on Hadoop-0.19.x versions of them, I was able to set the heap size to 4MB and create tens of thousands of files). To debug better, repeat your test with fuse_dfs in a second terminal with the "-d" option to make it stay in the foreground. In this case, you will be able to capture the error messages Hadoop spits out. Alternately, you can muck around with the log4j settings and get these to spit out to a log file. Brian
          Hide
          Zhang Bingjun added a comment -

          One memory leak problem were found in function hdfsFreeFileInfo() of file hdfs.c (under c++/libhdfs/). The bug can be easily fixed as in this issue:
          https://issues.apache.org/jira/browse/HDFS-596

          In my test, before fixing the bug, 1GB memory will be exhausted after writing 14000 files. After fixing the bug, only 34MB memory were exhausted after writing 18000 files.

          Another bug in fuse-dfs is the failure of releasing file system handle hdfsFS. hdfsFS is a global reference in the JNI code of libhdfs. The failure of releasing it will at least resulting the memory leak in storing the global reference itself and underlying java object is points to. The current implementation of fuse-dfs opens hdfs and generate a global reference (hdfsFS) for each file write/read operation. If the underlying objects (FS) are shared, the memory leak may mainly come from storing the global reference itself in Java VM.

          I am still thinking how to come up with a nice way to release the global reference (hdfsFS). Will update you here once I have something.

          Thanks!

          Show
          Zhang Bingjun added a comment - One memory leak problem were found in function hdfsFreeFileInfo() of file hdfs.c (under c++/libhdfs/). The bug can be easily fixed as in this issue: https://issues.apache.org/jira/browse/HDFS-596 In my test, before fixing the bug, 1GB memory will be exhausted after writing 14000 files. After fixing the bug, only 34MB memory were exhausted after writing 18000 files. Another bug in fuse-dfs is the failure of releasing file system handle hdfsFS. hdfsFS is a global reference in the JNI code of libhdfs. The failure of releasing it will at least resulting the memory leak in storing the global reference itself and underlying java object is points to. The current implementation of fuse-dfs opens hdfs and generate a global reference (hdfsFS) for each file write/read operation. If the underlying objects (FS) are shared, the memory leak may mainly come from storing the global reference itself in Java VM. I am still thinking how to come up with a nice way to release the global reference (hdfsFS). Will update you here once I have something. Thanks!
          Hide
          Zhang Bingjun added a comment -

          Hi Craig,

          You are right. The fuse-dfs process grows in size when I am writing continuously into HDFS through FUSE.

          In my test, I installed hadoop-0.20.0 on a 11 node cluster with one namenode host, one secondary host, and 9 datanode hosts all with ubuntu 9.04 linux. The namenode has 1GB memory. The script writing file into HDFS through FUSE was also running on the namenode. When 12418 files with 100K each were wrote (just used small files to write as many as possible in a short time), the fuse-dfs exhausted all the 1G memory and then died out. During the writing, the occupied memory of fuse-dfs kept growing.

          I guess the issue you mentioned about releasing hdfsFS is one cause. I am not sure whether there are still other places causing the memory leak.

          I am reading the libhdfs and fuse-dfs code now. And hopefully I can get familiar with the thing and propose some fixes. I will see. But if anyone also encountered this issue and is will to discuss how to solve it. Please give your input here. Really hope we can make fuse-dfs a stable tool to use so that HDFS will appear to linux as a nicely mounted local directory.

          Thanks!
          Bingjun

          Show
          Zhang Bingjun added a comment - Hi Craig, You are right. The fuse-dfs process grows in size when I am writing continuously into HDFS through FUSE. In my test, I installed hadoop-0.20.0 on a 11 node cluster with one namenode host, one secondary host, and 9 datanode hosts all with ubuntu 9.04 linux. The namenode has 1GB memory. The script writing file into HDFS through FUSE was also running on the namenode. When 12418 files with 100K each were wrote (just used small files to write as many as possible in a short time), the fuse-dfs exhausted all the 1G memory and then died out. During the writing, the occupied memory of fuse-dfs kept growing. I guess the issue you mentioned about releasing hdfsFS is one cause. I am not sure whether there are still other places causing the memory leak. I am reading the libhdfs and fuse-dfs code now. And hopefully I can get familiar with the thing and propose some fixes. I will see. But if anyone also encountered this issue and is will to discuss how to solve it. Please give your input here. Really hope we can make fuse-dfs a stable tool to use so that HDFS will appear to linux as a nicely mounted local directory. Thanks! Bingjun
          Hide
          Zhang Bingjun added a comment -

          I also encountered this bug.

          After writing / reading about 10000 small files in HDFS through fuse_dfs. The mounting point got disconnected. I had to umount and re-mount HDFS using fuse_dfs again.

          Another problem is that if I do list a particular directory, the returned message first tells me the files do not exist and then list the files out. The error message always comes out first when I run list command like below:

          hadoop@hadoop-001:~/hadoop-hdfs/user/hadoop$ ls -l
          ls: cannot access p/test: No such file or directory
          ls: cannot access p/test2: No such file or directory
          total 0
          d????????? ? ? ? ? ? p/test
          d????????? ? ? ? ? ? p/test2

          Another problem with my fuse_dfs is that the permission rights cannot be displayed correctly. Only "???" were shown in the permission right field like the above example.

          Yet another problem is that the df -kh command does not display the right information. For my case, the displayed info is as follows:

          hadoop@hadoop-001:~$ df -kh hadoop-hdfs
          Filesystem Size Used Avail Use% Mounted on
          fuse_dfs 1.2G -1024Y -2.0G 100% /home/hadoop/hadoop-hdfs

          May I know where these bug might come from? Or it is because my local compiled fuse_dfs problem? Thanks!

          Show
          Zhang Bingjun added a comment - I also encountered this bug. After writing / reading about 10000 small files in HDFS through fuse_dfs. The mounting point got disconnected. I had to umount and re-mount HDFS using fuse_dfs again. Another problem is that if I do list a particular directory, the returned message first tells me the files do not exist and then list the files out. The error message always comes out first when I run list command like below: hadoop@hadoop-001:~/hadoop-hdfs/user/hadoop$ ls -l ls: cannot access p/test: No such file or directory ls: cannot access p/test2: No such file or directory total 0 d????????? ? ? ? ? ? p/test d????????? ? ? ? ? ? p/test2 Another problem with my fuse_dfs is that the permission rights cannot be displayed correctly. Only "???" were shown in the permission right field like the above example. Yet another problem is that the df -kh command does not display the right information. For my case, the displayed info is as follows: hadoop@hadoop-001:~$ df -kh hadoop-hdfs Filesystem Size Used Avail Use% Mounted on fuse_dfs 1.2G -1024Y -2.0G 100% /home/hadoop/hadoop-hdfs May I know where these bug might come from? Or it is because my local compiled fuse_dfs problem? Thanks!
          Hide
          Craig Macdonald added a comment -

          The more I think about this, it does look like doConnectAsUser() need to maintain a cache of FS connections.

          At present, we in effect to hdfsConnectAsUser for each fuse-dfs operation (except read/write). In doing so, a Configuration object is created, used to obtain the correct file system, and then destroyed. It is true that FileSystem.get(URI,Configuration) does cache filesystems, however we create a new Configuration object for each call, with related expenses in parsing the XML configuration files etc (if I read the source correctly).

          However, in libhdfs, hdfsConnectAsUser creates a new global reference of type hdfsFS (i.e. a malloc) for every call, regardless of whether the returned FileSystem already had an existing global reference or not.

          I believe that fuse-dfs should maintain a cache of filesystem hdfsFS handles on a user specific basis. This should allow considerably faster fs operations. Pete, I believe you suggested ghashtable? We need a Map<uid_t, hdfsFS> in C effectively. Clearing out the cache would be minor issue.

          An alternative would be to add a hdfsFSFree function to libhdfs so that we dont leak hdfsFS handles. Performing hdfsDisconnect is not the correct course of action, as this will close the Java FileSystem object.

          I'm not sure that this solves the problem that Dima reports, however, I do believe there is a problem here.

          Show
          Craig Macdonald added a comment - The more I think about this, it does look like doConnectAsUser() need to maintain a cache of FS connections. At present, we in effect to hdfsConnectAsUser for each fuse-dfs operation (except read/write). In doing so, a Configuration object is created, used to obtain the correct file system, and then destroyed. It is true that FileSystem.get(URI,Configuration) does cache filesystems, however we create a new Configuration object for each call, with related expenses in parsing the XML configuration files etc (if I read the source correctly). However, in libhdfs, hdfsConnectAsUser creates a new global reference of type hdfsFS (i.e. a malloc) for every call, regardless of whether the returned FileSystem already had an existing global reference or not. I believe that fuse-dfs should maintain a cache of filesystem hdfsFS handles on a user specific basis. This should allow considerably faster fs operations. Pete, I believe you suggested ghashtable? We need a Map<uid_t, hdfsFS> in C effectively. Clearing out the cache would be minor issue. An alternative would be to add a hdfsFSFree function to libhdfs so that we dont leak hdfsFS handles. Performing hdfsDisconnect is not the correct course of action, as this will close the Java FileSystem object. I'm not sure that this solves the problem that Dima reports, however, I do believe there is a problem here.
          Hide
          Craig Macdonald added a comment -

          Dima,

          It's not the case that a connection is made for each FS call. Instead, in Java, Hadoop caches file systems for each each user that uses the file system, so the handle returned by libhdfs should have the same underlying FileSystem object. Did you fuse_dfs object grow to be very large in size?

          However, would be interesting to identify what's happening to fuse_dfs in this case. Are there corresponding error messages in your namenode/datanode(s), or can you increase the logging level of hadoop to debug?

          Show
          Craig Macdonald added a comment - Dima, It's not the case that a connection is made for each FS call. Instead, in Java, Hadoop caches file systems for each each user that uses the file system, so the handle returned by libhdfs should have the same underlying FileSystem object. Did you fuse_dfs object grow to be very large in size? However, would be interesting to identify what's happening to fuse_dfs in this case. Are there corresponding error messages in your namenode/datanode(s), or can you increase the logging level of hadoop to debug?
          Hide
          Dima Brodsky added a comment -

          Taking a closer look at fuse-dfs we are creating a connection for each filesystem call, but we are not doing an hdfsDisconnect afterwards. Should we not be? Could we be potentially be running out of sockets/file descriptors?

          ttyl
          Dima

          Show
          Dima Brodsky added a comment - Taking a closer look at fuse-dfs we are creating a connection for each filesystem call, but we are not doing an hdfsDisconnect afterwards. Should we not be? Could we be potentially be running out of sockets/file descriptors? ttyl Dima

            People

            • Assignee:
              Brian Bockelman
              Reporter:
              Dima Brodsky
            • Votes:
              2 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development