Details

    • Release Note:
      Hide
      Introduced FUSE module for HDFS. Module allows mount of HDFS as a Unix filesystem, and optionally the export of that mount point to other machines. Writes are disabled. rmdir, mv, mkdir, rm are supported, but not cp, touch, and the like. Usage information is attached to the Jira record.

      Show
      Introduced FUSE module for HDFS. Module allows mount of HDFS as a Unix filesystem, and optionally the export of that mount point to other machines. Writes are disabled. rmdir, mv, mkdir, rm are supported, but not cp, touch, and the like. Usage information is attached to the Jira record.

      Description

      This is a FUSE module for Hadoop's HDFS.

      It allows one to mount HDFS as a Unix filesystem and optionally export
      that mount point to other machines.

      rmdir, mv, mkdir, rm are all supported. just not cp, touch, ..., but actual writes require: https://issues.apache.org/jira/browse/HADOOP-3485

      For the most up-to-date documentation, see: http://wiki.apache.org/hadoop/MountableHDFS

      BUILDING:

      Requirements:

      1. a Linux kernel > 2.6.9 or a kernel module from FUSE - i.e., you
      compile it yourself and then modprobe it. Better off with the
      former option if possible. (Note for now if you use the kernel
      with fuse included, it doesn't allow you to export this through NFS
      so be warned. See the FUSE email list for more about this.)

      2. FUSE should be installed in /usr/local or FUSE_HOME ant
      environment variable

      To build:

      1. in HADOOP_HOME: ant compile-contrib -Dcompile.c++=1 -Dfusedfs=1 -Dlibhdfs=1

      NOTE: for amd64 architecture, libhdfs will not compile unless you edit
      the Makefile in src/c++/libhdfs/Makefile and set OS_ARCH=amd64
      (probably the same for others too).

      --------------------------------------------------------------------------------

      CONFIGURING:

      Look at all the paths in fuse_dfs_wrapper.sh and either correct them
      or set them in your environment before running. (note for automount
      and mount as root, you probably cannnot control the environment, so
      best to set them in the wrapper)

      INSTALLING:

      1. mkdir /mnt/dfs (or wherever you want to mount it)

      2. fuse_dfs_wrapper.sh dfs://hadoop_server1.foo.com:9000 /mnt/dfs -d
      ; and from another terminal, try ls /mnt/dfs

      If 2 works, try again dropping the debug mode, i.e., -d

      (note - common problems are that you don't have libhdfs.so or
      libjvm.so or libfuse.so on your LD_LIBRARY_PATH, and your CLASSPATH
      does not contain hadoop and other required jars.)

      --------------------------------------------------------------------------------

      DEPLOYING:

      in a root shell do the following:

      1. add the following to /etc/fstab -
      fuse_dfs#dfs://hadoop_server.foo.com:9000 /mnt/dfs fuse
      allow_other,rw 0 0

      2. mount /mnt/dfs Expect problems with not finding fuse_dfs. You will
      need to probably add this to /sbin and then problems finding the
      above 3 libraries. Add these using ldconfig.

      --------------------------------------------------------------------------------

      EXPORTING:

      Add the following to /etc/exports:

      /mnt/hdfs *.foo.com(no_root_squash,rw,fsid=1,sync)

      NOTE - you cannot export this with a FUSE module built into the kernel

      • e.g., kernel 2.6.17. For info on this, refer to the FUSE wiki.
        --------------------------------------------------------------------------------

      ADVANCED:

      you may want to ensure certain directories cannot be deleted from the
      shell until the FS has permissions. You can set this in the build.xml
      file in src/contrib/fuse-dfs/build.xml

      1. fuse_dfs.c
        25 kB
        Craig Macdonald
      2. fuse_dfs.c
        23 kB
        Pete Wyckoff
      3. fuse_dfs.c
        23 kB
        Pete Wyckoff
      4. fuse_dfs.c
        23 kB
        Pete Wyckoff
      5. fuse_dfs.c
        16 kB
        Pete Wyckoff
      6. fuse_dfs.sh
        0.6 kB
        Craig Macdonald
      7. fuse_dfs.tar.gz
        21 kB
        Pete Wyckoff
      8. fuse-dfs.tar.gz
        172 kB
        Pete Wyckoff
      9. fuse-dfs.tar.gz
        112 kB
        Pete Wyckoff
      10. fuse-dfs.tar.gz
        112 kB
        Pete Wyckoff
      11. fuse-dfs.tar.gz
        5 kB
        Pete Wyckoff
      12. fuse-dfs.tar.gz
        5 kB
        Pete Wyckoff
      13. fuse-hadoop-0.1.0_fuse-j.2.2.3_hadoop.0.5.0.tar.gz
        27 kB
        Nguyen Quoc Mai
      14. fuse-hadoop-0.1.0_fuse-j.2.4_hadoop.0.5.0.tar.gz
        27 kB
        Nguyen Quoc Mai
      15. fuse-hadoop-0.1.1.tar.gz
        5 kB
        John Xing
      16. fuse-j-hadoopfs-03.tar.gz
        11 kB
        Anurag Sharma
      17. HADOOP-4.patch
        65 kB
        Doug Cutting
      18. HADOOP-4.patch
        64 kB
        Pete Wyckoff
      19. HADOOP-4.patch
        63 kB
        Pete Wyckoff
      20. HADOOP-4.patch
        63 kB
        Pete Wyckoff
      21. HADOOP-4.patch
        95 kB
        Doug Cutting
      22. HADOOP-4.patch
        95 kB
        Doug Cutting
      23. HADOOP-4.patch
        95 kB
        Doug Cutting
      24. Makefile
        0.2 kB
        Sami Siren
      25. patch.txt
        61 kB
        Pete Wyckoff
      26. patch.txt
        80 kB
        Pete Wyckoff
      27. patch2.txt
        61 kB
        Pete Wyckoff
      28. patch3.txt
        97 kB
        Pete Wyckoff
      29. patch4.txt
        94 kB
        Pete Wyckoff
      30. patch4.txt
        94 kB
        Pete Wyckoff
      31. patch4.txt
        94 kB
        Pete Wyckoff
      32. patch5.txt
        94 kB
        Pete Wyckoff
      33. patch6.txt
        94 kB
        Pete Wyckoff
      34. patch6.txt
        94 kB
        Pete Wyckoff

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Pete Wyckoff
              Reporter:
              John Xing
            • Votes:
              4 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development