Details

    • Release Note:
      Hide
      Introduced FUSE module for HDFS. Module allows mount of HDFS as a Unix filesystem, and optionally the export of that mount point to other machines. Writes are disabled. rmdir, mv, mkdir, rm are supported, but not cp, touch, and the like. Usage information is attached to the Jira record.

      Show
      Introduced FUSE module for HDFS. Module allows mount of HDFS as a Unix filesystem, and optionally the export of that mount point to other machines. Writes are disabled. rmdir, mv, mkdir, rm are supported, but not cp, touch, and the like. Usage information is attached to the Jira record.

      Description

      This is a FUSE module for Hadoop's HDFS.

      It allows one to mount HDFS as a Unix filesystem and optionally export
      that mount point to other machines.

      rmdir, mv, mkdir, rm are all supported. just not cp, touch, ..., but actual writes require: https://issues.apache.org/jira/browse/HADOOP-3485

      For the most up-to-date documentation, see: http://wiki.apache.org/hadoop/MountableHDFS

      BUILDING:

      Requirements:

      1. a Linux kernel > 2.6.9 or a kernel module from FUSE - i.e., you
      compile it yourself and then modprobe it. Better off with the
      former option if possible. (Note for now if you use the kernel
      with fuse included, it doesn't allow you to export this through NFS
      so be warned. See the FUSE email list for more about this.)

      2. FUSE should be installed in /usr/local or FUSE_HOME ant
      environment variable

      To build:

      1. in HADOOP_HOME: ant compile-contrib -Dcompile.c++=1 -Dfusedfs=1 -Dlibhdfs=1

      NOTE: for amd64 architecture, libhdfs will not compile unless you edit
      the Makefile in src/c++/libhdfs/Makefile and set OS_ARCH=amd64
      (probably the same for others too).

      --------------------------------------------------------------------------------

      CONFIGURING:

      Look at all the paths in fuse_dfs_wrapper.sh and either correct them
      or set them in your environment before running. (note for automount
      and mount as root, you probably cannnot control the environment, so
      best to set them in the wrapper)

      INSTALLING:

      1. mkdir /mnt/dfs (or wherever you want to mount it)

      2. fuse_dfs_wrapper.sh dfs://hadoop_server1.foo.com:9000 /mnt/dfs -d
      ; and from another terminal, try ls /mnt/dfs

      If 2 works, try again dropping the debug mode, i.e., -d

      (note - common problems are that you don't have libhdfs.so or
      libjvm.so or libfuse.so on your LD_LIBRARY_PATH, and your CLASSPATH
      does not contain hadoop and other required jars.)

      --------------------------------------------------------------------------------

      DEPLOYING:

      in a root shell do the following:

      1. add the following to /etc/fstab -
      fuse_dfs#dfs://hadoop_server.foo.com:9000 /mnt/dfs fuse
      allow_other,rw 0 0

      2. mount /mnt/dfs Expect problems with not finding fuse_dfs. You will
      need to probably add this to /sbin and then problems finding the
      above 3 libraries. Add these using ldconfig.

      --------------------------------------------------------------------------------

      EXPORTING:

      Add the following to /etc/exports:

      /mnt/hdfs *.foo.com(no_root_squash,rw,fsid=1,sync)

      NOTE - you cannot export this with a FUSE module built into the kernel

      • e.g., kernel 2.6.17. For info on this, refer to the FUSE wiki.
        --------------------------------------------------------------------------------

      ADVANCED:

      you may want to ensure certain directories cannot be deleted from the
      shell until the FS has permissions. You can set this in the build.xml
      file in src/contrib/fuse-dfs/build.xml

      1. patch6.txt
        94 kB
        Pete Wyckoff
      2. HADOOP-4.patch
        65 kB
        Doug Cutting
      3. HADOOP-4.patch
        64 kB
        Pete Wyckoff
      4. HADOOP-4.patch
        63 kB
        Pete Wyckoff
      5. HADOOP-4.patch
        63 kB
        Pete Wyckoff
      6. HADOOP-4.patch
        95 kB
        Doug Cutting
      7. patch6.txt
        94 kB
        Pete Wyckoff
      8. patch5.txt
        94 kB
        Pete Wyckoff
      9. patch4.txt
        94 kB
        Pete Wyckoff
      10. HADOOP-4.patch
        95 kB
        Doug Cutting
      11. HADOOP-4.patch
        95 kB
        Doug Cutting
      12. patch4.txt
        94 kB
        Pete Wyckoff
      13. patch4.txt
        94 kB
        Pete Wyckoff
      14. patch3.txt
        97 kB
        Pete Wyckoff
      15. patch2.txt
        61 kB
        Pete Wyckoff
      16. patch.txt
        61 kB
        Pete Wyckoff
      17. patch.txt
        80 kB
        Pete Wyckoff
      18. fuse_dfs.tar.gz
        21 kB
        Pete Wyckoff
      19. fuse_dfs.c
        25 kB
        Craig Macdonald
      20. fuse-dfs.tar.gz
        172 kB
        Pete Wyckoff
      21. fuse-dfs.tar.gz
        112 kB
        Pete Wyckoff
      22. fuse_dfs.c
        23 kB
        Pete Wyckoff
      23. fuse_dfs.c
        23 kB
        Pete Wyckoff
      24. fuse_dfs.sh
        0.6 kB
        Craig Macdonald
      25. fuse-dfs.tar.gz
        112 kB
        Pete Wyckoff
      26. fuse-dfs.tar.gz
        5 kB
        Pete Wyckoff
      27. Makefile
        0.2 kB
        Sami Siren
      28. fuse_dfs.c
        23 kB
        Pete Wyckoff
      29. fuse-dfs.tar.gz
        5 kB
        Pete Wyckoff
      30. fuse-j-hadoopfs-03.tar.gz
        11 kB
        Anurag Sharma
      31. fuse_dfs.c
        16 kB
        Pete Wyckoff
      32. fuse-hadoop-0.1.0_fuse-j.2.4_hadoop.0.5.0.tar.gz
        27 kB
        Nguyen Quoc Mai
      33. fuse-hadoop-0.1.0_fuse-j.2.2.3_hadoop.0.5.0.tar.gz
        27 kB
        Nguyen Quoc Mai
      34. fuse-hadoop-0.1.1.tar.gz
        5 kB
        John Xing

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Pete Wyckoff
              Reporter:
              John Xing
            • Votes:
              4 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development