Details

    • Release Note:
      Hide
      Introduced FUSE module for HDFS. Module allows mount of HDFS as a Unix filesystem, and optionally the export of that mount point to other machines. Writes are disabled. rmdir, mv, mkdir, rm are supported, but not cp, touch, and the like. Usage information is attached to the Jira record.

      Show
      Introduced FUSE module for HDFS. Module allows mount of HDFS as a Unix filesystem, and optionally the export of that mount point to other machines. Writes are disabled. rmdir, mv, mkdir, rm are supported, but not cp, touch, and the like. Usage information is attached to the Jira record.

      Description

      This is a FUSE module for Hadoop's HDFS.

      It allows one to mount HDFS as a Unix filesystem and optionally export
      that mount point to other machines.

      rmdir, mv, mkdir, rm are all supported. just not cp, touch, ..., but actual writes require: https://issues.apache.org/jira/browse/HADOOP-3485

      For the most up-to-date documentation, see: http://wiki.apache.org/hadoop/MountableHDFS

      BUILDING:

      Requirements:

      1. a Linux kernel > 2.6.9 or a kernel module from FUSE - i.e., you
      compile it yourself and then modprobe it. Better off with the
      former option if possible. (Note for now if you use the kernel
      with fuse included, it doesn't allow you to export this through NFS
      so be warned. See the FUSE email list for more about this.)

      2. FUSE should be installed in /usr/local or FUSE_HOME ant
      environment variable

      To build:

      1. in HADOOP_HOME: ant compile-contrib -Dcompile.c++=1 -Dfusedfs=1 -Dlibhdfs=1

      NOTE: for amd64 architecture, libhdfs will not compile unless you edit
      the Makefile in src/c++/libhdfs/Makefile and set OS_ARCH=amd64
      (probably the same for others too).

      --------------------------------------------------------------------------------

      CONFIGURING:

      Look at all the paths in fuse_dfs_wrapper.sh and either correct them
      or set them in your environment before running. (note for automount
      and mount as root, you probably cannnot control the environment, so
      best to set them in the wrapper)

      INSTALLING:

      1. mkdir /mnt/dfs (or wherever you want to mount it)

      2. fuse_dfs_wrapper.sh dfs://hadoop_server1.foo.com:9000 /mnt/dfs -d
      ; and from another terminal, try ls /mnt/dfs

      If 2 works, try again dropping the debug mode, i.e., -d

      (note - common problems are that you don't have libhdfs.so or
      libjvm.so or libfuse.so on your LD_LIBRARY_PATH, and your CLASSPATH
      does not contain hadoop and other required jars.)

      --------------------------------------------------------------------------------

      DEPLOYING:

      in a root shell do the following:

      1. add the following to /etc/fstab -
      fuse_dfs#dfs://hadoop_server.foo.com:9000 /mnt/dfs fuse
      allow_other,rw 0 0

      2. mount /mnt/dfs Expect problems with not finding fuse_dfs. You will
      need to probably add this to /sbin and then problems finding the
      above 3 libraries. Add these using ldconfig.

      --------------------------------------------------------------------------------

      EXPORTING:

      Add the following to /etc/exports:

      /mnt/hdfs *.foo.com(no_root_squash,rw,fsid=1,sync)

      NOTE - you cannot export this with a FUSE module built into the kernel

      • e.g., kernel 2.6.17. For info on this, refer to the FUSE wiki.
        --------------------------------------------------------------------------------

      ADVANCED:

      you may want to ensure certain directories cannot be deleted from the
      shell until the FS has permissions. You can set this in the build.xml
      file in src/contrib/fuse-dfs/build.xml

        Attachments

        1. patch6.txt
          94 kB
          Pete Wyckoff
        2. HADOOP-4.patch
          65 kB
          Doug Cutting
        3. HADOOP-4.patch
          64 kB
          Pete Wyckoff
        4. HADOOP-4.patch
          63 kB
          Pete Wyckoff
        5. HADOOP-4.patch
          63 kB
          Pete Wyckoff
        6. HADOOP-4.patch
          95 kB
          Doug Cutting
        7. patch6.txt
          94 kB
          Pete Wyckoff
        8. patch5.txt
          94 kB
          Pete Wyckoff
        9. patch4.txt
          94 kB
          Pete Wyckoff
        10. HADOOP-4.patch
          95 kB
          Doug Cutting
        11. HADOOP-4.patch
          95 kB
          Doug Cutting
        12. patch4.txt
          94 kB
          Pete Wyckoff
        13. patch4.txt
          94 kB
          Pete Wyckoff
        14. patch3.txt
          97 kB
          Pete Wyckoff
        15. patch2.txt
          61 kB
          Pete Wyckoff
        16. patch.txt
          61 kB
          Pete Wyckoff
        17. patch.txt
          80 kB
          Pete Wyckoff
        18. fuse_dfs.tar.gz
          21 kB
          Pete Wyckoff
        19. fuse_dfs.c
          25 kB
          Craig Macdonald
        20. fuse-dfs.tar.gz
          172 kB
          Pete Wyckoff
        21. fuse-dfs.tar.gz
          112 kB
          Pete Wyckoff
        22. fuse_dfs.c
          23 kB
          Pete Wyckoff
        23. fuse_dfs.c
          23 kB
          Pete Wyckoff
        24. fuse_dfs.sh
          0.6 kB
          Craig Macdonald
        25. fuse-dfs.tar.gz
          112 kB
          Pete Wyckoff
        26. fuse-dfs.tar.gz
          5 kB
          Pete Wyckoff
        27. Makefile
          0.2 kB
          Sami Siren
        28. fuse_dfs.c
          23 kB
          Pete Wyckoff
        29. fuse-dfs.tar.gz
          5 kB
          Pete Wyckoff
        30. fuse-j-hadoopfs-03.tar.gz
          11 kB
          Anurag Sharma
        31. fuse_dfs.c
          16 kB
          Pete Wyckoff
        32. fuse-hadoop-0.1.0_fuse-j.2.4_hadoop.0.5.0.tar.gz
          27 kB
          Nguyen Quoc Mai
        33. fuse-hadoop-0.1.0_fuse-j.2.2.3_hadoop.0.5.0.tar.gz
          27 kB
          Nguyen Quoc Mai
        34. fuse-hadoop-0.1.1.tar.gz
          5 kB
          John Xing

          Issue Links

            Activity

              People

              • Assignee:
                wyckoff Pete Wyckoff
                Reporter:
                johnx John Xing
              • Votes:
                4 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: