tool to mount dfs on linux

• Key: HADOOP-4
• Type: Improvement
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Pete Wyckoff
• Reporter: John Xing
• Fix Version/s: 0.18.0

Environment

OSs that support FUSE. Includes Linux, MacOSx, OpenSolaris... http://fuse.sourceforge.net/wiki/index.php/OperatingSystems

Description

This is a FUSE module for Hadoop's HDFS.

It allows one to mount HDFS as a Unix filesystem and optionally export
that mount point to other machines.

rmdir, mv, mkdir, rm are all supported. just not cp, touch, ..., but actual writes require: https://issues.apache.org/jira/browse/HADOOP-3485

For the most up-to-date documentation, see: http://wiki.apache.org/hadoop/MountableHDFS

BUILDING:

Requirements:

1. a Linux kernel > 2.6.9 or a kernel module from FUSE - i.e., you
compile it yourself and then modprobe it. Better off with the
former option if possible. (Note for now if you use the kernel
with fuse included, it doesn't allow you to export this through NFS
so be warned. See the FUSE email list for more about this.)

2. FUSE should be installed in /usr/local or FUSE_HOME ant
environment variable

To build:

1. in HADOOP_HOME: ant compile-contrib -Dcompile.c++=1 -Dfusedfs=1 -Dlibhdfs=1

NOTE: for amd64 architecture, libhdfs will not compile unless you edit
the Makefile in src/c++/libhdfs/Makefile and set OS_ARCH=amd64
(probably the same for others too).

--------------------------------------------------------------------------------

CONFIGURING:

Look at all the paths in fuse_dfs_wrapper.sh and either correct them
or set them in your environment before running. (note for automount
and mount as root, you probably cannnot control the environment, so
best to set them in the wrapper)

INSTALLING:

1. mkdir /mnt/dfs (or wherever you want to mount it)

2. fuse_dfs_wrapper.sh dfs://hadoop_server1.foo.com:9000 /mnt/dfs -d
; and from another terminal, try ls /mnt/dfs

If 2 works, try again dropping the debug mode, i.e., -d

(note - common problems are that you don't have libhdfs.so or
libjvm.so or libfuse.so on your LD_LIBRARY_PATH, and your CLASSPATH
does not contain hadoop and other required jars.)

--------------------------------------------------------------------------------

DEPLOYING:

in a root shell do the following:

1. add the following to /etc/fstab -
fuse_dfs#dfs://hadoop_server.foo.com:9000 /mnt/dfs fuse
allow_other,rw 0 0

2. mount /mnt/dfs Expect problems with not finding fuse_dfs. You will
need to probably add this to /sbin and then problems finding the
above 3 libraries. Add these using ldconfig.

--------------------------------------------------------------------------------

EXPORTING:

Add the following to /etc/exports:

/mnt/hdfs *.foo.com(no_root_squash,rw,fsid=1,sync)

NOTE - you cannot export this with a FUSE module built into the kernel

• e.g., kernel 2.6.17. For info on this, refer to the FUSE wiki.
  --------------------------------------------------------------------------------

ADVANCED:

you may want to ensure certain directories cannot be deleted from the
shell until the FS has permissions. You can set this in the build.xml
file in src/contrib/fuse-dfs/build.xml