Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12830

Bash environment for quick command operations



    • New Feature
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • None
    • None
    • bin
    • None


      Hadoop file system shell commands are slow. This issue is about building a shell environment for quick command operations.

      Previously an interactive shell is tried to build in HADOOP-6541. But, it seems to be poor because users are used to powerful shells like bash. This issue is not about creating a new shell, but just opening a new bash process. Therefore, user can operate commands as before.

      fjk@x240:~/hadoop-2.7.2$ ./bin/hadoop shell
      fjk@x240 hadoop> hadoop fs -ls /
      Found 2 items
      -rw-r--r--   3 fjk supergroup          0 2016-02-21 00:26 /file1
      -rw-r--r--   3 fjk supergroup          0 2016-02-21 00:26 /file2

      The shell has a mini daemon process that is living until the shell is closed. The hadoop fs command delegates the operation to the daemon. They communicate with named pipes. The daemon conducts the operation and returns the result to the command.

      In this shell the hadoop fs commands operation becomes quick. In a local environment, "hadoop fs -ls" command is about 100 times faster than the normal command.

      fjk@x240 hadoop> time hadoop fs -ls hdfs://localhost:8020/ > /dev/null
      real	0m0.021s
      user	0m0.003s
      sys	0m0.011s

      Using bash's function, commands and file names are automatically completed.

      fjk@x240 hadoop> hadoop fs -ch<TAB><TAB>
      -checksum  -chgrp     -chmod     -chown
      fjk@x240 hadoop> hadoop fs -ls /file<TAB><TAB>
      /file1  /file2  /file3

      Additionally, we can make equivalents with bash build-in commands, e.g., cd, umask. In this shell, they can work because the daemon remembers the state.


        1. HADOOP-12830.001.patch
          13 kB
          Kazuho Fujii
        2. HADOOP-12830.002.patch
          32 kB
          Kazuho Fujii
        3. HADOOP-12830.003.patch
          32 kB
          Kazuho Fujii



            fjk Kazuho Fujii
            fjk Kazuho Fujii
            0 Vote for this issue
            9 Start watching this issue