Uploaded image for project: 'Apache Knox'
  1. Apache Knox
  2. KNOX-2240

KnoxShell Custom Command for WEBHDFS Use

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4.0
    • Component/s: KnoxShell
    • Labels:
      None

      Description

      I'd like to extend the KnoxShell environment to have a Custom Command for interacting with Hadoop FileSystems that support the WebHDFS REST API. We will wrap the use of the KnoxShell HDFS classes for posix like commands in the shell environment.

      In order to interact with multiple webhdfs filesystems we introduce the notion of mounting a webhdfs service from a Knox topology URL. For instance, see the following mount command to mount such a service to a mount point:

      :fs mount https://knoxhost.example.com/topology/cdp-proxy-api retail
      

      The above mounts a Hadoop filesystem to a mount point called "retail".

      Given this mounted filesystem we are able to interact with it through the mountpoint as the root.

      :fs ls /retail/user/lmccay
      

      The above will list the contents of my home directory from the mounted filesystem as well as return the listing as a KnoxShellTable that can be sorted, selected, filtered, etc as any other KnoxShellTable.

      Additional commands

      :fs mkdir /retail/user/lmccay/tmp 755
      :fs cat /retail/user/lmccay/tmp/README
      :fs put /home/lmccay/README /retail/user/lmccay/tmp/README
      :fs get /retail/user/lmccay/tmp/README /home/lmccay/README
      :fs rename /retail/user/lmccay/tmp/README /retail/user/lmccay/tmp/README2
      :fs rm /retail/user/lmccay/tmp/README
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                lmccay Larry McCay
                Reporter:
                lmccay Larry McCay
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h 20m
                  2h 20m