Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-5278

Add a CLI allowing a user to enter a container.

    Details

    • Type: Improvement
    • Status: Accepted
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Containers created by the unified containerizer (Mesos containerizer) uses various namespaces (e.g., mount, network, etc.).

      To improve debugability, we should create a CLI that allows an operator or a user to enter the namespaces associated with the container, and execute an arbitrary command in that container (similar to `docker exec`).

        Issue Links

          Activity

          Hide
          idownes Ian Downes added a comment -

          I wrote an internal version of this tool which was simplified to support our environment. I can share that if you're interested? Either way, I'm definitely interested in providing input and can review.

          Show
          idownes Ian Downes added a comment - I wrote an internal version of this tool which was simplified to support our environment. I can share that if you're interested? Either way, I'm definitely interested in providing input and can review.
          Hide
          xujyan Yan Xu added a comment - - edited

          This is basically MESOS-2349 right?

          Ian Downes we are interested too and can help with review if you share it.

          Wait... you are offering to review... not sharing it?

          Had to read it a third time. Anyhow, +1 on this and it would be great if you can share it. A simplified version is a good start.

          Show
          xujyan Yan Xu added a comment - - edited This is basically MESOS-2349 right? Ian Downes we are interested too and can help with review if you share it. Wait... you are offering to review... not sharing it? Had to read it a third time. Anyhow, +1 on this and it would be great if you can share it. A simplified version is a good start.
          Hide
          vinodkone Vinod Kone added a comment -

          Would be great to add this as a subcommand to the existing "mesos" CLI ( does that work anymore?) instead of creating a new one.

          Show
          vinodkone Vinod Kone added a comment - Would be great to add this as a subcommand to the existing "mesos" CLI ( does that work anymore?) instead of creating a new one.
          Hide
          idownes Ian Downes added a comment -

          Yan Xu yeah it's got some code specific to Aurora's Thermos executor where it verifies the calling uid has access permission on the sandbox directory to implement crude access control when run under setuid root. It's only a small part though which could be generalized if people agreed with the broader approach.

          Vinod Kone I just tried various mesos subcommands and they seem to be broken on at least 0.26.x and 0.27.x...?

          $ mesos ps
            File "/usr/local/bin/mesos-ps", line 194
              with ThreadingExecutor() as executor:
                                   ^
          SyntaxError: invalid syntax
          
          Show
          idownes Ian Downes added a comment - Yan Xu yeah it's got some code specific to Aurora's Thermos executor where it verifies the calling uid has access permission on the sandbox directory to implement crude access control when run under setuid root. It's only a small part though which could be generalized if people agreed with the broader approach. Vinod Kone I just tried various mesos subcommands and they seem to be broken on at least 0.26.x and 0.27.x...? $ mesos ps File "/usr/local/bin/mesos-ps" , line 194 with ThreadingExecutor() as executor: ^ SyntaxError: invalid syntax
          Hide
          gyliu Guangya Liu added a comment -

          Ian Downes Which mesos command are you using? I tried `mesos` command but found that it does not have the `ps` option.

          root@mesos002:~/src/mesos/m3/mesos/build/src# ./mesos --help
          Not expecting '--help' before command
          Usage: lt-mesos <command> [OPTIONS]
          
          Available commands:
              help
              health-check
              execute
              docker-executor
              resolve
              containerizer
              executor
              logrotate-logger
              usage
              master
              fetcher
              agent
              log
              tests
              slave
              http-executor
              local
          
          Show
          gyliu Guangya Liu added a comment - Ian Downes Which mesos command are you using? I tried `mesos` command but found that it does not have the `ps` option. root@mesos002:~/src/mesos/m3/mesos/build/src# ./mesos --help Not expecting '--help' before command Usage: lt-mesos <command> [OPTIONS] Available commands: help health-check execute docker-executor resolve containerizer executor logrotate-logger usage master fetcher agent log tests slave http-executor local
          Hide
          haosdent@gmail.com haosdent added a comment -

          I use homebrew to install mesos, and could find it. mesos-ps maps to src/cli/mesos-ps in Mesos code.

          mesos help
          Usage: mesos <command> [OPTIONS]
          
          Available commands:
              help
              cat
              execute
              local
              log
              ps
              resolve
              scp
              tail
              daemon.sh
              master
              slave
              start-cluster.sh
              start-masters.sh
              start-slaves.sh
              stop-cluster.sh
              stop-masters.sh
              stop-slaves.sh
              cat
              execute
              local
              log
              ps
              resolve
              scp
              tail
          
          Show
          haosdent@gmail.com haosdent added a comment - I use homebrew to install mesos, and could find it. mesos-ps maps to src/cli/mesos-ps in Mesos code. mesos help Usage: mesos <command> [OPTIONS] Available commands: help cat execute local log ps resolve scp tail daemon.sh master slave start-cluster.sh start-masters.sh start-slaves.sh stop-cluster.sh stop-masters.sh stop-slaves.sh cat execute local log ps resolve scp tail
          Hide
          gyliu Guangya Liu added a comment -

          Ian Downes Can you please share your internal version of this tool? I want to take it as a reference. Thanks.

          Show
          gyliu Guangya Liu added a comment - Ian Downes Can you please share your internal version of this tool? I want to take it as a reference. Thanks.
          Hide
          gyliu Guangya Liu added a comment -

          haosdent , we need to put mesos- under $PATH if want mesos command pick up those sub commands, after putting mesos- under $PATH, I can get all subcommands for mesos.

          Show
          gyliu Guangya Liu added a comment - haosdent , we need to put mesos- under $PATH if want mesos command pick up those sub commands, after putting mesos- under $PATH , I can get all subcommands for mesos .
          Hide
          gyliu Guangya Liu added a comment -

          Jie Yu , one question want to get some help from you:

          1) The "mesos ps" can get all containers from the mesos cluster, so the operator can run this command on any host in the mesos cluster.
          2) for the new introduced CLI "mesos enter", as we need call setns to enter the container and execvp to run the command in the container, seems this command needs to be run on the agent where the container is running; otherwise, we may need some logic to ssh to the agent where the container is running first.

          Any comments?

          Ian Downes what is the behaviour of your internal mesos enter? Thanks.

          Show
          gyliu Guangya Liu added a comment - Jie Yu , one question want to get some help from you: 1) The "mesos ps" can get all containers from the mesos cluster, so the operator can run this command on any host in the mesos cluster. 2) for the new introduced CLI "mesos enter", as we need call setns to enter the container and execvp to run the command in the container, seems this command needs to be run on the agent where the container is running; otherwise, we may need some logic to ssh to the agent where the container is running first. Any comments? Ian Downes what is the behaviour of your internal mesos enter ? Thanks.
          Hide
          gyliu Guangya Liu added a comment -

          Did some test with python as follows.

          1) Create a volume file

          root@mesos002:~/test/setns# cat /root/test/v6.json
            [{
            "container_path":"\/tmp\/abc2",
            "mode":"RW",
            "source":
              {
                "docker_volume":
                  {
                    "driver":"convoy",
                    "driver_options":
                      {"parameter":[
                        {
                          "key":"iops",
                          "value":"150"
                        }
                      ]},
                      "name":"dvd2"
                   },
                   "type":"DOCKER_VOLUME"
              }
          }]
          

          2) Start up mesos-executor

          root@mesos002:~/src/mesos/m2/mesos/build# ./src/mesos-execute --master=192.168.56.12:5050 --command="sleep 100000" --name=test --docker_image=ubuntu:14.04 --volumes=/root/test/v6.json
          I0601 00:19:22.391978 31447 scheduler.cpp:187] Version: 1.0.0
          I0601 00:19:22.394105 31471 scheduler.cpp:471] New master detected at master@192.168.56.12:5050
          Subscribed with ID '6ead2bbc-ae7e-4973-9a8d-0c9c02668573-0000'
          Submitted task 'test' to agent '6ead2bbc-ae7e-4973-9a8d-0c9c02668573-S0'
          Received status update TASK_RUNNING for task 'test'
            source: SOURCE_EXECUTOR
          

          3) Check convoy volume list and mount point

          root@mesos002:~/test/setns# convoy list
          {
          	"01569b03-81f2-47be-8d16-169c1adcb541": {
          		"UUID": "01569b03-81f2-47be-8d16-169c1adcb541",
          		"Name": "dvd2",
          		"Driver": "devicemapper",
          		"MountPoint": "/var/lib/convoy/devicemapper/mounts/01569b03-81f2-47be-8d16-169c1adcb541",
          		"CreatedTime": "Mon Apr 18 10:57:32 +0800 2016",
          		"DriverInfo": {
          			"DevID": "12",
          			"Device": "/dev/mapper/01569b03-81f2-47be-8d16-169c1adcb541",
          			"Driver": "devicemapper",
          			"MountPoint": "/var/lib/convoy/devicemapper/mounts/01569b03-81f2-47be-8d16-169c1adcb541",
          			"Size": "107374182400"
          		},
          		"Snapshots": {}
          	}
          }
          root@mesos002:~/test/setns# ls /var/lib/convoy/devicemapper/mounts/01569b03-81f2-47be-8d16-169c1adcb541
          ibm2  lost+found
          

          4) Using a python script to enter the mnt namespace of the container.

          root@mesos002:~/test/setns# cat setns.py
          #!/usr/bin/env python
          
          import ctypes
          import sys
          import os
          import subprocess
          
          f = None
          libc = ctypes.CDLL('libc.so.6')
          myfd = os.open('/proc/31563/ns/mnt', os.O_RDONLY)
          libc.setns(myfd, 0)
          
          subprocess.Popen(['ls', '/tmp/abc2'])
          root@mesos002:~/test/setns# python setns.py
          ibm2  lost+found
          

          There are currently two issues for this CLI:
          1) The agent do not export the executor pid or task pid, so I cannot get the task process id now. Seems we need to expose the executor pid or task pid to mesos task endpoint?
          2) Where does mesos enter run? Only in master or on the host where the container is running? I can see that for docker, I have to run docker exec on the docker server where the container is running; but with swarm, I can run docker exec anywhere. If want to enable mesos enter run anywhere, we may need to add some logic such as remote exec to mesos enter.

          Show
          gyliu Guangya Liu added a comment - Did some test with python as follows. 1) Create a volume file root@mesos002:~/test/setns# cat /root/test/v6.json [{ "container_path" : "\/tmp\/abc2" , "mode" : "RW" , "source" : { "docker_volume" : { "driver" : "convoy" , "driver_options" : { "parameter" :[ { "key" : "iops" , "value" : "150" } ]}, "name" : "dvd2" }, "type" : "DOCKER_VOLUME" } }] 2) Start up mesos-executor root@mesos002:~/src/mesos/m2/mesos/build# ./src/mesos-execute --master=192.168.56.12:5050 --command= "sleep 100000" --name=test --docker_image=ubuntu:14.04 --volumes=/root/test/v6.json I0601 00:19:22.391978 31447 scheduler.cpp:187] Version: 1.0.0 I0601 00:19:22.394105 31471 scheduler.cpp:471] New master detected at master@192.168.56.12:5050 Subscribed with ID '6ead2bbc-ae7e-4973-9a8d-0c9c02668573-0000' Submitted task 'test' to agent '6ead2bbc-ae7e-4973-9a8d-0c9c02668573-S0' Received status update TASK_RUNNING for task 'test' source: SOURCE_EXECUTOR 3) Check convoy volume list and mount point root@mesos002:~/test/setns# convoy list { "01569b03-81f2-47be-8d16-169c1adcb541" : { "UUID" : "01569b03-81f2-47be-8d16-169c1adcb541" , "Name" : "dvd2" , "Driver" : "devicemapper" , "MountPoint" : "/ var /lib/convoy/devicemapper/mounts/01569b03-81f2-47be-8d16-169c1adcb541" , "CreatedTime" : "Mon Apr 18 10:57:32 +0800 2016" , "DriverInfo" : { "DevID" : "12" , "Device" : "/dev/mapper/01569b03-81f2-47be-8d16-169c1adcb541" , "Driver" : "devicemapper" , "MountPoint" : "/ var /lib/convoy/devicemapper/mounts/01569b03-81f2-47be-8d16-169c1adcb541" , "Size" : "107374182400" }, "Snapshots" : {} } } root@mesos002:~/test/setns# ls / var /lib/convoy/devicemapper/mounts/01569b03-81f2-47be-8d16-169c1adcb541 ibm2 lost+found 4) Using a python script to enter the mnt namespace of the container. root@mesos002:~/test/setns# cat setns.py #!/usr/bin/env python import ctypes import sys import os import subprocess f = None libc = ctypes.CDLL('libc.so.6') myfd = os.open('/proc/31563/ns/mnt', os.O_RDONLY) libc.setns(myfd, 0) subprocess.Popen(['ls', '/tmp/abc2']) root@mesos002:~/test/setns# python setns.py ibm2 lost+found There are currently two issues for this CLI: 1) The agent do not export the executor pid or task pid, so I cannot get the task process id now. Seems we need to expose the executor pid or task pid to mesos task endpoint? 2) Where does mesos enter run? Only in master or on the host where the container is running? I can see that for docker, I have to run docker exec on the docker server where the container is running; but with swarm, I can run docker exec anywhere. If want to enable mesos enter run anywhere, we may need to add some logic such as remote exec to mesos enter .

            People

            • Assignee:
              gyliu Guangya Liu
              Reporter:
              jieyu Jie Yu
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:

                Development