Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-9014

runC container runtime

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Release Note:
      Hide
      The new RuncContainerRuntime is an experimental feature that allows Hadoop tasks to be run inside of runC containers. This feature is similar to the DockerLinuxContainerRuntime, but is catered more to Hadoop. Notably, the RuncContainerRuntime leverages squashFS for its image format and spreads the images across all disks accessible to Hadoop using the distributed cache. Additionally, unlike Docker, the RuncContainerRuntime does not rely on any daemons.
      Show
      The new RuncContainerRuntime is an experimental feature that allows Hadoop tasks to be run inside of runC containers. This feature is similar to the DockerLinuxContainerRuntime, but is catered more to Hadoop. Notably, the RuncContainerRuntime leverages squashFS for its image format and spreads the images across all disks accessible to Hadoop using the distributed cache. Additionally, unlike Docker, the RuncContainerRuntime does not rely on any daemons.

      Description

      This JIRA tracks a YARN container runtime that supports running containers in images built by Docker but the runtime does not use Docker directly, and Docker does not have to be installed on the nodes. The runtime leverages the OCI runtime standard to launch containers, so an OCI-compliant runtime like runc is required. runc has the benefit of not requiring a daemon like dockerd to be running in order to launch/control containers.

      The layers comprising the Docker image are uploaded to HDFS as squashfs images, enabling the runtime to efficiently download and execute directly on the compressed layers. This saves image unpack time and space on the local disk. The image layers, like other entries in the YARN distributed cache, can be spread across the YARN local disks, increasing the available space for storing container images on each node.

      A design document will be posted shortly.

        Attachments

        1. OciSquashfsRuntime.v001.pdf
          176 kB
          Jason Darrell Lowe
        2. RuncContainerRuntime.v002.pdf
          228 kB
          Eric Badger

          Activity

            People

            • Assignee:
              ebadger Eric Badger
              Reporter:
              jlowe Jason Darrell Lowe
            • Votes:
              2 Vote for this issue
              Watchers:
              22 Start watching this issue

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 5h 10m
                5h 10m