Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-6020

Rework Docker-inside-Docker for JobServer container

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: runner-flink

      Description

      Docker-inside-Docker causes problems with permissions. For example:

      Caused by: java.lang.Exception: The user defined 'open()' method caused an exception: java.io.IOException: Cannot run program "docker": error=13, Permission denied
      	at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:498)
      	at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:368)
      	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:712)
      	... 1 more
      Caused by: org.apache.beam.repackaged.beam_runners_java_fn_execution.com.google.common.util.concurrent.UncheckedExecutionException: java.io.IOException: Cannot run program "docker": error=13, Permission denied
      	at org.apache.beam.repackaged.beam_runners_java_fn_execution.com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4994)
      
      	... 7 more
      

      See discussion:
      https://lists.apache.org/thread.html/b2b97efaf1f4a6e283eaf92e05605f10f405e9eb9a9c41896041d0ac@%3Cdev.beam.apache.org%3E

      We could get rid of Docker-inside-Docker entirely. It would be cleaner to start the SDK harness container alongside with JobServer container.

      Alternatively, it should be possible to use an image with Docker pre-installed, which should fix the permissions errors which come from mounting the Docker executable of the host.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              mxm Maximilian Michels
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: