Details

      Description

      GSOC: Add Docker support to Taverna

      The proposed GSOC project is to add support for invoking Docker containers within Taverna by adding a Docker Activity plugin.

      Tasks include:

      • Propose JSON model for describing a docker run command
      • (Optional) Validate Docker activity config, e.g. can the docker image be pulled?
      • Investigate: New Docker activity, or modify existing External Tool activity?
      • Make/modify a Taverna Activity plugin for executing Docker (may or may not be based on the External Tool activity)
      • (Optional) Capture docker metadata and add to workflow run provenance (e.g. which docker image ID was pulled)
      • (Optional) Add Bioboxes support
      • (Optional) Integrate with CWL support (TAVERNA-900)

      Other Taverna/Docker--related tasks can of course also be proposed by the students.

      Docker

      Docker is a Linux container virtualization platform. A Linux container is a special kernel feature, which similar to chroot jails behave as a separate machine, but unlike Virtual Machines do not have the overhead of virtualization of hardware.

      Docker is popular in the devops movement as it provides an easy way to install dependencies for software development and deployment, e.g. to run servers for mySQL, Apache Solr or node.js.

      In brief a Docker Image contains a virtual Linux file system (e.g. a miniature Debian installation). A Docker Container is a particular execution of a Docker Image, which typically runs a single process as installed within the container, and may have network ports exposed to the world, or have parts of the host computer's file system mounted within the inner container.

      One great advantage of Docker is that it simplifies tool installation, as each Docker image is a self-contained Linux distribution which don't have to be compatible with the host computer (beyond the kernel).

      For Windows and OS X users Docker automatically manage a virtual machine running the Linux containers, but Docker containers can also be deployed on the cloud or a local cluster, e,g. using Docker Machine.

      Docker images can be created from a Dockerfile, which basically lists the commands to run to prepare the image. Docker images can be chained together using base images - for instance to build on an image with mySQL, the Dockerfile says FROM mysql.

      Thus Docker is also an important tool for reproducibility, as these images can be automatically kept up to date and are distributed through the Docker hub. In bioinformatics, this has led to Bioboxes, a standard for creating interchangable bioinformatics software containers.

      Taverna

      Apache Taverna (incubating) is a Java-based workflow system with a graphical design interface. Taverna workflows can combine many different service types, including REST and WSDL services, command line tools, scripts (e.g. BeanShell, R) and custom plugins (e.g. BioMart).

      Taverna workflows can be executed on the desktop, on the command line, or on a Taverna server installation, which can be controlled from a web portal, a mobile app, or integrated into third-party applications.

      Taverna is used in a wide range of sciences for data analysis and processing, including bioinformatics, cheminformatics, biodiversity and musicology. Workflow engine features include provenance tracking, implicit parallelism/iterations, retry/failover and looping.

      Taverna workflows are commonly shared on myExperiment, and can either be created graphically in the Taverna workbench, programmatically using the Taverna Language API or by generating workflow definitions in the SCUFL2 format.

      Community engagement

      Interested GSOC students are requested to engage early with the dev@taverna mailing list to describe their ideas for approaching this project, to clarify the tasks and for any questions and issues.

      As a first step, the prospective applicant should leave a comment on this Jira issue to indicate their interest, and the GSOC mentors would be happy to assist on any questions.

      As the project starts we are expecting the student to become part of the dev@taverna community to regularly discuss their progress.

      Mentors

      An important part of GSOC is the personal mentoring from existing members of the open source community. Our job is not just to teach you how to successfully get through the GSOC programme, but also to motivate you and make sure you progress. We will show you how to contribute to open source, debug, improve, document, test and release your code as part of Apache Taverna.

      The GSOC mentors for Apache Taverna have experience from guiding multiple earlier GSOC students and local students, and can be contacted privately for day-to-day interaction and trouble-shooting.

      Mentors for this GSOC project:

      • Stian Soiland-Reyes
      1. Docker Activity Plugin UnitTest User Guide.pdf
        185 kB
        Hewa Vidana Gamage Nadeesh Dilanga

        Issue Links

          Activity

          Hide
          Namodaya Namodaya added a comment - - edited

          Hi Stian,
          My name is Namodaya Balaarachchi and I am a final year undergraduate of Faculty of Engineering University of Peradeniya Sri Lanka.
          I went through the project ideas and all of them are very interesting for me. So I decided to work on this project for GSOC. I have knowledge about docker and java.
          I've started going through the references already. It would be very helpful If you could give me some more guidelines about this.

          Thank you.

          Show
          Namodaya Namodaya added a comment - - edited Hi Stian, My name is Namodaya Balaarachchi and I am a final year undergraduate of Faculty of Engineering University of Peradeniya Sri Lanka. I went through the project ideas and all of them are very interesting for me. So I decided to work on this project for GSOC. I have knowledge about docker and java. I've started going through the references already. It would be very helpful If you could give me some more guidelines about this. Thank you.
          Hide
          stain Stian Soiland-Reyes added a comment -

          Hi,

          Thanks for showing interest in Taverna and GSOC!

          The best would be if you can sign up to the dev@taverna mailing list, introduce yourself there, and then we can see what would be a good place for you to start. See http://taverna.incubator.apache.org/community/ for details.

          You will probably want to learn a bit about Taverna itself so you see how your contribution would fit in. See the introduction pages at http://taverna.incubator.apache.org/introduction/ - and try the Quick Start guide in http://taverna.incubator.apache.org/documentation/quick-start-guide/
          with Taverna Workbench Core 2.5 from http://taverna.incubator.apache.org/download/workbench/

          Note that your plugin would be for Taverna 3 which has slightly modified how activities are defined and installed - but we have a tutorial for Taverna 2.5 here you might want to try out first: http://dev.mygrid.org.uk/wiki/display/developer/Tutorial+-+Service+invocation+plugin

          You don't need to follow-on to the next tutorial on how to add it to the workbench (as I think that would be too much for this GSOC project), but can have a look if you want.

          We can discuss on the dev@taverna list what would be the changes. In brief the configuration for a Docker Activity would need to be a JSON object to be compatible with Taverna 3. (The tutorial will show you how to make a Java bean configuration).

          So for your project proposal you will need to define some smaller tasks, like the ones proposed in this issue.

          Show
          stain Stian Soiland-Reyes added a comment - Hi, Thanks for showing interest in Taverna and GSOC! The best would be if you can sign up to the dev@taverna mailing list, introduce yourself there, and then we can see what would be a good place for you to start. See http://taverna.incubator.apache.org/community/ for details. You will probably want to learn a bit about Taverna itself so you see how your contribution would fit in. See the introduction pages at http://taverna.incubator.apache.org/introduction/ - and try the Quick Start guide in http://taverna.incubator.apache.org/documentation/quick-start-guide/ with Taverna Workbench Core 2.5 from http://taverna.incubator.apache.org/download/workbench/ Note that your plugin would be for Taverna 3 which has slightly modified how activities are defined and installed - but we have a tutorial for Taverna 2.5 here you might want to try out first: http://dev.mygrid.org.uk/wiki/display/developer/Tutorial+-+Service+invocation+plugin You don't need to follow-on to the next tutorial on how to add it to the workbench (as I think that would be too much for this GSOC project), but can have a look if you want. We can discuss on the dev@taverna list what would be the changes. In brief the configuration for a Docker Activity would need to be a JSON object to be compatible with Taverna 3. (The tutorial will show you how to make a Java bean configuration). So for your project proposal you will need to define some smaller tasks, like the ones proposed in this issue.
          Hide
          Namodaya Namodaya added a comment -

          Hi Stian Soiland-Reyes,
          Thank you for your reply.
          I already sign up to the dev@taverna mailing list. Also I went through those posted mails on mailing list. Now I started to learn about 'Teverna' as you guide. I think it will be very useful to get more familiar with 'Teverna' and adept to the project to be done.

          Thank you

          Show
          Namodaya Namodaya added a comment - Hi Stian Soiland-Reyes , Thank you for your reply. I already sign up to the dev@taverna mailing list. Also I went through those posted mails on mailing list. Now I started to learn about 'Teverna' as you guide. I think it will be very useful to get more familiar with 'Teverna' and adept to the project to be done. Thank you
          Hide
          amila1991 Amila Karunathilaka added a comment - - edited

          Hi Stian,
          I'm Amila Karunathilaka 3rd year undergraduate student in Computer Science & Engineering from University of Moratuwa Sri Lanka.
          I'm interested about this project idea. I have knowledge and experience about Java, Docker and Kuberneties.
          I followed your references.
          I would like to know more about this project. Please give me some tips to getting started.

          Thank you.

          Show
          amila1991 Amila Karunathilaka added a comment - - edited Hi Stian, I'm Amila Karunathilaka 3rd year undergraduate student in Computer Science & Engineering from University of Moratuwa Sri Lanka. I'm interested about this project idea. I have knowledge and experience about Java, Docker and Kuberneties. I followed your references. I would like to know more about this project. Please give me some tips to getting started. Thank you.
          Hide
          stain Stian Soiland-Reyes added a comment -

          Hi, potential Google Summer of Code (GSOC) students!

          In a tiny hour the GSOC student application period opens:

          > https://developers.google.com/open-source/gsoc/timeline
          > 14 March 19:00 UTC Student application period opens.
          > 25 March 19:00 UTC Student application deadline.

          http://write.flossmanuals.net/gsocstudentguide/ has a good description
          of the process.

          I've tried to summarize who is interested in what here:

          https://cwiki.apache.org/confluence/display/TAVERNADEV/2016-03+GSOC+2016

          My apologies for any mistakes or if I forgot someone.. it was a bit
          hard to track across all the emails! Let me know what to fix.

          If you are preparing a GSOC proposal for Apache Taverna, e.g. inspired
          by our ideas at:

          https://s.apache.org/taverna-gsoc-2016

          or your own ideas (e.g further developing Taverna Mobile), then
          now is a good time to start working on your project proposal.

          I suggest writing a Google Docs page, and sharing it on dev@taverna (which hopefully you have signed up to - see http://taverna.incubator.apache.org/contact)

          .. or leave a comment on this Jira Issue so we can discuss it and help you get a good proposal!

          See http://write.flossmanuals.net/gsocstudentguide/writing-a-proposal/
          for suggestions for your proposal, and a template here:

          http://community.staging.apache.org/gsoc#application-template

          Show
          stain Stian Soiland-Reyes added a comment - Hi, potential Google Summer of Code (GSOC) students! In a tiny hour the GSOC student application period opens: > https://developers.google.com/open-source/gsoc/timeline > 14 March 19:00 UTC Student application period opens. > 25 March 19:00 UTC Student application deadline. http://write.flossmanuals.net/gsocstudentguide/ has a good description of the process. I've tried to summarize who is interested in what here: https://cwiki.apache.org/confluence/display/TAVERNADEV/2016-03+GSOC+2016 My apologies for any mistakes or if I forgot someone.. it was a bit hard to track across all the emails! Let me know what to fix. If you are preparing a GSOC proposal for Apache Taverna, e.g. inspired by our ideas at: https://s.apache.org/taverna-gsoc-2016 or your own ideas (e.g further developing Taverna Mobile), then now is a good time to start working on your project proposal. I suggest writing a Google Docs page, and sharing it on dev@taverna (which hopefully you have signed up to - see http://taverna.incubator.apache.org/contact ) .. or leave a comment on this Jira Issue so we can discuss it and help you get a good proposal! See http://write.flossmanuals.net/gsocstudentguide/writing-a-proposal/ for suggestions for your proposal, and a template here: http://community.staging.apache.org/gsoc#application-template
          Hide
          nadeesh092hvgd Hewa Vidana Gamage Nadeesh Dilanga added a comment -

          Here I am attaching the use guide PDF with screen captures on understanding docker activity plugin unit test case(which expose the end to end docker functionality)

          Show
          nadeesh092hvgd Hewa Vidana Gamage Nadeesh Dilanga added a comment - Here I am attaching the use guide PDF with screen captures on understanding docker activity plugin unit test case(which expose the end to end docker functionality)

            People

            • Assignee:
              Unassigned
              Reporter:
              stain Stian Soiland-Reyes
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:

                Development