Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-2460

docker-hadoop.sh fails when deploying on ubunt 16.04 images

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Won't Fix
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.0
    • Labels:
      None
    • Environment:

      ubuntu 16.04 on x86 and ppc

      Description

      this issue is due to unavailability of openjdk 1.7 packages in ubuntu 16.04 PPA

      1. BIGTOP-2460.patch
        0.8 kB
        Amir Sanjar

        Issue Links

          Activity

          Hide
          asanjar Amir Sanjar added a comment -

          In order to install openjdk 1.7 in Ubuntu 16.04 environment, you must first do "add-apt-repository -y ppa:openjdk-r/ppa "

          Show
          asanjar Amir Sanjar added a comment - In order to install openjdk 1.7 in Ubuntu 16.04 environment, you must first do "add-apt-repository -y ppa:openjdk-r/ppa "
          Hide
          cos Konstantin Boudnik added a comment -

          I'd suggest we stop dragging the old JDK7 release for the deployment purposes. JDK8 works just fine for the runtime, and we already had this discussion on the dev@ list to ditch old version, if only for the runtime. Does it make sense?

          Show
          cos Konstantin Boudnik added a comment - I'd suggest we stop dragging the old JDK7 release for the deployment purposes. JDK8 works just fine for the runtime, and we already had this discussion on the dev@ list to ditch old version, if only for the runtime. Does it make sense?
          Hide
          asanjar Amir Sanjar added a comment -

          this patch doesn't install JDK7 but enables the installation if user decides to do so.
          Personally, I would rather use JDK8 anytime If the decision is to switch to JDK8 at runtime, I could replace this patch with a new patch for https://github.com/apache/bigtop/blob/master/bigtop-deploy/vm/vagrant-puppet-docker/vagrantconfig_ubuntu.yaml

          Show
          asanjar Amir Sanjar added a comment - this patch doesn't install JDK7 but enables the installation if user decides to do so. Personally, I would rather use JDK8 anytime If the decision is to switch to JDK8 at runtime, I could replace this patch with a new patch for https://github.com/apache/bigtop/blob/master/bigtop-deploy/vm/vagrant-puppet-docker/vagrantconfig_ubuntu.yaml
          Hide
          cos Konstantin Boudnik added a comment -

          There was this discussion a couple of weeks ago, and seems like JDK8 is what makes sense in the deployment. Overall, unless we try it, we never figure out if it works, right?

          Show
          cos Konstantin Boudnik added a comment - There was this discussion a couple of weeks ago , and seems like JDK8 is what makes sense in the deployment. Overall, unless we try it, we never figure out if it works, right?
          Hide
          oflebbe Olaf Flebbe added a comment -

          I am inclined to seperate compile from deployment architectures as follows:

          Only supporting: ubuntu-14.04 as development and support newer versions as runtime platforms, reusing the 14.04 packages.

          The same have to be done for fedora. Using centos-7 and deploy on recent fedora.

          Or more radically: Using docker as deployment format and switch to docker deployment with only – for instance – an debian build. Not in the sense of microservices, using docker only for deployment

          docker run --network=host  -v /etc/hadoop/conf:/etc/hadoop/conf -v /etc/hadoop/hdfs.keytab:/etc/hadoop/hdfs.keytab -v /var/log/hadoop-hdfs:/var/log/hadoop-hdfs bigtop/hadoop-hdfs-namenode:latest  /etc/init.d/hadoop-hdfs-namenode
          

          and so on

          Show
          oflebbe Olaf Flebbe added a comment - I am inclined to seperate compile from deployment architectures as follows: Only supporting: ubuntu-14.04 as development and support newer versions as runtime platforms, reusing the 14.04 packages. The same have to be done for fedora. Using centos-7 and deploy on recent fedora. Or more radically: Using docker as deployment format and switch to docker deployment with only – for instance – an debian build. Not in the sense of microservices, using docker only for deployment docker run --network=host -v /etc/hadoop/conf:/etc/hadoop/conf -v /etc/hadoop/hdfs.keytab:/etc/hadoop/hdfs.keytab -v / var /log/hadoop-hdfs:/ var /log/hadoop-hdfs bigtop/hadoop-hdfs-namenode:latest /etc/init.d/hadoop-hdfs-namenode and so on
          Hide
          cos Konstantin Boudnik added a comment -

          So, you're basically suggesting we should add another packaging format "container" to what we already have? I guess it makes quite a bit of sense, although it will require a different (much simpler, perhaps) deployment mechanism.

          Show
          cos Konstantin Boudnik added a comment - So, you're basically suggesting we should add another packaging format "container" to what we already have? I guess it makes quite a bit of sense, although it will require a different (much simpler, perhaps) deployment mechanism.
          Hide
          oflebbe Olaf Flebbe added a comment -

          Yes that was the idea. But testing containers within docker containers is "challenging". See https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/

          Show
          oflebbe Olaf Flebbe added a comment - Yes that was the idea. But testing containers within docker containers is "challenging". See https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/
          Hide
          asanjar Amir Sanjar added a comment -

          Olaf, if I understand you correctly, it sounds similar to one of my side-projects I've have been working on
          https://github.com/ibmsoe/bigtop-docker

          Show
          asanjar Amir Sanjar added a comment - Olaf, if I understand you correctly, it sounds similar to one of my side-projects I've have been working on https://github.com/ibmsoe/bigtop-docker
          Hide
          oflebbe Olaf Flebbe added a comment -

          Thanks for pointing to this. How do you configure your nodes ?

          My idea was to use puppet on the docker host to generate /etc/hadoop/conf etc. and bind mounting it into the container.

          The second part is if we can still use docker for deployment test, since this will need docker containers within docker containers. I tested this to some extent, but still have issues – maybe unrelated.

          Show
          oflebbe Olaf Flebbe added a comment - Thanks for pointing to this. How do you configure your nodes ? My idea was to use puppet on the docker host to generate /etc/hadoop/conf etc. and bind mounting it into the container. The second part is if we can still use docker for deployment test, since this will need docker containers within docker containers. I tested this to some extent, but still have issues – maybe unrelated.
          Hide
          asanjar Amir Sanjar added a comment -

          Hi olaf,
          I don't use puppet since most of our target audience, unfortunately, don't have puppet skills. I use shell scripts similar to https://github.com/ibmsoe/bigtop-docker/blob/master/common/update-conf.sh and https://github.com/ibmsoe/bigtop-docker/blob/master/helper_scripts/pre_install.sh to configure the nodes.. It is still work in progress, but the idea is to provide a light weight but configurable environment .

          Show
          asanjar Amir Sanjar added a comment - Hi olaf, I don't use puppet since most of our target audience, unfortunately, don't have puppet skills. I use shell scripts similar to https://github.com/ibmsoe/bigtop-docker/blob/master/common/update-conf.sh and https://github.com/ibmsoe/bigtop-docker/blob/master/helper_scripts/pre_install.sh to configure the nodes.. It is still work in progress, but the idea is to provide a light weight but configurable environment .
          Hide
          cos Konstantin Boudnik added a comment -

          Amir Sanjar, Olaf Flebbe do you guys plan on committing this? If not, please move the ticket to 1.3. Thanks!

          Show
          cos Konstantin Boudnik added a comment - Amir Sanjar , Olaf Flebbe do you guys plan on committing this? If not, please move the ticket to 1.3. Thanks!
          Hide
          oflebbe Olaf Flebbe added a comment -

          I will not commit this, SInce I like to support JDK 8 only

          Show
          oflebbe Olaf Flebbe added a comment - I will not commit this, SInce I like to support JDK 8 only
          Hide
          cos Konstantin Boudnik added a comment -

          So, should be closed, moved to backlog or else?

          Show
          cos Konstantin Boudnik added a comment - So, should be closed, moved to backlog or else?
          Hide
          oflebbe Olaf Flebbe added a comment -

          No support for JDK 7 anymore: closing it.

          Show
          oflebbe Olaf Flebbe added a comment - No support for JDK 7 anymore: closing it.

            People

            • Assignee:
              asanjar Amir Sanjar
              Reporter:
              asanjar Amir Sanjar
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Due:
                Created:
                Updated:
                Resolved:

                Development