Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-2062

cluster.yaml declares undefined vars; apply is broken

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.1.0
    • Component/s: deployment
    • Labels:
      None

      Description

      Running puppet apply on master produces the following error

      Error: Error from DataBinding 'hiera' while looking up 'hadoop::kerberos_realm': can't convert nil into String on node bigtop1.docker
      Wrapped exception:
      can't convert nil into String
      

      The change has been added in BIGTOP-1662, but wasn't affecting the deployment until recently (e.g. 1.0 deployment works just fine). Looks like the recent commit of roles code has revealed the bug.

      1. BIGTOP-2062.patch
        0.6 kB
        Konstantin Boudnik
      2. site.yaml
        0.8 kB
        Konstantin Boudnik

        Activity

        Hide
        cos Konstantin Boudnik added a comment -

        Here's the patch to get around the issue. Perhaps our Puppet exports would be able to do a better one, but this let me go through the deployment now.

        Show
        cos Konstantin Boudnik added a comment - Here's the patch to get around the issue. Perhaps our Puppet exports would be able to do a better one, but this let me go through the deployment now.
        Hide
        evans_ye Evans Ye added a comment - - edited

        Are you saying the the deployment won't work entirely?
        Can you show me your site.yaml? I'm constantly doing deployment and does not encounter the issue.
        And you can see our deployment is still working for our smoke testing CI.

        Show
        evans_ye Evans Ye added a comment - - edited Are you saying the the deployment won't work entirely? Can you show me your site.yaml? I'm constantly doing deployment and does not encounter the issue. And you can see our deployment is still working for our smoke testing CI .
        Hide
        cos Konstantin Boudnik added a comment -

        Here's the site.yaml I've used for the deployment. Didn't work without the patch

        Show
        cos Konstantin Boudnik added a comment - Here's the site.yaml I've used for the deployment. Didn't work without the patch
        Hide
        evans_ye Evans Ye added a comment -

        Works well in my env. I wonder what hiera version you're using? Mine is 1.3.4.
        We used to encounter a bug in hiera 1.3.0 that it can parse empty yaml file. I suspect this is another similar one.

        Show
        evans_ye Evans Ye added a comment - Works well in my env. I wonder what hiera version you're using? Mine is 1.3.4. We used to encounter a bug in hiera 1.3.0 that it can parse empty yaml file. I suspect this is another similar one.
        Hide
        cos Konstantin Boudnik added a comment - - edited

        My Hiera is whatever the standard one on our 14.04 image, and that's the niceness of using the same environment. And quick check shows that is 1.3.0
        BTW, The reason I think this is the real issue, is because the rest of the kerberos vars are commented out but this one. Go figure. Shall we get this in? What do you think?

        Show
        cos Konstantin Boudnik added a comment - - edited My Hiera is whatever the standard one on our 14.04 image, and that's the niceness of using the same environment. And quick check shows that is 1.3.0 BTW, The reason I think this is the real issue, is because the rest of the kerberos vars are commented out but this one. Go figure. Shall we get this in? What do you think?
        Hide
        evans_ye Evans Ye added a comment -

        I'm curious about how you get the cluster deployed. With hiera 1.3.0, puppet apply will complain noha.yaml is empty. Which is exactly why we require to install puppetlab repo on ubuntu in bigtop_toolchain/bin/puppetize.sh. It will install newer hiera on system.

        And i saw many other uncommented kerberos realm setting for other components:

        hue::server::kerberos_realm: "%{hiera('kerberos::site::realm')}"
        hadoop-zookeeper::server::kerberos_realm: "%{hiera('kerberos::site::realm')}"
        hadoop-hbase::common_config::kerberos_realm: "%{hiera('kerberos::site::realm')}"
        solr::server::kerberos_realm: "%{hiera('kerberos::site::realm')}"
        

        My thought is not to comment out them because if otherwise we'd like to deploy kerberos enabled cluster, we need to either uncomment those lines in cluster.yaml, or add those lines in site.yaml manually, which looks odd.
        I think we better upgrade our images to have hiera version > 1.3.0. Thoughts?

        Show
        evans_ye Evans Ye added a comment - I'm curious about how you get the cluster deployed. With hiera 1.3.0, puppet apply will complain noha.yaml is empty. Which is exactly why we require to install puppetlab repo on ubuntu in bigtop_toolchain/bin/puppetize.sh . It will install newer hiera on system. And i saw many other uncommented kerberos realm setting for other components: hue::server::kerberos_realm: "%{hiera('kerberos::site::realm')}" hadoop-zookeeper::server::kerberos_realm: "%{hiera('kerberos::site::realm')}" hadoop-hbase::common_config::kerberos_realm: "%{hiera('kerberos::site::realm')}" solr::server::kerberos_realm: "%{hiera('kerberos::site::realm')}" My thought is not to comment out them because if otherwise we'd like to deploy kerberos enabled cluster, we need to either uncomment those lines in cluster.yaml, or add those lines in site.yaml manually, which looks odd. I think we better upgrade our images to have hiera version > 1.3.0. Thoughts?
        Hide
        cos Konstantin Boudnik added a comment -

        Well, the bug is extremely easy to reproduce on the stock bigtop/slave ubuntu 14.04. Just run a container, follow the steps in the deployment readme and you'll hit it right on 'puppet apply'. With the patch in place everything goes smoothly.

        We surely should be looking for a better solution - ie Hiera upgrade - but the deployment on the standard build image is broken right now and we can fix it easily.

        Show
        cos Konstantin Boudnik added a comment - Well, the bug is extremely easy to reproduce on the stock bigtop/slave ubuntu 14.04. Just run a container, follow the steps in the deployment readme and you'll hit it right on 'puppet apply'. With the patch in place everything goes smoothly. We surely should be looking for a better solution - ie Hiera upgrade - but the deployment on the standard build image is broken right now and we can fix it easily.
        Hide
        evans_ye Evans Ye added a comment -

        I can reproduce. Honestly, I still don't think its a good idea to add this workaround.
        Even we fix hadoop, hadoop-hbase::common_config::kerberos_realm will fail.
        Our code is ready to release new set of docker images with new hiera(BIGTOP-1996).
        Let me try to push those new trunk images.

        Show
        evans_ye Evans Ye added a comment - I can reproduce. Honestly, I still don't think its a good idea to add this workaround. Even we fix hadoop, hadoop-hbase::common_config::kerberos_realm will fail. Our code is ready to release new set of docker images with new hiera( BIGTOP-1996 ). Let me try to push those new trunk images.
        Hide
        cos Konstantin Boudnik added a comment -

        I think you're right - if we are about to fix the issue at large, the workaround doesn't make much sense. Thank you!

        Show
        cos Konstantin Boudnik added a comment - I think you're right - if we are about to fix the issue at large, the workaround doesn't make much sense. Thank you!
        Hide
        cos Konstantin Boudnik added a comment -

        Shall we close as the docker image is getting fixed?

        Show
        cos Konstantin Boudnik added a comment - Shall we close as the docker image is getting fixed?
        Hide
        evans_ye Evans Ye added a comment -

        Yes. I think I'll come back and close this once our puppet has been updated.

        Show
        evans_ye Evans Ye added a comment - Yes. I think I'll come back and close this once our puppet has been updated.
        Hide
        evans_ye Evans Ye added a comment -

        Newly built puppet and deploy images are pushed on docker hub.
        http://ci.bigtop.apache.org:8080/view/Docker/job/Docker-Puppet/
        http://ci.bigtop.apache.org:8080/view/Docker/job/Docker-Deploy/
        Sorry for quite a long delay.

        Show
        evans_ye Evans Ye added a comment - Newly built puppet and deploy images are pushed on docker hub. http://ci.bigtop.apache.org:8080/view/Docker/job/Docker-Puppet/ http://ci.bigtop.apache.org:8080/view/Docker/job/Docker-Deploy/ Sorry for quite a long delay.

          People

          • Assignee:
            cos Konstantin Boudnik
            Reporter:
            cos Konstantin Boudnik
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development