Uploaded image for project: 'Infrastructure'
  1. Infrastructure
  2. INFRA-20657

Fix puppet installation on lucene2-us-west

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Fix Version/s: None
    • Component/s: VM
    • Labels:
      None
    • Project:
      Lucene

      Description

      The Lucene build infrastruture had some issues withone of our 2 VMs for Jenkins. Lucene1 has Ubuntu 18.04 and is working fine, while Lucene2 had (!) the problem of still running with Ubuntu 16.04. For this reason, we were not able to use HTTPS correctly with python 3.5 (preinstalled on Ubuntu 16) not having all root certificates and SSL methods. This is only fixed in python 3.6, which is part of Ubuntu18.

      I fixed the issue by doing do-release-upgrade with preserving all /etc/apt/sources.list.d puppet additions. After the upgrade and a reboot all was fine again, the only problem is that puppet no longer comes up:

      Aug 7 16:49:08 lucene2-us-west puppet-agent[30327]: Starting Puppet client version 5.4.0
      Aug 7 16:49:08 lucene2-us-west systemd[1]: Started Puppet agent.
      Aug 7 16:49:08 lucene2-us-west puppet-agent[30332]: Downgrading to PSON for future requests
      Aug 7 16:49:08 lucene2-us-west puppet-agent[30332]: Unable to fetch my node definition, but the agent run will continue:
      Aug 7 16:49:08 lucene2-us-west puppet-agent[30332]: Find /puppet/v3/node/lucene2-us-west.apache.org?environment=production&configured_environment=producti... resulted in 404 with the message: Not Found: Could not find environment 'puppet'
      Aug 7 16:49:08 lucene2-us-west puppet-agent[30332]: (/File[/var/cache/puppet/facts.d]) Could not evaluate: Could not retrieve information from environment production source(s) puppet:///pluginfacts
      Aug 7 16:49:09 lucene2-us-west puppet-agent[30332]: (/File[/var/cache/puppet/lib]) Could not evaluate: Could not retrieve information from environment production source(s) puppet:///plugins
      Aug 7 16:49:09 lucene2-us-west puppet-agent[30332]: Could not retrieve catalog from remote server: Find /puppet/v3/catalog/lucene2-us-west.apache.org?environment=production&facts_format=pson&facts=%257... resulted in 404 with the message: Not Found: Could not find environment 'puppet'
      Aug 7 16:49:09 lucene2-us-west puppet-agent[30332]: Could not retrieve catalog; skipping run

      When checking the lucene1 machine which already had Ubuntu 18.04, it looks like it uses a newer puppet version shipped from a separate non-Ubuntu Package. Lucene2 is using the Ubuntu shipped puppet and also has configuration for this one. Updating to same version as on lucene1 does not work, too.

      I'm sorry for the uncommunicated update of the OS, but I was not aware that this breaks puppet. To fix it I'd suggest to do the following:
      - Remove Puppet completely, also rm /etc/apt/sources.list.d/* (to get rid of all puppet additions)
      - Bootstrap puppet like on lucene1 (newer version, better configs)

      I can't do this on my own. The machine works fine, it is just no longer correctly updated (like jenkins JVMs,...)

      Uwe

        Attachments

          Activity

            People

            • Assignee:
              gmcdonald Gav McDuck
              Reporter:
              uschindler Uwe Schindler
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: