[AMBARI-21854] Adapt Repository Files For Existing Deployments

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.6.0
• Fix Version/s: 2.6.0
• Component/s: ambari-agent
• Epic Link: Ambari: Patch Upgrades
• Assignee: Dmytro Grinenko
• Reporter: Jonathan Hurley
• Created: 30/Aug/17 14:28
• Updated: 06/Oct/17 14:46
• Resolved: 06/Oct/17 14:46

Description

The recent changes in repository creation and management (AMBARI-20871, AMBARI-21719, AMBARI-21398) has caused a regression with previously installed packages. Using RPM as an example, consider the following case:

• A version of Ambari is installed which writes out a repo file for HDP. This file could be named anything and may vary depending on the version of Ambari being used. It could be hdp.repo, hdp-2.6.repo, or really any variation. The point is that it's not predictable.

• The repo ID inside of the file can also be anything. In some cases it may be "HDP-2.6.2.0" or "HDP-2.6.2.0-193". But once again, the point is that it's not predictable.

• What is known is that packages previously installed with this repo are now associated with it permanently:

  [root@c6401 yum.repos.d]# yum list installed | grep hive2
  hive2_2_6_2_0_193.noarch                2.1.0.2.6.2.0-193       @HDP-2.6.2.0-193
  hive2_2_6_2_0_193-jdbc.noarch           2.1.0.2.6.2.0-193       @HDP-2.6.2.0-193
  

• Ambari has now changed the way in which we create repo files and the naming scheme we use for the repo ID. Consider a fresh installation of Ambari 2.6:
  /etc/yum.repos.d/ambari-hdp-1.repo

  [HDP-2.5-repo-1]
  name=HDP-2.5-repo-1
  baseurl=http://repo.ambari.apache.org/hdp/centos6/HDP-2.5.0.0-1237
  
  path=/
  enabled=1
  gpgcheck=0
  
  [HDP-UTILS-1.1.0.21-repo-1]
  name=HDP-UTILS-1.1.0.21-repo-1
  baseurl=http://repo.ambari.apache.org/hdp/centos6/HDP-UTILS-1.1.0.21
  
  path=/
  enabled=1
  gpgcheck=0
  

– "1" is the ID of the repo_version entry in Ambari - so the filename is ambari-<stack-name>-<repo-version-id>
– Yum Repo ID (HDP-2.5-repo-1) - the repo name used in the file is <stack-name><stack-version>-repo<repo-version-id>

This presents a problem when upgrading a prior version of Ambari. Although there is still only 1 record in the repo_version table, we create a 2nd repo file in /etc/yum.repos.d with an entirely different repository ID and name. Because the packages were previously installed with another repo identifier, our new code which restricts to the repository associated with the command, can't find packages and produces an error.

• This prevents new components from being added to hosts
• Client restarts and reinstalls fail
• Sysprepped Hosts can't be managed

The yum command doesn't really allow for us to determine the packages installed from other repos nor does it allow the reassignment of installed packages from one repo to another.

Consider the case where Hive is already installed and then the cluster is upgraded to Ambari 2.6. When we go to add new hive components, we are restricting the packages to that of the current repo (HDP-2.5-repo-1):

[root@c6401 ~]# yum list available --disablerepo=* --enablerepo=HDP-2.5-repo-1 | grep hive2
hive2.noarch                            2.1.0.2.5.0.0-1237.el6    HDP-2.5-repo-1
hive2-jdbc.noarch                       2.1.0.2.5.0.0-1237.el6    HDP-2.5-repo-1

Unfortunately, this misses the hive already installed since they are associated with another repo (@HDP-2.6.2.0-193)

hive2_2_6_2_0_193.noarch                2.1.0.2.6.2.0-193       @HDP-2.6.2.0-193
hive2_2_6_2_0_193-jdbc.noarch           2.1.0.2.6.2.0-193       @HDP-2.6.2.0-193

I think the following needs to be done to fix this situation:

• If the repo file exists, then do nothing if the file contents match
• If the repo file does not exist, then scan /etc/yum.repos.d to see if any repos match the URL. If we hit a match, then we need to scan the repo to extract the old, original repository ID

Issue Links

• links to: Review Board