[AMBARI-19130] Downgrade Can Create Multiple Mappings For Latest Configs

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.5.0
• Component/s: ambari-server
• Assignee: Jonathan Hurley
• Reporter: Jonathan Hurley
• Created: 07/Dec/16 18:27
• Updated: 09/Dec/16 14:35
• Resolved: 09/Dec/16 14:35

Description

During a downgrade which crosses major stack versions (such as from HDP 2.y to HDP 2.x), Ambari attempts to set the latest configurations from HDP 2.x as "current". However, after the downgrade succeeds, the database fails with the following consistency check:

ERROR - You have config(s), in cluster c1, that is(are) selected more than once in clusterconfigmapping table: ranger-storm-plugin-properties,mapred-env,webhcat-env,hive-exec-log4j,hive-log4j,webhcat-log4j,storm-env,yarn-log4j,hcat-env

The problem is actually not with clusterconfig but with the state of the clusterconfigmapping table before upgrade. Apparently, it is possible to have multiple mappings for the same config. We match on the config type and tag (such as hdfs-site / version1234566789). There should never, EVER be duplicate mappings for the same config and version tag.

The current downgrade code doesn't "break" after finding its first match:

      for(ClusterConfigMappingEntity configMappingEntity: configMappingEntities){
        String type = configMappingEntity.getType();
        String tag =  configMappingEntity.getTag();

        for (ClusterConfigMappingEntity latest : latestConfigMappingByStack) {
          String latestType = latest.getType();
          String latestTag = latest.getTag();

          // find the latest config of a given mapping entity
          if (StringUtils.equals(type, latestType) && StringUtils.equals(tag, latestTag)) {
            LOG.info("{} with version tag {} is selected for stack {}", type, tag, stackId.toString());
            configMappingEntity.setSelected(1);
          }
        }
      }

I'm not sure that breaking will work here since we don't know the ordering. We could order the results and then break on the first match. In any event, it's a problem. But it's only a problem for the latest version of a config since that's what we're going to try to make selected.

Here's a query from a test cluster which shows that if this cluster were to be downgraded, 5 configs would have multiple mappings created. This is because for the latest config, there are multiple mappings in clusterconfigmapping.

SELECT
  mapping.type_name,
  mapping.version_tag,
  COUNT(*)
FROM clusterconfigmapping mapping
JOIN (SELECT
  config.type_name,
  config.version_tag,
  MAX(config.version) AS latest_version
FROM clusterconfig config
GROUP BY config.type_name) AS latestConfig
  ON latestConfig.type_name = mapping.type_name
  AND latestConfig.version_tag = mapping.version_tag
GROUP BY
    type_name, version_tag  
HAVING COUNT(*) > 1

+-------------------------------+----------------------+----------+
| type_name                     | version_tag          | COUNT(*) |
+-------------------------------+----------------------+----------+
| hcat-env                      | version1             |        5 |
| hive-exec-log4j               | version1             |        5 |
| hive-log4j                    | version1             |        5 |
| ranger-hive-plugin-properties | version1436918769763 |        3 |
| webhcat-env                   | version1             |        5 |
+-------------------------------+----------------------+----------+
5 rows in set (0.00 sec)

STR:

1. Install a cluster, and instrument the clusterconfigmapping table to have multiple mappings for a given type (with only 1 currently selected).
2. Upgrade and then downgrade

Issue Links

• links to: Reviewboard