[AMBARI-17970] Ambari Sends Cached Configurations On Initial Task Execution Attempt

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.4.0
• Component/s: ambari-server
• Assignee: Jonathan Hurley
• Reporter: Jonathan Hurley
• Created: 31/Jul/16 00:52
• Updated: 01/Aug/16 06:42
• Resolved: 01/Aug/16 04:56

Description

• Install Ambari with HDP 2.4 and Ranger
• Perform a rolling upgrade HDP 2.5, but don't finalize
• Start a downgrade

The first command when a downgrade starts is the restart command for Ranger Admin. The daemon is currently running on HDP 2.5, however the restart command which is sent must be for HDP 2.4 as it will have the 2.4 configurations to write out.

The command fails with:

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py", line 199, in <module>
    RangerAdmin().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute
    method(env)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 685, in restart
    self.stop(env, upgrade_type=upgrade_type)
  File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py", line 60, in stop
    import params
  File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/params.py", line 178, in <module>
    audit_jdbc_url = format('jdbc:postgresql://{db_host}/{ranger_auditdb_name}') if stack_supports_ranger_audit_db else None
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/format.py",
...
packages/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__
    raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!")
resource_management.core.exceptions.Fail: Configuration parameter 'audit_db_name' was not found in configurations dictionary!

The audit_db_name parameter was removed from Ranger in HDP 2.5. On the downgrade restart command, we are supposed to be sending HDP 2.4 configurations. However, audit_db_name is missing.

Well, it looks like what we have here is an EclipseLink caching problem.

Essentually, at time index 15:47:22,172, thread ambari-action-scheduler gets the wrong desired configs from the entities that JPA has cached for it. However, a mere second later at 15:47:23,258, it has the right ones.

It's the first second that is so critical; ambari-client-thread-74 updates the entities and can see the correct data immediately. But it takes ambari-action-scheduler a second or so to see them.

2016-07-29 15:47:21,917 INFO  [ambari-client-thread-74] (ClusterImpl.java:3347) applyLatestConfigurations() - Applied latest configurations for c1 on stack HDP-2.4. The desired configurations are now {zookeeper-env={tag=TOPOLOGY_RESOLVED}, zookeeper-log4j={tag=TOPOLOGY_RESOLVED}, cluster-env={tag=TOPOLOGY_RESOLVED}, zoo.cfg={tag=TOPOLOGY_RESOLVED}}

2016-07-29 15:47:22,172 INFO  [ambari-action-scheduler] (ExecutionCommandWrapper.java:131) getExecutionCommand() - While scheduling task 102 on cluster c1, configurations are being refreshed using desired configurations of {zookeeper-env={tag=version1469808399092}, zookeeper-log4j={tag=version1469808399069}, cluster-env={tag=version1469808399109}, zoo.cfg={tag=version1469808399131}}

2016-07-29 15:47:23,258 INFO  [ambari-action-scheduler] (ExecutionCommandWrapper.java:131) getExecutionCommand() - While scheduling task 102 on cluster c1, configurations are being refreshed using desired configurations of {zookeeper-env={tag=TOPOLOGY_RESOLVED}, zookeeper-log4j={tag=TOPOLOGY_RESOLVED}, cluster-env={tag=TOPOLOGY_RESOLVED}, zoo.cfg={tag=TOPOLOGY_RESOLVED}}

Issue Links

• links to: Reviewboard