[AMBARI-24462] ambari-server upgrade stuck with NPE - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Blocker
Resolution: Fixed
Affects Version/s: 2.7.0
Fix Version/s: 2.7.1
Component/s: ambari-server
Labels:
- pull-request-available

Description

====================================
Ambari 2.6.2
HDP 2.6.5.0-292
Cluster Details : https://github.com/hortonworks/HCube#stanley-hotel-hdp-26
Please login with okta credential to these cluster machines
https://hcube1-1n01.eng.hortonworks.com:8443/
====================================

This is cluster is being upgraded to Ambari 2.7.1 and the schema upgrade step is hung.

root@hcube1-1n01 ~]# ambari-server upgrade
Using python  /usr/bin/python
Upgrading ambari-server
INFO: Upgrade Ambari Server
INFO: Updating Ambari Server properties in ambari.properties ...
INFO: Updating Ambari Server properties in ambari-env.sh ...
INFO: Original file ambari-env.sh kept
WARNING: Original file krb5JAASLogin.conf kept
INFO: File krb5JAASLogin.conf updated.
INFO: Fixing database objects owner
Ambari Server configured for Postgres. Confirm you have made a backup of the Ambari Server database [y/n] (n)? y
INFO: Upgrading database schema

Below exception is noticed in ambari-server.log

2018-08-10 06:57:50,795 ERROR [main] AbstractUpgradeCatalog:375 - Error in transaction
java.lang.NullPointerException
        at org.apache.ambari.server.controller.KerberosHelperImpl.addIdentities(KerberosHelperImpl.java:1617)
        at org.apache.ambari.server.serveraction.kerberos.AbstractPrepareKerberosServerAction.processServiceComponents(AbstractPrepareKerberosServerAction.java:184)
        at org.apache.ambari.server.serveraction.kerberos.AbstractPrepareKerberosServerAction.processServiceComponentHosts(AbstractPrepareKerberosServerAction.java:94)
        at org.apache.ambari.server.upgrade.UpgradeCatalog270$3.run(UpgradeCatalog270.java:1637)
        at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.executeInTransaction(AbstractUpgradeCatalog.java:367)
        at org.apache.ambari.server.upgrade.UpgradeCatalog270.updateKerberosConfigurations(UpgradeCatalog270.java:1633)
        at org.apache.ambari.server.upgrade.UpgradeCatalog270.executeDMLUpdates(UpgradeCatalog270.java:1060)
        at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeData(AbstractUpgradeCatalog.java:985)
        at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:237)
        at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:457)
2018-08-10 06:57:50,799 ERROR [main] SchemaUpgradeHelper:239 - Upgrade failed.
org.apache.ambari.server.AmbariException: Failed to upgrade kerberos tables
        at org.apache.ambari.server.upgrade.UpgradeCatalog270.updateKerberosConfigurations(UpgradeCatalog270.java:1644)
        at org.apache.ambari.server.upgrade.UpgradeCatalog270.executeDMLUpdates(UpgradeCatalog270.java:1060)
        at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeData(AbstractUpgradeCatalog.java:985)
        at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:237)
        at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:457)
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
        at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.executeInTransaction(AbstractUpgradeCatalog.java:379)
        at org.apache.ambari.server.upgrade.UpgradeCatalog270.updateKerberosConfigurations(UpgradeCatalog270.java:1633)
        ... 4 more
Caused by: java.lang.NullPointerException
        at org.apache.ambari.server.controller.KerberosHelperImpl.addIdentities(KerberosHelperImpl.java:1617)
        at org.apache.ambari.server.serveraction.kerberos.AbstractPrepareKerberosServerAction.processServiceComponents(AbstractPrepareKerberosServerAction.java:184)
        at org.apache.ambari.server.serveraction.kerberos.AbstractPrepareKerberosServerAction.processServiceComponentHosts(AbstractPrepareKerberosServerAction.java:94)
        at org.apache.ambari.server.upgrade.UpgradeCatalog270$3.run(UpgradeCatalog270.java:1637)
        at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.executeInTransaction(AbstractUpgradeCatalog.java:367)
        ... 5 more

Attached is the complete ambari-server logs

Below is the jstack dump of schemaupgradehelper

[root@hcube1-1n01 ambari-server]# /opt/java/jdk1.8.0_152/bin/jstack -l 8239
2018-08-10 07:10:00
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.152-b16 mixed mode):

"Attach Listener" #40 daemon prio=9 os_prio=0 tid=0x00007f9a6c001000 nid=0x3afa waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"DestroyJavaVM" #39 prio=5 os_prio=0 tid=0x00007f9b6c00d000 nid=0x2030 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"FileWatchdog:ambari.properties" #38 daemon prio=5 os_prio=0 tid=0x00007f9b6d684000 nid=0x2c19 waiting on condition [0x00007f9b2e9d4000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:104)

   Locked ownable synchronizers:
	- None

"pool-6-thread-1" #37 prio=5 os_prio=0 tid=0x00007f9a10006800 nid=0x210f waiting on condition [0x00007f9b2eed9000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x0000000080dc3270> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
	- None

"pool-5-thread-1" #36 prio=5 os_prio=0 tid=0x00007f9b6d497800 nid=0x2108 waiting on condition [0x00007f9b2edd8000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x0000000080dc2fe8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
	- None

"pool-3-thread-1" #35 prio=5 os_prio=0 tid=0x00007f9b6d0cb000 nid=0x2107 waiting on condition [0x00007f9b2f2dd000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x0000000080dc2b38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
	- None

"server-action-executor-cache-timer" #23 daemon prio=5 os_prio=0 tid=0x00007f9b6da29000 nid=0x20d8 in Object.wait() [0x00007f9b34120000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x0000000080f0b6e0> (a java.util.TaskQueue)
	at java.util.TimerThread.mainLoop(Timer.java:552)
	- locked <0x0000000080f0b6e0> (a java.util.TaskQueue)
	at java.util.TimerThread.run(Timer.java:505)

   Locked ownable synchronizers:
	- None

"InMemoryCredentialStore active cleanup timer" #22 daemon prio=5 os_prio=0 tid=0x00007f9b6d541800 nid=0x20cd waiting on condition [0x00007f9b35021000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x0000000080f2c750> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
	- None

"Service Thread" #20 daemon prio=9 os_prio=0 tid=0x00007f9b6c107800 nid=0x205c runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"C1 CompilerThread14" #19 daemon prio=9 os_prio=0 tid=0x00007f9b6c104800 nid=0x205b waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"C1 CompilerThread13" #18 daemon prio=9 os_prio=0 tid=0x00007f9b6c102800 nid=0x205a waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"C1 CompilerThread12" #17 daemon prio=9 os_prio=0 tid=0x00007f9b6c100000 nid=0x2059 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"C1 CompilerThread11" #16 daemon prio=9 os_prio=0 tid=0x00007f9b6c0fe000 nid=0x2058 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"C1 CompilerThread10" #15 daemon prio=9 os_prio=0 tid=0x00007f9b6c0fb800 nid=0x2057 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"C2 CompilerThread9" #14 daemon prio=9 os_prio=0 tid=0x00007f9b6c0f9800 nid=0x2056 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"C2 CompilerThread8" #13 daemon prio=9 os_prio=0 tid=0x00007f9b6c0f7800 nid=0x2054 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"C2 CompilerThread7" #12 daemon prio=9 os_prio=0 tid=0x00007f9b6c0f5000 nid=0x2053 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x00007f9b6c0f3000 nid=0x2052 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"C2 CompilerThread5" #10 daemon prio=9 os_prio=0 tid=0x00007f9b6c0f0800 nid=0x2051 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"C2 CompilerThread4" #9 daemon prio=9 os_prio=0 tid=0x00007f9b6c0ee800 nid=0x2050 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007f9b6c0ec000 nid=0x204f waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f9b6c0ea000 nid=0x204e waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f9b6c0e8000 nid=0x204d waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f9b6c0e5000 nid=0x204c waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f9b6c0e3800 nid=0x204b runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f9b6c0af800 nid=0x204a in Object.wait() [0x00007f9b3d0e2000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x000000008020b148> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
	- locked <0x000000008020b148> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

   Locked ownable synchronizers:
	- None

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f9b6c0ab000 nid=0x2049 in Object.wait() [0x00007f9b3d1e3000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x000000008020c698> (a java.lang.ref.Reference$Lock)
	at java.lang.Object.wait(Object.java:502)
	at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
	- locked <0x000000008020c698> (a java.lang.ref.Reference$Lock)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

   Locked ownable synchronizers:
	- None

"VM Thread" os_prio=0 tid=0x00007f9b6c0a3800 nid=0x2048 runnable 

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f9b6c022800 nid=0x2031 runnable 

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f9b6c024000 nid=0x2032 runnable 

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f9b6c026000 nid=0x2033 runnable 

"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f9b6c028000 nid=0x2034 runnable 

"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007f9b6c029800 nid=0x2035 runnable 

"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007f9b6c02b800 nid=0x2036 runnable 

"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007f9b6c02d000 nid=0x2037 runnable 

"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007f9b6c02f000 nid=0x2038 runnable 

"GC task thread#8 (ParallelGC)" os_prio=0 tid=0x00007f9b6c031000 nid=0x2039 runnable 

"GC task thread#9 (ParallelGC)" os_prio=0 tid=0x00007f9b6c032800 nid=0x203a runnable 

"GC task thread#10 (ParallelGC)" os_prio=0 tid=0x00007f9b6c034800 nid=0x203b runnable 

"GC task thread#11 (ParallelGC)" os_prio=0 tid=0x00007f9b6c036800 nid=0x203c runnable 

"GC task thread#12 (ParallelGC)" os_prio=0 tid=0x00007f9b6c038000 nid=0x203d runnable 

"GC task thread#13 (ParallelGC)" os_prio=0 tid=0x00007f9b6c03a000 nid=0x203e runnable 

"GC task thread#14 (ParallelGC)" os_prio=0 tid=0x00007f9b6c03c000 nid=0x203f runnable 

"GC task thread#15 (ParallelGC)" os_prio=0 tid=0x00007f9b6c03d800 nid=0x2040 runnable 

"GC task thread#16 (ParallelGC)" os_prio=0 tid=0x00007f9b6c03f800 nid=0x2041 runnable 

"GC task thread#17 (ParallelGC)" os_prio=0 tid=0x00007f9b6c041800 nid=0x2042 runnable 

"GC task thread#18 (ParallelGC)" os_prio=0 tid=0x00007f9b6c043000 nid=0x2043 runnable 

"GC task thread#19 (ParallelGC)" os_prio=0 tid=0x00007f9b6c045000 nid=0x2044 runnable 

"GC task thread#20 (ParallelGC)" os_prio=0 tid=0x00007f9b6c046800 nid=0x2045 runnable 

"GC task thread#21 (ParallelGC)" os_prio=0 tid=0x00007f9b6c048800 nid=0x2046 runnable 

"GC task thread#22 (ParallelGC)" os_prio=0 tid=0x00007f9b6c04a800 nid=0x2047 runnable 

"VM Periodic Task Thread" os_prio=0 tid=0x00007f9b6c10c800 nid=0x205d waiting on condition 

JNI global references: 328

Attachments

Issue Links

links to

GitHub Pull Request #2043

GitHub Pull Request #2045

Activity

People

Assignee:: Attila Magyar

Reporter:: Attila Magyar

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 13/Aug/18 09:53

Updated:: 13/Aug/18 14:50

Resolved:: 13/Aug/18 13:28

Time Tracking

Estimated:

Not Specified

Remaining:

Logged:

1.5h