Uploaded image for project: 'Ambari'
  1. Ambari
  2. AMBARI-24462

ambari-server upgrade stuck with NPE

    XMLWordPrintableJSON

    Details

      Description

      ====================================
      Ambari 2.6.2
      HDP 2.6.5.0-292
      Cluster Details : https://github.com/hortonworks/HCube#stanley-hotel-hdp-26
      Please login with okta credential to these cluster machines
      https://hcube1-1n01.eng.hortonworks.com:8443/
      ====================================

      This is cluster is being upgraded to Ambari 2.7.1 and the schema upgrade step is hung.

      root@hcube1-1n01 ~]# ambari-server upgrade
      Using python  /usr/bin/python
      Upgrading ambari-server
      INFO: Upgrade Ambari Server
      INFO: Updating Ambari Server properties in ambari.properties ...
      INFO: Updating Ambari Server properties in ambari-env.sh ...
      INFO: Original file ambari-env.sh kept
      WARNING: Original file krb5JAASLogin.conf kept
      INFO: File krb5JAASLogin.conf updated.
      INFO: Fixing database objects owner
      Ambari Server configured for Postgres. Confirm you have made a backup of the Ambari Server database [y/n] (n)? y
      INFO: Upgrading database schema
      
      
      
      

      Below exception is noticed in ambari-server.log

      2018-08-10 06:57:50,795 ERROR [main] AbstractUpgradeCatalog:375 - Error in transaction
      java.lang.NullPointerException
              at org.apache.ambari.server.controller.KerberosHelperImpl.addIdentities(KerberosHelperImpl.java:1617)
              at org.apache.ambari.server.serveraction.kerberos.AbstractPrepareKerberosServerAction.processServiceComponents(AbstractPrepareKerberosServerAction.java:184)
              at org.apache.ambari.server.serveraction.kerberos.AbstractPrepareKerberosServerAction.processServiceComponentHosts(AbstractPrepareKerberosServerAction.java:94)
              at org.apache.ambari.server.upgrade.UpgradeCatalog270$3.run(UpgradeCatalog270.java:1637)
              at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.executeInTransaction(AbstractUpgradeCatalog.java:367)
              at org.apache.ambari.server.upgrade.UpgradeCatalog270.updateKerberosConfigurations(UpgradeCatalog270.java:1633)
              at org.apache.ambari.server.upgrade.UpgradeCatalog270.executeDMLUpdates(UpgradeCatalog270.java:1060)
              at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeData(AbstractUpgradeCatalog.java:985)
              at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:237)
              at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:457)
      2018-08-10 06:57:50,799 ERROR [main] SchemaUpgradeHelper:239 - Upgrade failed.
      org.apache.ambari.server.AmbariException: Failed to upgrade kerberos tables
              at org.apache.ambari.server.upgrade.UpgradeCatalog270.updateKerberosConfigurations(UpgradeCatalog270.java:1644)
              at org.apache.ambari.server.upgrade.UpgradeCatalog270.executeDMLUpdates(UpgradeCatalog270.java:1060)
              at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeData(AbstractUpgradeCatalog.java:985)
              at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeDMLUpdates(SchemaUpgradeHelper.java:237)
              at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:457)
      Caused by: java.lang.RuntimeException: java.lang.NullPointerException
              at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.executeInTransaction(AbstractUpgradeCatalog.java:379)
              at org.apache.ambari.server.upgrade.UpgradeCatalog270.updateKerberosConfigurations(UpgradeCatalog270.java:1633)
              ... 4 more
      Caused by: java.lang.NullPointerException
              at org.apache.ambari.server.controller.KerberosHelperImpl.addIdentities(KerberosHelperImpl.java:1617)
              at org.apache.ambari.server.serveraction.kerberos.AbstractPrepareKerberosServerAction.processServiceComponents(AbstractPrepareKerberosServerAction.java:184)
              at org.apache.ambari.server.serveraction.kerberos.AbstractPrepareKerberosServerAction.processServiceComponentHosts(AbstractPrepareKerberosServerAction.java:94)
              at org.apache.ambari.server.upgrade.UpgradeCatalog270$3.run(UpgradeCatalog270.java:1637)
              at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.executeInTransaction(AbstractUpgradeCatalog.java:367)
              ... 5 more
      

      Attached is the complete ambari-server logs

      Below is the jstack dump of schemaupgradehelper

      [root@hcube1-1n01 ambari-server]# /opt/java/jdk1.8.0_152/bin/jstack -l 8239
      2018-08-10 07:10:00
      Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.152-b16 mixed mode):
      
      "Attach Listener" #40 daemon prio=9 os_prio=0 tid=0x00007f9a6c001000 nid=0x3afa waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
         Locked ownable synchronizers:
      	- None
      
      "DestroyJavaVM" #39 prio=5 os_prio=0 tid=0x00007f9b6c00d000 nid=0x2030 waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
         Locked ownable synchronizers:
      	- None
      
      "FileWatchdog:ambari.properties" #38 daemon prio=5 os_prio=0 tid=0x00007f9b6d684000 nid=0x2c19 waiting on condition [0x00007f9b2e9d4000]
         java.lang.Thread.State: TIMED_WAITING (sleeping)
      	at java.lang.Thread.sleep(Native Method)
      	at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:104)
      
         Locked ownable synchronizers:
      	- None
      
      "pool-6-thread-1" #37 prio=5 os_prio=0 tid=0x00007f9a10006800 nid=0x210f waiting on condition [0x00007f9b2eed9000]
         java.lang.Thread.State: WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x0000000080dc3270> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
      	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      
         Locked ownable synchronizers:
      	- None
      
      "pool-5-thread-1" #36 prio=5 os_prio=0 tid=0x00007f9b6d497800 nid=0x2108 waiting on condition [0x00007f9b2edd8000]
         java.lang.Thread.State: WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x0000000080dc2fe8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
      	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      
         Locked ownable synchronizers:
      	- None
      
      "pool-3-thread-1" #35 prio=5 os_prio=0 tid=0x00007f9b6d0cb000 nid=0x2107 waiting on condition [0x00007f9b2f2dd000]
         java.lang.Thread.State: WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x0000000080dc2b38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
      	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
      	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      
         Locked ownable synchronizers:
      	- None
      
      "server-action-executor-cache-timer" #23 daemon prio=5 os_prio=0 tid=0x00007f9b6da29000 nid=0x20d8 in Object.wait() [0x00007f9b34120000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x0000000080f0b6e0> (a java.util.TaskQueue)
      	at java.util.TimerThread.mainLoop(Timer.java:552)
      	- locked <0x0000000080f0b6e0> (a java.util.TaskQueue)
      	at java.util.TimerThread.run(Timer.java:505)
      
         Locked ownable synchronizers:
      	- None
      
      "InMemoryCredentialStore active cleanup timer" #22 daemon prio=5 os_prio=0 tid=0x00007f9b6d541800 nid=0x20cd waiting on condition [0x00007f9b35021000]
         java.lang.Thread.State: WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x0000000080f2c750> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
      	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      
         Locked ownable synchronizers:
      	- None
      
      "Service Thread" #20 daemon prio=9 os_prio=0 tid=0x00007f9b6c107800 nid=0x205c runnable [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
         Locked ownable synchronizers:
      	- None
      
      "C1 CompilerThread14" #19 daemon prio=9 os_prio=0 tid=0x00007f9b6c104800 nid=0x205b waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
         Locked ownable synchronizers:
      	- None
      
      "C1 CompilerThread13" #18 daemon prio=9 os_prio=0 tid=0x00007f9b6c102800 nid=0x205a waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
         Locked ownable synchronizers:
      	- None
      
      "C1 CompilerThread12" #17 daemon prio=9 os_prio=0 tid=0x00007f9b6c100000 nid=0x2059 waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
         Locked ownable synchronizers:
      	- None
      
      "C1 CompilerThread11" #16 daemon prio=9 os_prio=0 tid=0x00007f9b6c0fe000 nid=0x2058 waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
         Locked ownable synchronizers:
      	- None
      
      "C1 CompilerThread10" #15 daemon prio=9 os_prio=0 tid=0x00007f9b6c0fb800 nid=0x2057 waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
         Locked ownable synchronizers:
      	- None
      
      "C2 CompilerThread9" #14 daemon prio=9 os_prio=0 tid=0x00007f9b6c0f9800 nid=0x2056 waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
         Locked ownable synchronizers:
      	- None
      
      "C2 CompilerThread8" #13 daemon prio=9 os_prio=0 tid=0x00007f9b6c0f7800 nid=0x2054 waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
         Locked ownable synchronizers:
      	- None
      
      "C2 CompilerThread7" #12 daemon prio=9 os_prio=0 tid=0x00007f9b6c0f5000 nid=0x2053 waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
         Locked ownable synchronizers:
      	- None
      
      "C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x00007f9b6c0f3000 nid=0x2052 waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
         Locked ownable synchronizers:
      	- None
      
      "C2 CompilerThread5" #10 daemon prio=9 os_prio=0 tid=0x00007f9b6c0f0800 nid=0x2051 waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
         Locked ownable synchronizers:
      	- None
      
      "C2 CompilerThread4" #9 daemon prio=9 os_prio=0 tid=0x00007f9b6c0ee800 nid=0x2050 runnable [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
         Locked ownable synchronizers:
      	- None
      
      "C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007f9b6c0ec000 nid=0x204f waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
         Locked ownable synchronizers:
      	- None
      
      "C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f9b6c0ea000 nid=0x204e waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
         Locked ownable synchronizers:
      	- None
      
      "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f9b6c0e8000 nid=0x204d waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
         Locked ownable synchronizers:
      	- None
      
      "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f9b6c0e5000 nid=0x204c waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
         Locked ownable synchronizers:
      	- None
      
      "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f9b6c0e3800 nid=0x204b runnable [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
         Locked ownable synchronizers:
      	- None
      
      "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f9b6c0af800 nid=0x204a in Object.wait() [0x00007f9b3d0e2000]
         java.lang.Thread.State: WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x000000008020b148> (a java.lang.ref.ReferenceQueue$Lock)
      	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
      	- locked <0x000000008020b148> (a java.lang.ref.ReferenceQueue$Lock)
      	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
      	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
      
         Locked ownable synchronizers:
      	- None
      
      "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f9b6c0ab000 nid=0x2049 in Object.wait() [0x00007f9b3d1e3000]
         java.lang.Thread.State: WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x000000008020c698> (a java.lang.ref.Reference$Lock)
      	at java.lang.Object.wait(Object.java:502)
      	at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
      	- locked <0x000000008020c698> (a java.lang.ref.Reference$Lock)
      	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
      
         Locked ownable synchronizers:
      	- None
      
      "VM Thread" os_prio=0 tid=0x00007f9b6c0a3800 nid=0x2048 runnable 
      
      "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f9b6c022800 nid=0x2031 runnable 
      
      "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f9b6c024000 nid=0x2032 runnable 
      
      "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f9b6c026000 nid=0x2033 runnable 
      
      "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f9b6c028000 nid=0x2034 runnable 
      
      "GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007f9b6c029800 nid=0x2035 runnable 
      
      "GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007f9b6c02b800 nid=0x2036 runnable 
      
      "GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007f9b6c02d000 nid=0x2037 runnable 
      
      "GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007f9b6c02f000 nid=0x2038 runnable 
      
      "GC task thread#8 (ParallelGC)" os_prio=0 tid=0x00007f9b6c031000 nid=0x2039 runnable 
      
      "GC task thread#9 (ParallelGC)" os_prio=0 tid=0x00007f9b6c032800 nid=0x203a runnable 
      
      "GC task thread#10 (ParallelGC)" os_prio=0 tid=0x00007f9b6c034800 nid=0x203b runnable 
      
      "GC task thread#11 (ParallelGC)" os_prio=0 tid=0x00007f9b6c036800 nid=0x203c runnable 
      
      "GC task thread#12 (ParallelGC)" os_prio=0 tid=0x00007f9b6c038000 nid=0x203d runnable 
      
      "GC task thread#13 (ParallelGC)" os_prio=0 tid=0x00007f9b6c03a000 nid=0x203e runnable 
      
      "GC task thread#14 (ParallelGC)" os_prio=0 tid=0x00007f9b6c03c000 nid=0x203f runnable 
      
      "GC task thread#15 (ParallelGC)" os_prio=0 tid=0x00007f9b6c03d800 nid=0x2040 runnable 
      
      "GC task thread#16 (ParallelGC)" os_prio=0 tid=0x00007f9b6c03f800 nid=0x2041 runnable 
      
      "GC task thread#17 (ParallelGC)" os_prio=0 tid=0x00007f9b6c041800 nid=0x2042 runnable 
      
      "GC task thread#18 (ParallelGC)" os_prio=0 tid=0x00007f9b6c043000 nid=0x2043 runnable 
      
      "GC task thread#19 (ParallelGC)" os_prio=0 tid=0x00007f9b6c045000 nid=0x2044 runnable 
      
      "GC task thread#20 (ParallelGC)" os_prio=0 tid=0x00007f9b6c046800 nid=0x2045 runnable 
      
      "GC task thread#21 (ParallelGC)" os_prio=0 tid=0x00007f9b6c048800 nid=0x2046 runnable 
      
      "GC task thread#22 (ParallelGC)" os_prio=0 tid=0x00007f9b6c04a800 nid=0x2047 runnable 
      
      "VM Periodic Task Thread" os_prio=0 tid=0x00007f9b6c10c800 nid=0x205d waiting on condition 
      
      JNI global references: 328
      

        Attachments

          Activity

            People

            • Assignee:
              amagyar Attila Magyar
              Reporter:
              amagyar Attila Magyar
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 1.5h
                1.5h