Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Incomplete
-
0.8.2
-
None
Description
Steps to reproduce:
1. Produce messages using hive hook - create view with 300 columns. Example, command below
create table t1_300(col0 string, col1 string, col2 string, col3 string, col4 string, col5 string, col6 string, col7 string, col8 string, col9 string, col10 string, col11 string, col12 string, col13 string, col14 string, col15 string, col16 string, col17 string, col18 string, col19 string, col20 string, col21 string, col22 string, col23 string, col24 string, col25 string, col26 string, col27 string, col28 string, col29 string, col30 string, col31 string, col32 string, col33 string, col34 string, col35 string, col36 string, col37 string, col38 string, col39 string, col40 string, col41 string, col42 string, col43 string, col44 string, col45 string, col46 string, col47 string, col48 string, col49 string, col50 string, col51 string, col52 string, col53 string, col54 string, col55 string, col56 string, col57 string, col58 string, col59 string, col60 string, col61 string, col62 string, col63 string, col64 string, col65 string, col66 string, col67 string, col68 string, col69 string, col70 string, col71 string, col72 string, col73 string, col74 string, col75 string, col76 string, col77 string, col78 string, col79 string, col80 string, col81 string, col82 string, col83 string, col84 string, col85 string, col86 string, col87 string, col88 string, col89 string, col90 string, col91 string, col92 string, col93 string, col94 string, col95 string, col96 string, col97 string, col98 string, col99 string, col100 string, col101 string, col102 string, col103 string, col104 string, col105 string, col106 string, col107 string, col108 string, col109 string, col110 string, col111 string, col112 string, col113 string, col114 string, col115 string, col116 string, col117 string, col118 string, col119 string, col120 string, col121 string, col122 string, col123 string, col124 string, col125 string, col126 string, col127 string, col128 string, col129 string, col130 string, col131 string, col132 string, col133 string, col134 string, col135 string, col136 string, col137 string, col138 string, col139 string, col140 string, col141 string, col142 string, col143 string, col144 string, col145 string, col146 string, col147 string, col148 string, col149 string, col150 string, col151 string, col152 string, col153 string, col154 string, col155 string, col156 string, col157 string, col158 string, col159 string, col160 string, col161 string, col162 string, col163 string, col164 string, col165 string, col166 string, col167 string, col168 string, col169 string, col170 string, col171 string, col172 string, col173 string, col174 string, col175 string, col176 string, col177 string, col178 string, col179 string, col180 string, col181 string, col182 string, col183 string, col184 string, col185 string, col186 string, col187 string, col188 string, col189 string, col190 string, col191 string, col192 string, col193 string, col194 string, col195 string, col196 string, col197 string, col198 string, col199 string, col200 string, col201 string, col202 string, col203 string, col204 string, col205 string, col206 string, col207 string, col208 string, col209 string, col210 string, col211 string, col212 string, col213 string, col214 string, col215 string, col216 string, col217 string, col218 string, col219 string, col220 string, col221 string, col222 string, col223 string, col224 string, col225 string, col226 string, col227 string, col228 string, col229 string, col230 string, col231 string, col232 string, col233 string, col234 string, col235 string, col236 string, col237 string, col238 string, col239 string, col240 string, col241 string, col242 string, col243 string, col244 string, col245 string, col246 string, col247 string, col248 string, col249 string, col250 string, col251 string, col252 string, col253 string, col254 string, col255 string, col256 string, col257 string, col258 string, col259 string, col260 string, col261 string, col262 string, col263 string, col264 string, col265 string, col266 string, col267 string, col268 string, col269 string, col270 string, col271 string, col272 string, col273 string, col274 string, col275 string, col276 string, col277 string, col278 string, col279 string, col280 string, col281 string, col282 string, col283 string, col284 string, col285 string, col286 string, col287 string, col288 string, col289 string, col290 string, col291 string, col292 string, col293 string, col294 string, col295 string, col296 string, col297 string, col298 string, col299 string); create view v_t1_300 as select * from t1_300;
2. With this Atlas throws out of memory exception. Below error log confirming the same.
[root@ctr-e134-1499953498516-209460-01-000004 atlas]# cat atlas.20171009-192118.err Exception in thread "pool-2-thread-6" Exception in thread "pool-2-thread-2" java.lang.OutOfMemoryError: Java heap space at java.util.HashMap$KeySet.iterator(HashMap.java:916) at java.util.HashSet.iterator(HashSet.java:172) at sun.nio.ch.Util$3.iterator(Util.java:324) at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:600) at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Exception in thread "Thread-22" Exception in thread "pool-2-thread-3" Exception in thread "HashSessionScavenger-0" java.lang.OutOfMemoryError: Java heap space at java.util.HashMap$KeySet.iterator(HashMap.java:916) at java.util.HashSet.iterator(HashSet.java:172) at sun.nio.ch.Util$3.iterator(Util.java:324) at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:600) at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) java.lang.OutOfMemoryError: Java heap space at java.util.concurrent.ConcurrentHashMap$ValuesView.iterator(ConcurrentHashMap.java:4683) at org.eclipse.jetty.server.session.HashSessionManager.scavenge(HashSessionManager.java:314) at org.eclipse.jetty.server.session.HashSessionManager$2.run(HashSessionManager.java:285) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) java.lang.OutOfMemoryError: Java heap space at java.util.HashMap$KeySet.iterator(HashMap.java:916) at java.util.HashSet.iterator(HashSet.java:172) at sun.nio.ch.Util$3.iterator(Util.java:324) at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:600) at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) java.lang.OutOfMemoryError: Java heap space Exception in thread "pool-2-thread-7" java.lang.OutOfMemoryError: Java heap space Exception in thread "pool-2-thread-1" java.lang.OutOfMemoryError: Java heap space [root@ctr-e134-1499953498516-209460-01-000004 atlas]#
3. After this step, Atlas became unresponsive, no requests are honoured. All requests are timedout.
4. JSTACK output confirming the deadlock.
[root@ctr-e134-1499953498516-209460-01-000004 atlas]# ps axu | grep Atlas atlas 131887 3.6 0.9 20609684 2612572 ? Sl Oct09 31:03 /usr/lib/jvm/java-openjdk/bin/java -Datlas.log.dir=/grid/0/log/atlas -Datlas.log.file=application.log -Datlas.home=/usr/hdp/.3.0-175/atlas -Datlas.conf=/usr/hdp/current/atlas-server/conf -Xms2048m -Xmx2048m -XX:MaxNewSize=600m -XX:MetaspaceSize=100m -XX:MaxMetaspaceSize=512m -server -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+PrintTenuringDistribution -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/grid/0/log/atlas/atlas_server.hprof -Xloggc:/grid/0/log/atlas/gc-worker.log -verbose:gc -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1m -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCTimeStamps -Dlog4j.configuration=atlas-log4j.xml -Djava..auth.login.config=/usr/hdp/current/atlas-server/conf/atlas_jaas.conf -classpath /usr/hdp/current/atlas-server/conf:/usr/hdp/current/atlas-server/server/webapp/atlas/WEB-INF/classes:/usr/hdp/current/atlas-server/server/webapp/atlas/WEB-INF/lib/*:/usr/hdp/.3.0-175/atlas/libext/*:/etc/hbase/conf org.apache.atlas.Atlas -app /usr/hdp/current/atlas-server/server/webapp/atlas root 1666487 0.0 0.0 10632 968 pts/1 S+ 09:34 0:00 grep --color=auto Atlas [root@ctr-e134-1499953498516-209460-01-000004 atlas]#
[root@ctr-e134-1499953498516-209460-01-000004 atlas]# /usr/lib/jvm/java-openjdk/bin/jstack -F -l 131887 Attaching to process ID 131887, please wait... Debugger attached successfully. Server compiler detected. JVM version is 25.144-b01 Deadlock Detection: java.lang.RuntimeException: Unable to deduce type of thread from address 0x00007f8838013800 (expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread) at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:169) at sun.jvm.hotspot.runtime.Threads.first(Threads.java:153) at sun.jvm.hotspot.runtime.DeadlockDetector.createThreadTable(DeadlockDetector.java:149) at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:56) at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:39) at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:62) at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45) at sun.jvm.hotspot.tools.JStack.run(JStack.java:66) at sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:260) at sun.jvm.hotspot.tools.Tool.start(Tool.java:223) at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118) at sun.jvm.hotspot.tools.JStack.main(JStack.java:92) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sun.tools.jstack.JStack.runJStackTool(JStack.java:140) at sun.tools.jstack.JStack.main(JStack.java:106) Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address 0x00007f8838013800 at sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62) at sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80) at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:165) ... 17 more Can't print deadlocks:Unable to deduce type of thread from address 0x00007f8838013800 (expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread) Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sun.tools.jstack.JStack.runJStackTool(JStack.java:140) at sun.tools.jstack.JStack.main(JStack.java:106) Caused by: java.lang.InternalError: Metadata does not appear to be polymorphic at sun.jvm.hotspot.types.basic.BasicTypeDataBase.findDynamicTypeForAddress(BasicTypeDataBase.java:278) at sun.jvm.hotspot.runtime.VirtualBaseConstructor.instantiateWrapperFor(VirtualBaseConstructor.java:102) at sun.jvm.hotspot.oops.Metadata.instantiateWrapperFor(Metadata.java:68) at sun.jvm.hotspot.memory.DictionaryEntry.klass(DictionaryEntry.java:71) at sun.jvm.hotspot.memory.DictionaryEntry.equals(DictionaryEntry.java:79) at sun.jvm.hotspot.memory.Dictionary.getEntry(Dictionary.java:99) at sun.jvm.hotspot.memory.Dictionary.find(Dictionary.java:87) at sun.jvm.hotspot.memory.SystemDictionary.find(SystemDictionary.java:159) at sun.jvm.hotspot.memory.SystemDictionary.find(SystemDictionary.java:151) at sun.jvm.hotspot.memory.SystemDictionary.getAbstractOwnableSynchronizerKlass(SystemDictionary.java:135) at sun.jvm.hotspot.runtime.ConcurrentLocksPrinter.fillLocks(ConcurrentLocksPrinter.java:66) at sun.jvm.hotspot.runtime.ConcurrentLocksPrinter.<init>(ConcurrentLocksPrinter.java:36) at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:71) at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45) at sun.jvm.hotspot.tools.JStack.run(JStack.java:66) at sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:260) at sun.jvm.hotspot.tools.Tool.start(Tool.java:223) at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118) at sun.jvm.hotspot.tools.JStack.main(JStack.java:92) ... 6 more [root@ctr-e134-1499953498516-209460-01-000004 atlas]#
Attaching the thread dump output as well.