[ACCUMULO-4332] Tserver exiting due to uncaught error

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 1.6.5
• Assignee: Unassigned
• Reporter: Michael Wall
• Created: 07/Jun/16 14:04
• Updated: 19/Mar/18 18:08
• Resolved: 19/Mar/18 18:08

Description

After the following log statements, the tserver became a zombie and did nothing but spit GC and Cache log statements. The fix was to add more open files, ulimit -n was 1024. This ticket is to see if we can catch this exception and kill the tserver.

ERROR: run() exiting due to uncaught error
java.lang.OutOfMemoryError: unable to create new native thread
  at java.lang.Thread.start0(Native Method)
  at java.lang.Thread.start(Thread.java:714)
  at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
  at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1360)
  at org.apache.thrift.server.THsHaServer.requestInvoke(THsHaServer.java:157)
  at org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleRead(AbstractNonblockingServer.java:210)
  at org.apache.accumulo.server.util.CustomNonBlockerServer$SelectAcceptThread.select(CustomNonBlockingServer.java:227)
  at org.apache.accumulo.server.util.CustomNonBlockerServer$SelectAcceptThread.run(CustomNonBlockingServer.java:183)

Issue Links

• relates to: ACCUMULO-1708 Error during minor compaction left tserver in bad state