[ZEPPELIN-1828] Flaky Test: RemoteInterpreterTest hanged in zeppelin-interpreter - ASF JIRA

Details

Type: Sub-task
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 0.8.0
Component/s: None
Labels:
None

Description

I saw this kind of hanged tests multiple times

raw log1: https://s3.amazonaws.com/archive.travis-ci.org/jobs/184305306/log.txt
build profile1: https://travis-ci.org/apache/zeppelin/jobs/184305306/config
raw log2: https://api.travis-ci.org/jobs/183941225/log.txt?deansi=true
build profile2: https://travis-ci.org/apache/zeppelin/jobs/183941225/config
raw log3: https://s3.amazonaws.com/archive.travis-ci.org/jobs/184470869/log.txt
build profile3: https://travis-ci.org/apache/zeppelin/jobs/184470869/config

Running org.apache.zeppelin.interpreter.remote.RemoteInterpreterTest
17:31:24,423  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:120 - Run interpreter process [/home/travis/build/apache/zeppelin/zeppelin-interpreter/../bin/interpreter.sh, -d, fake, -p, 43625, -l, fakeRepo]
17:31:24,926  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreter:238 - Create remote interpreter org.apache.zeppelin.interpreter.remote.mock.MockInterpreterA
17:31:25,090  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreter:238 - Create remote interpreter org.apache.zeppelin.interpreter.remote.mock.MockInterpreterB
17:31:25,124 ERROR org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller:245 - Can't handle event RemoteInterpreterEvent(type:OUTPUT_UPDATE_ALL, data:{"noteId":"note","paragraphId":"id","messages":[]})
java.lang.NullPointerException
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:186)
17:31:25,162  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:121 - shutdown interpreter process
17:31:26,126 ERROR org.apache.zeppelin.interpreter.remote.AppendOutputRunner:68 - Wait for OutputBuffer queue interrupted: null
17:31:27,274  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:131 - Exception in RemoteInterpreterProcess while synchronized dereference, can safely ignore exception while client.shutdown() may terminates remote process
17:31:27,276  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:164 - Interpreter process exited 0
17:31:27,776  INFO org.apache.zeppelin.interpreter.InterpreterGroup:145 - Close interpreter group InterpreterGroup_1481823084409_-1062367753
17:31:27,778  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:120 - Run interpreter process [/home/travis/build/apache/zeppelin/zeppelin-interpreter/../bin/interpreter.sh, -d, fake, -p, 37763, -l, fakeRepo]
17:31:28,281  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreter:238 - Create remote interpreter org.apache.zeppelin.interpreter.remote.mock.MockInterpreterA
17:31:28,448  INFO org.apache.zeppelin.scheduler.SchedulerFactory:131 - Job 0 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote220694569
17:31:28,480 ERROR org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller:245 - Can't handle event RemoteInterpreterEvent(type:OUTPUT_UPDATE_ALL, data:{"noteId":"note","paragraphId":"0","messages":[]})
java.lang.NullPointerException
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:186)
17:31:28,520  INFO org.apache.zeppelin.scheduler.SchedulerFactory:137 - Job 0 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote220694569
17:31:28,521  INFO org.apache.zeppelin.scheduler.SchedulerFactory:131 - Job 1 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote220694569
17:31:28,523 ERROR org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller:245 - Can't handle event RemoteInterpreterEvent(type:OUTPUT_UPDATE_ALL, data:{"noteId":"note","paragraphId":"1","messages":[]})
java.lang.NullPointerException
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:186)
17:31:28,525  INFO org.apache.zeppelin.scheduler.SchedulerFactory:137 - Job 1 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote220694569
17:31:28,526  INFO org.apache.zeppelin.scheduler.SchedulerFactory:131 - Job 2 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote220694569
17:31:28,527 ERROR org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller:245 - Can't handle event RemoteInterpreterEvent(type:OUTPUT_UPDATE_ALL, data:{"noteId":"note","paragraphId":"2","messages":[]})
java.lang.NullPointerException
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:186)
17:31:28,530  INFO org.apache.zeppelin.scheduler.SchedulerFactory:137 - Job 2 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote220694569
17:31:28,535  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:121 - shutdown interpreter process
17:31:29,528 ERROR org.apache.zeppelin.interpreter.remote.AppendOutputRunner:68 - Wait for OutputBuffer queue interrupted: null
17:31:30,961  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:131 - Exception in RemoteInterpreterProcess while synchronized dereference, can safely ignore exception while client.shutdown() may terminates remote process
17:31:30,963  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:164 - Interpreter process exited 0
17:31:31,463  INFO org.apache.zeppelin.interpreter.InterpreterGroup:145 - Close interpreter group InterpreterGroup_1481823087778_1506341451
17:31:31,464  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:120 - Run interpreter process [/home/travis/build/apache/zeppelin/zeppelin-interpreter/../bin/interpreter.sh, -d, fake, -p, 45441, -l, fakeRepo]
17:31:31,967  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreter:238 - Create remote interpreter org.apache.zeppelin.interpreter.remote.mock.MockInterpreterA
17:31:32,131  INFO org.apache.zeppelin.scheduler.SchedulerFactory:131 - Job 0 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote959023117
17:31:32,165 ERROR org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller:245 - Can't handle event RemoteInterpreterEvent(type:OUTPUT_UPDATE_ALL, data:{"noteId":"note","paragraphId":"0","messages":[]})
java.lang.NullPointerException
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:186)
17:31:32,631  INFO org.apache.zeppelin.scheduler.SchedulerFactory:131 - Job 1 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote959023117
17:31:32,634 ERROR org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller:245 - Can't handle event RemoteInterpreterEvent(type:OUTPUT_UPDATE_ALL, data:{"noteId":"note","paragraphId":"1","messages":[]})
java.lang.NullPointerException
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:186)
17:31:33,132  INFO org.apache.zeppelin.scheduler.SchedulerFactory:131 - Job 2 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote959023117
17:31:33,134 ERROR org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller:245 - Can't handle event RemoteInterpreterEvent(type:OUTPUT_UPDATE_ALL, data:{"noteId":"note","paragraphId":"2","messages":[]})
java.lang.NullPointerException
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:186)
17:31:33,200  INFO org.apache.zeppelin.scheduler.SchedulerFactory:137 - Job 0 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote959023117
17:31:33,636  INFO org.apache.zeppelin.scheduler.SchedulerFactory:137 - Job 1 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote959023117
17:31:33,637  INFO org.apache.zeppelin.scheduler.SchedulerFactory:131 - Job 3 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote959023117
17:31:33,639 ERROR org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller:245 - Can't handle event RemoteInterpreterEvent(type:OUTPUT_UPDATE_ALL, data:{"noteId":"note","paragraphId":"3","messages":[]})
java.lang.NullPointerException
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:186)
17:31:34,136  INFO org.apache.zeppelin.scheduler.SchedulerFactory:137 - Job 2 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote959023117
17:31:34,641  INFO org.apache.zeppelin.scheduler.SchedulerFactory:137 - Job 3 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote959023117
17:31:34,646  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:121 - shutdown interpreter process
17:31:35,641 ERROR org.apache.zeppelin.interpreter.remote.AppendOutputRunner:68 - Wait for OutputBuffer queue interrupted: null
17:31:37,072  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:131 - Exception in RemoteInterpreterProcess while synchronized dereference, can safely ignore exception while client.shutdown() may terminates remote process
17:31:37,074  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:164 - Interpreter process exited 0
17:31:37,574  INFO org.apache.zeppelin.interpreter.InterpreterGroup:145 - Close interpreter group InterpreterGroup_1481823091464_1066082224
17:31:37,576  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:120 - Run interpreter process [/home/travis/build/apache/zeppelin/zeppelin-interpreter/../bin/interpreter.sh, -d, fake, -p, 37531, -l, fakeRepo]
17:31:38,080  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreter:238 - Create remote interpreter org.apache.zeppelin.interpreter.remote.mock.MockInterpreterA
17:31:38,249  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreter:238 - Create remote interpreter org.apache.zeppelin.interpreter.remote.mock.MockInterpreterB
17:31:38,253  INFO org.apache.zeppelin.scheduler.SchedulerFactory:131 - Job jobA started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote582688427
17:31:38,288 ERROR org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller:245 - Can't handle event RemoteInterpreterEvent(type:OUTPUT_UPDATE_ALL, data:{"noteId":"note","paragraphId":"jobA","messages":[]})
java.lang.NullPointerException
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:186)
17:31:38,821  INFO org.apache.zeppelin.scheduler.SchedulerFactory:137 - Job jobA finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote582688427
17:31:38,822  INFO org.apache.zeppelin.scheduler.SchedulerFactory:131 - Job jobB started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote582688427
17:31:38,826 ERROR org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller:245 - Can't handle event RemoteInterpreterEvent(type:OUTPUT_UPDATE_ALL, data:{"noteId":"note","paragraphId":"jobB","messages":[]})
java.lang.NullPointerException
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:186)
17:31:39,828  INFO org.apache.zeppelin.scheduler.SchedulerFactory:137 - Job jobB finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote582688427
17:31:39,861  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:121 - shutdown interpreter process
17:31:40,828 ERROR org.apache.zeppelin.interpreter.remote.AppendOutputRunner:68 - Wait for OutputBuffer queue interrupted: null
17:31:42,289  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:131 - Exception in RemoteInterpreterProcess while synchronized dereference, can safely ignore exception while client.shutdown() may terminates remote process
17:31:42,290  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:164 - Interpreter process exited 0
17:31:42,790  INFO org.apache.zeppelin.interpreter.InterpreterGroup:145 - Close interpreter group InterpreterGroup_1481823097576_2123324288
17:31:42,792  INFO org.apache.zeppelin.interpreter.InterpreterGroup:145 - Close interpreter group InterpreterGroup_1481823102792_1551732183
17:31:42,793  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:120 - Run interpreter process [/home/travis/build/apache/zeppelin/zeppelin-interpreter/../bin/interpreter.sh, -d, fake, -p, 43603, -l, fakeRepo]
17:31:43,296  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreter:238 - Create remote interpreter org.apache.zeppelin.interpreter.remote.mock.MockInterpreterA
17:31:43,497 ERROR org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller:245 - Can't handle event RemoteInterpreterEvent(type:OUTPUT_UPDATE_ALL, data:{"noteId":"noteId","paragraphId":"id","messages":[]})
java.lang.NullPointerException
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:186)
17:31:43,545  INFO org.apache.zeppelin.interpreter.InterpreterGroup:145 - Close interpreter group InterpreterGroup_1481823102793_1859137179
17:31:43,552  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:121 - shutdown interpreter process
17:31:44,498 ERROR org.apache.zeppelin.interpreter.remote.AppendOutputRunner:68 - Wait for OutputBuffer queue interrupted: null
17:31:45,664  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:131 - Exception in RemoteInterpreterProcess while synchronized dereference, can safely ignore exception while client.shutdown() may terminates remote process
17:31:45,665  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:164 - Interpreter process exited 0
17:31:46,267  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreter:532 - Push local angular object registry from ZeppelinServer to remote interpreter group groupId
17:31:46,269  INFO org.apache.zeppelin.interpreter.InterpreterGroup:145 - Close interpreter group InterpreterGroup_1481823106166_-1057400618
17:31:46,271  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:120 - Run interpreter process [/home/travis/build/apache/zeppelin/zeppelin-interpreter/../bin/interpreter.sh, -d, fake, -p, 39299, -l, fakeRepo]
17:31:46,774  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreter:238 - Create remote interpreter org.apache.zeppelin.interpreter.remote.mock.MockInterpreterA
17:31:46,943  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreter:238 - Create remote interpreter org.apache.zeppelin.interpreter.remote.mock.MockInterpreterB
17:31:46,977 ERROR org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller:245 - Can't handle event RemoteInterpreterEvent(type:OUTPUT_UPDATE_ALL, data:{"noteId":"note","paragraphId":"id","messages":[]})
java.lang.NullPointerException
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:186)
17:31:47,512 ERROR org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller:245 - Can't handle event RemoteInterpreterEvent(type:OUTPUT_UPDATE_ALL, data:{"noteId":"note","paragraphId":"id","messages":[]})
java.lang.NullPointerException
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:186)
17:31:48,519  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:121 - shutdown interpreter process
17:31:49,514 ERROR org.apache.zeppelin.interpreter.remote.AppendOutputRunner:68 - Wait for OutputBuffer queue interrupted: null
17:31:50,632  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:131 - Exception in RemoteInterpreterProcess while synchronized dereference, can safely ignore exception while client.shutdown() may terminates remote process
17:31:50,633  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:164 - Interpreter process exited 0
17:31:51,133  INFO org.apache.zeppelin.interpreter.InterpreterGroup:145 - Close interpreter group InterpreterGroup_1481823106270_2108336050
17:31:51,136  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:120 - Run interpreter process [/home/travis/build/apache/zeppelin/zeppelin-interpreter/../bin/interpreter.sh, -d, fake, -p, 34103, -l, fakeRepo]
17:31:51,638  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreter:238 - Create remote interpreter org.apache.zeppelin.interpreter.remote.mock.MockInterpreterA
17:31:51,803  INFO org.apache.zeppelin.scheduler.SchedulerFactory:131 - Job jobA started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote1184497162
17:31:51,837 ERROR org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller:245 - Can't handle event RemoteInterpreterEvent(type:OUTPUT_UPDATE_ALL, data:{"noteId":"note","paragraphId":"jobA","messages":[]})
java.lang.NullPointerException
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:186)
17:31:51,911  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:121 - shutdown interpreter process
17:31:52,838 ERROR org.apache.zeppelin.interpreter.remote.AppendOutputRunner:68 - Wait for OutputBuffer queue interrupted: null
17:31:54,339  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:131 - Exception in RemoteInterpreterProcess while synchronized dereference, can safely ignore exception while client.shutdown() may terminates remote process
17:31:54,341  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:164 - Interpreter process exited 0
17:31:54,841  INFO org.apache.zeppelin.scheduler.SchedulerFactory:137 - Job jobA finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreternote1184497162
17:31:54,841  INFO org.apache.zeppelin.interpreter.InterpreterGroup:145 - Close interpreter group InterpreterGroup_1481823111136_509381734
17:31:54,843  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:120 - Run interpreter process [/home/travis/build/apache/zeppelin/zeppelin-interpreter/../bin/interpreter.sh, -d, fake, -p, 42097, -l, fakeRepo]
17:31:55,345  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreter:238 - Create remote interpreter org.apache.zeppelin.interpreter.remote.mock.MockInterpreterA
17:31:55,520  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreter:238 - Create remote interpreter org.apache.zeppelin.interpreter.remote.mock.MockInterpreterB
17:31:55,522  INFO org.apache.zeppelin.interpreter.InterpreterGroup:145 - Close interpreter group InterpreterGroup_1481823114842_-936486082
17:31:55,529  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:121 - shutdown interpreter process
17:31:56,373 ERROR org.apache.zeppelin.interpreter.remote.AppendOutputRunner:68 - Wait for OutputBuffer queue interrupted: null
17:31:57,954  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:131 - Exception in RemoteInterpreterProcess while synchronized dereference, can safely ignore exception while client.shutdown() may terminates remote process
17:31:57,955  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:164 - Interpreter process exited 0
17:31:58,464  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:120 - Run interpreter process [/home/travis/build/apache/zeppelin/zeppelin-interpreter/../bin/interpreter.sh, -d, fake, -p, 35227, -l, fakeRepo]
17:31:58,966  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreter:238 - Create remote interpreter org.apache.zeppelin.interpreter.remote.mock.MockInterpreterEnv
17:31:59,223  INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess:121 - shutdown interpreter process
17:31:59,419 ERROR org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller:245 - Can't handle event RemoteInterpreterEvent(type:OUTPUT_UPDATE_ALL, data:{"noteId":"noteId","paragraphId":"id","messages":[]})
java.lang.NullPointerException
	at org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:186)
17:31:59,420 ERROR org.apache.zeppelin.interpreter.remote.AppendOutputRunner:68 - Wait for OutputBuffer queue interrupted: null


No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received

The build has been terminated