[MESOS-2161] AbstractState JNI check fails for Marathon framework

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.21.0
• Fix Version/s: 0.22.1, 0.23.0

Environment

Mesos 0.21.0
Marathon 0.7.5
Fedora 20

• Target Version/s: 0.22.1
• Sprint: Mesosphere Q1 Sprint 5 - 3/20, Mesosphere Q1 Sprint 6 - 4/3
• Assignee: Joris Van Remoortere
• Reporter: Matthew Sanders
• Shepherd: Benjamin Hindman
• Created: 26/Nov/14 00:36
• Updated: 02/Jul/15 04:20
• Resolved: 29/Mar/15 20:08

Description

I've recently upgraded to mesos 0.21.0 and now it seems that every few minutes or so I see the following error, which kills marathon.

Nov 25 18:12:42 gianthornet.trading.imc.intra marathon[5453]: [2014-11-25 18:12:42,064] INFO 10.133.128.26 - - [26/Nov/2014:00:12:41 +0000] "GET /v2/apps HTTP/1.1" 200 2321 "http://marathon:8080/" "Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Firefox/31.0" (mesosphere.chaos.http.ChaosRequestLog:15)
Nov 25 18:12:42 gianthornet.trading.imc.intra marathon[5453]: [2014-11-25 18:12:42,238] INFO 10.133.128.26 - - [26/Nov/2014:00:12:42 +0000] "GET /v2/deployments HTTP/1.1" 200 2 "http://marathon:8080/" "Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Firefox/31.0" (mesosphere.chaos.http.ChaosRequestLog:15)
Nov 25 18:12:42 gianthornet.trading.imc.intra marathon[5453]: [2014-11-25 18:12:42,961] INFO 10.192.221.95 - - [26/Nov/2014:00:12:42 +0000] "GET /v2/apps HTTP/1.1" 200 2321 "http://marathon:8080/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.65 Safari/537...
Nov 25 18:12:43 gianthornet.trading.imc.intra marathon[5453]: [2014-11-25 18:12:43,032] INFO 10.192.221.95 - - [26/Nov/2014:00:12:42 +0000] "GET /v2/deployments HTTP/1.1" 200 2 "http://marathon:8080/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.65 Safari...
Nov 25 18:12:44 gianthornet.trading.imc.intra marathon[5454]: F1125 18:12:44.146260 5897 check.hpp:79] Check failed: f.isReady()
Nov 25 18:12:44 gianthornet.trading.imc.intra marathon[5454]: *** Check failure stack trace: ***
Nov 25 18:12:44 gianthornet.trading.imc.intra marathon[5454]: @ 0x7f8176a2b17c google::LogMessage::Fail()
Nov 25 18:12:44 gianthornet.trading.imc.intra marathon[5454]: @ 0x7f8176a2b0d5 google::LogMessage::SendToLog()
Nov 25 18:12:44 gianthornet.trading.imc.intra marathon[5454]: @ 0x7f8176a2aab3 google::LogMessage::Flush()
Nov 25 18:12:44 gianthornet.trading.imc.intra marathon[5454]: @ 0x7f8176a2da3b google::LogMessageFatal::~LogMessageFatal()
Nov 25 18:12:44 gianthornet.trading.imc.intra marathon[5454]: @ 0x7f8176a1ea64 _checkReady<>()
Nov 25 18:12:44 gianthornet.trading.imc.intra marathon[5454]: @ 0x7f8176a1d43b Java_org_apache_mesos_state_AbstractState__1_1names_1get
Nov 25 18:12:44 gianthornet.trading.imc.intra marathon[5454]: @ 0x7f81f644ca70 (unknown)
Nov 25 18:12:44 gianthornet.trading.imc.intra systemd[1]: marathon.service: main process exited, code=killed, status=6/ABRT

Here's the command that mesos-master is being run with
/usr/local/sbin/mesos-master --zk=zk://usint-zk-d01-node1chi:2191,usint-zk-d01-node2chi:2192,usint-zk-d01-node3chi:2193/mesos --port=5050 --log_dir=/var/log/mesos --quorum=1 --work_dir=/var/lib/mesos

Here's the command that the slave is running with:
/usr/local/sbin/mesos-slave --master=zk://usint-zk-d01-node1chi:2191,usint-zk-d01-node2chi:2192,usint-zk-d01-node3chi:2193/mesos --log_dir=/var/log/mesos --containerizers=docker,mesos --executor_registration_timeout=5mins --attributes=country:us;datacenter:njl3;environment:dev;region:amer;timezone:America/Chicago

I realize this could also be filed to marathon, but it sort of looks like a c++ issue to me, which is why I came here to post this. Any help would be greatly appreciated.

Issue Links

• blocks: MESOS-2579 0.22.1 release
• is duplicated by: MESOS-2531 Libmesos terminates JVM
• is related to: MESOS-1795 Assertion failure in state abstraction crashes JVM