[AURORA-1779] BatchWorker fails with PersistenceException

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.16.0
• Assignee: Maxim Khutornenko
• Reporter: Stephan Erb
• Created: 21/Sep/16 21:35
• Updated: 03/Nov/16 16:17
• Resolved: 22/Sep/16 17:54

Description

Steps to reproduce (if you are lucky):

vagrant destroy
vagrant up
vagrant ssh
cd /vagrant
./build-support/release/verify-release-candidate 0.16.0-rc1

For me, this resulted in the following initial stack trace (and a couple different follow ups due to a corrupted DB state):

I0921 19:45:38.730 [TaskEventBatchWorker, JobUpdateControllerImpl:359] Forwarding task change for vagrant/test/http_example/0 
I0921 19:45:38.744 [qtp1742582311-39, Slf4jRequestLog:60] 127.0.0.1 - - [21/Sep/2016:19:45:38 +0000] "POST //aurora.local/api HTTP/1.1" 200 108  
I0921 19:45:38.771 [TaskEventBatchWorker, JobUpdateControllerImpl:587] IJobUpdateKey{job=IJobKey{role=vagrant, environment=test, name=http_example}, id=97ff31cc-c17f-45ef-a1d9-c16cb07388c7} evaluation result: EvaluationResult{status=WORKING, sideEffects={0=SideEffect{action=Optional.of(ADD_TASK), statusChanges=[]}}} 
I0921 19:45:38.786 [TaskEventBatchWorker, JobUpdateControllerImpl:654] Executing side-effects for update of IJobUpdateKey{job=IJobKey{role=vagrant, environment=test, name=http_example}, id=97ff31cc-c17f-45ef-a1d9-c16cb07388c7}: {0=SideEffect{action=Optional.of(ADD_TASK), statusChanges=[]}} 
I0921 19:45:38.789 [TaskEventBatchWorker, InstanceActionHandler$AddTask:95] Adding instance IInstanceKey{jobKey=IJobKey{role=vagrant, environment=test, name=http_example}, instanceId=0} while ROLLING_FORWARD 
I0921 19:45:38.805 [TaskEventBatchWorker, StateMachine$Builder:389] vagrant-test-http_example-0-100b76ee-ed70-4e15-8e52-046f992d0b4a state machine transition INIT -> PENDING 
I0921 19:45:38.806 [TaskEventBatchWorker, TaskStateMachine:474] Adding work command SAVE_STATE for vagrant-test-http_example-0-100b76ee-ed70-4e15-8e52-046f992d0b4a 
I0921 19:45:38.808 [ShutdownHook, SchedulerMain:101] Stopping scheduler services. 
I0921 19:45:38.822 [TimeSeriesRepositoryImpl STOPPING, TimeSeriesRepositoryImpl:168] Variable sampler shut down 
I0921 19:45:38.839 [TearDownShutdownRegistry STOPPING, ShutdownRegistry$ShutdownRegistryImpl:77] Executing 4 shutdown commands. 
E0921 19:45:38.845 [TaskEventBatchWorker, BatchWorker:217] TaskEventBatchWorker: Failed to process batch item. Error: {} org.apache.ibatis.exceptions.PersistenceException: 
### Error querying database.  Cause: org.h2.jdbc.JdbcSQLException: Database is already closed (to disable automatic closing at VM shutdown, add ";DB_CLOSE_ON_EXIT=FALSE" to the db URL) [90121-190]
### The error may exist in org/apache/aurora/scheduler/storage/db/TaskMapper.xml
### The error may involve org.apache.aurora.scheduler.storage.db.TaskMapper.selectById
### The error occurred while executing a query
### SQL: SELECT       t.id AS row_id,       t.task_config_row_id AS task_config_row_id,       t.task_id AS task_id,       t.instance_id AS instance_id,       t.status AS status,       t.failure_count AS failure_count,       t.ancestor_task_id AS ancestor_id,       j.role AS c_j_role,       j.environment AS c_j_environment,       j.name AS c_j_name,       h.slave_id AS slave_id,       h.host AS slave_host,       tp.id as tp_id,       tp.name as tp_name,       tp.port as tp_port,       te.id as te_id,       te.timestamp_ms as te_timestamp,       te.status as te_status,       te.message as te_message,       te.scheduler_host as te_scheduler     FROM tasks AS t     INNER JOIN task_configs as c ON c.id = t.task_config_row_id     INNER JOIN job_keys AS j ON j.id = c.job_key_id     LEFT OUTER JOIN task_ports as tp ON tp.task_row_id = t.id     LEFT OUTER JOIN task_events as te ON te.task_row_id = t.id     LEFT OUTER JOIN host_attributes AS h ON h.id = t.slave_row_id         WHERE       t.task_id = ?

Issue Links

• is related to: AURORA-1771 Consider scheduling multiple tasks per scheduling round