[AURORA-769] No-op job update fails with "Missing both initial and desired states. At least one is required."

• Type: Story
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.6.0
• Component/s: Scheduler
• Assignee: Bill Farner
• Reporter: Bill Farner
• Created: 30/Sep/14 00:38
• Updated: 28/Oct/14 22:37
• Resolved: 30/Sep/14 22:09

Description

This is pretty easy to reproduce - do a server-driven job update, then do it again.

This should result in no action by the scheduler, and console output including:

Job is unchanged by proposed update

Instead you get:

log(info): Message from scheduler: Missing both initial and desired states. At least one is required.

Relevant stack trace:

W0930 00:32:24.335 THREAD120 org.apache.aurora.scheduler.thrift.aop.LoggingInterceptor.invoke: Uncaught exception while handling startJobUpdate(JobUpdateRequest(taskConfig:TaskConfig(owner:Identity(role:www-data, user:vagrant), environment:devel, jobName:hello_world, isService:true, numCpus:1.0, ramMb:1, diskMb:8, priority:0, maxTaskFailures:1, production:false, constraints:[], requestedPorts:[], taskLinks:{}, executorConfig:ExecutorConfig(name:AuroraExecutor, data:{"priority": 0, "health_check_config": {"initial_interval_secs": 15.0, "interval_secs": 10.0, "timeout_secs": 1.0, "max_consecutive_failures": 0}, "name": "hello_world", "environment": "devel", "max_task_failures": 1, "task": {"processes": [{"daemon": false, "name": "fetch_package", "ephemeral": false, "max_failures": 1, "min_duration": 5, "cmdline": "cp /vagrant/hello_world.py . && echo f0cc21c10c2b91974b0cac2629bbee6c && chmod +x hello_world.py", "final": false}, {"daemon": false, "name": "hello_world", "ephemeral": false, "max_failures": 1, "min_duration": 5, "cmdline": "python hello_world.py", "final": false}], "name": "fetch_package", "finalization_wait": 30, "max_failures": 1, "max_concurrency": 0, "resources": {"disk": 8388608, "ram": 1048576, "cpu": 1.0}, "constraints": [{"order": ["fetch_package", "hello_world"]}]}, "enable_hooks": false, "cluster": "devcluster", "production": false, "role": "www-data"}), metadata:[]), instanceCount:1, settings:JobUpdateSettings(updateGroupSize:1, maxPerInstanceFailures:0, maxFailedInstances:0, maxWaitToInstanceRunningMs:60000, minWaitInInstanceRunningMs:45000, rollbackOnFailure:true, updateOnlyTheseInstances:null, waitForBatchCompletion:false)), SessionKey(mechanism:UNAUTHENTICATED, data:50 D0 14 4C 71 0D 4C 80 80 4C 40))
java.lang.IllegalArgumentException: Missing both initial and desired states. At least one is required.
        at org.apache.aurora.scheduler.storage.db.DBJobUpdateStore.saveJobUpdate(DBJobUpdateStore.java:73)
        at com.twitter.common.inject.TimedInterceptor.invoke(TimedInterceptor.java:87)
        at org.apache.aurora.scheduler.storage.log.WriteAheadStorage.saveJobUpdate(WriteAheadStorage.java:287)
        at org.apache.aurora.scheduler.updater.JobUpdateControllerImpl$1.execute(JobUpdateControllerImpl.java:153)
        at org.apache.aurora.scheduler.storage.Storage$MutateWork$NoResult.apply(Storage.java:132)
        at org.apache.aurora.scheduler.storage.Storage$MutateWork$NoResult.apply(Storage.java:128)
        at org.apache.aurora.scheduler.storage.log.LogStorage.write(LogStorage.java:551)
        at org.apache.aurora.scheduler.storage.CallOrderEnforcingStorage.write(CallOrderEnforcingStorage.java:130)
        at org.apache.aurora.scheduler.updater.JobUpdateControllerImpl.start(JobUpdateControllerImpl.java:131)
        at org.apache.aurora.scheduler.thrift.SchedulerThriftInterface$14.apply(SchedulerThriftInterface.java:1456)
        at org.apache.aurora.scheduler.thrift.SchedulerThriftInterface$14.apply(SchedulerThriftInterface.java:1416)
        at org.apache.aurora.scheduler.storage.log.LogStorage$7.apply(LogStorage.java:559)
        at org.apache.aurora.scheduler.storage.log.LogStorage$7.apply(LogStorage.java:556)
        at org.apache.aurora.scheduler.storage.mem.MemStorage.doWork(MemStorage.java:175)
        at org.apache.aurora.scheduler.storage.mem.MemStorage.access$400(MemStorage.java:59)
        at org.apache.aurora.scheduler.storage.mem.MemStorage$4.apply(MemStorage.java:200)
        at org.apache.aurora.scheduler.storage.mem.MemStorage$4.apply(MemStorage.java:197)
        at org.apache.aurora.scheduler.storage.db.DbStorage.write(DbStorage.java:150)
        at org.mybatis.guice.transactional.TransactionalMethodInterceptor.invoke(TransactionalMethodInterceptor.java:101)
        at org.apache.aurora.scheduler.storage.mem.MemStorage.write(MemStorage.java:197)
        at org.apache.aurora.scheduler.storage.log.LogStorage.write(LogStorage.java:556)
        at org.apache.aurora.scheduler.storage.CallOrderEnforcingStorage.write(CallOrderEnforcingStorage.java:130)
        at org.apache.aurora.scheduler.thrift.SchedulerThriftInterface.startJobUpdate(SchedulerThriftInterface.java:1416)

Issue Links

• duplicates: AURORA-759 Scheduler updater stuck in ROLLING_BACK