[SLIDER-1082] Error validating container request

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Duplicate
• Component/s: appmaster
• Assignee: Unassigned
• Reporter: Billie Rinaldi
• Created: 03/Feb/16 16:58
• Updated: 03/Feb/16 17:13
• Resolved: 03/Feb/16 17:13

Description

I’m starting an app with one container and the default placement policy. When I kill the container (or the container dies OOM), a new one never comes up, and the AM log has the error below.

2016-02-03 08:18:56,083 [AmExecutor-006] INFO  state.AppState - Reviewing RoleStatus{name='UI', key=1, desired=1, actual=0, requested=0, releasing=0, failed=1, startFailed=0, started=1, completed=0, totalRequested=1, preempted=0, nodeFailed=0, failedRecently=1, limitsExceeded=1, resourceRequirements=<memory:1024, vCores:2>, isAntiAffinePlacement=false, failureMessage='Failure container_e3741_1454001598828_0117_01_000002 on host myhost (-104): http://myhost:19888/jobhistory/logs/myhost:45454/container_e3741_1454001598828_0117_01_000002/ctx/root', providerRole=ProviderRole{name='UI', id=1, placementPolicy=0, nodeFailureThreshold=3, placementTimeoutSeconds=30, labelExpression='null'}} : 
2016-02-03 08:18:56,083 [AmExecutor-006] INFO  state.AppState - UI: Asking for 1 more nodes(s) for a total of 1 
2016-02-03 08:18:56,085 [AmExecutor-006] INFO  state.OutstandingRequest - Submitting request for container on myhost
2016-02-03 08:18:56,086 [AmExecutor-006] ERROR actions.QueueExecutor - Exception processing org.apache.slider.server.appmaster.actions.ReviewAndFlexApplicationSize@c2085fb name='onContainersCompleted', delay=0, attrs=4, sequenceNumber=10}: org.apache.hadoop.yarn.client.api.InvalidContainerRequestException: relax location flag false doesn't match container location flag in priority false: OutstandingRequest{roleId=1, hostname='myhost', node=myhost, hasLocation=false, requestedTimeMillis=1454516336084, mayEscalate=true, escalated=false, escalationTimeoutMillis=1454516366084, issuedRequest=Capability[<memory:1024, vCores:2>]Priority[1]; relaxLocality=false; Nodes = [ myhost ]; }
org.apache.hadoop.yarn.client.api.InvalidContainerRequestException: relax location flag false doesn't match container location flag in priority false: OutstandingRequest{roleId=1, hostname='myhost', node=myhost, hasLocation=false, requestedTimeMillis=1454516336084, mayEscalate=true, escalated=false, escalationTimeoutMillis=1454516366084, issuedRequest=Capability[<memory:1024, vCores:2>]Priority[1]; relaxLocality=false; Nodes = [ myhost ]; }
	at org.apache.slider.server.appmaster.state.OutstandingRequest.validateContainerRequest(OutstandingRequest.java:421)
	at org.apache.slider.server.appmaster.state.OutstandingRequest.validate(OutstandingRequest.java:384)
	at org.apache.slider.server.appmaster.state.OutstandingRequest.buildContainerRequest(OutstandingRequest.java:272)
	at org.apache.slider.server.appmaster.state.RoleHistory.requestInstanceOnNode(RoleHistory.java:625)
	at org.apache.slider.server.appmaster.state.RoleHistory.requestContainerForRole(RoleHistory.java:644)
	at org.apache.slider.server.appmaster.state.AppState.createContainerRequest(AppState.java:1202)
	at org.apache.slider.server.appmaster.state.AppState.reviewOneRole(AppState.java:1992)
	at org.apache.slider.server.appmaster.state.AppState.reviewRequestAndReleaseNodes(AppState.java:1821)
	at org.apache.slider.server.appmaster.SliderAppMaster.executeNodeReview(SliderAppMaster.java:1919)
	at org.apache.slider.server.appmaster.SliderAppMaster.handleReviewAndFlexApplicationSize(SliderAppMaster.java:1905)
	at org.apache.slider.server.appmaster.actions.ReviewAndFlexApplicationSize.execute(ReviewAndFlexApplicationSize.java:41)
	at org.apache.slider.server.appmaster.actions.QueueExecutor.run(QueueExecutor.java:73)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
2016-02-03 08:18:56,089 [AmExecutor-006] ERROR appmaster.SliderAppMaster - Exception in AmExecutor-006: org.apache.hadoop.yarn.client.api.InvalidContainerRequestException: relax location flag false doesn't match container location flag in priority false: OutstandingRequest{roleId=1, hostname='myhost', node=myhost, hasLocation=false, requestedTimeMillis=1454516336084, mayEscalate=true, escalated=false, escalationTimeoutMillis=1454516366084, issuedRequest=Capability[<memory:1024, vCores:2>]Priority[1]; relaxLocality=false; Nodes = [ myhost ]; }
org.apache.hadoop.yarn.client.api.InvalidContainerRequestException: relax location flag false doesn't match container location flag in priority false: OutstandingRequest{roleId=1, hostname='myhost', node=myhost, hasLocation=false, requestedTimeMillis=1454516336084, mayEscalate=true, escalated=false, escalationTimeoutMillis=1454516366084, issuedRequest=Capability[<memory:1024, vCores:2>]Priority[1]; relaxLocality=false; Nodes = [ myhost ]; }
	at org.apache.slider.server.appmaster.state.OutstandingRequest.validateContainerRequest(OutstandingRequest.java:421)
	at org.apache.slider.server.appmaster.state.OutstandingRequest.validate(OutstandingRequest.java:384)
	at org.apache.slider.server.appmaster.state.OutstandingRequest.buildContainerRequest(OutstandingRequest.java:272)
	at org.apache.slider.server.appmaster.state.RoleHistory.requestInstanceOnNode(RoleHistory.java:625)
	at org.apache.slider.server.appmaster.state.RoleHistory.requestContainerForRole(RoleHistory.java:644)
	at org.apache.slider.server.appmaster.state.AppState.createContainerRequest(AppState.java:1202)
	at org.apache.slider.server.appmaster.state.AppState.reviewOneRole(AppState.java:1992)
	at org.apache.slider.server.appmaster.state.AppState.reviewRequestAndReleaseNodes(AppState.java:1821)
	at org.apache.slider.server.appmaster.SliderAppMaster.executeNodeReview(SliderAppMaster.java:1919)
	at org.apache.slider.server.appmaster.SliderAppMaster.handleReviewAndFlexApplicationSize(SliderAppMaster.java:1905)
	at org.apache.slider.server.appmaster.actions.ReviewAndFlexApplicationSize.execute(ReviewAndFlexApplicationSize.java:41)
	at org.apache.slider.server.appmaster.actions.QueueExecutor.run(QueueExecutor.java:73)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Issue Links

• duplicates: SLIDER-1051 labelled container request is being vetoed in validation process