[YARN-11208] SampleContainerLogAggregationPolicy is inconsistent with expectations

• Type: Improvement
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.2.2
• Component/s: log-aggregation, nodemanager
• Assignee: Unassigned
• Reporter: chuanjie.duan
• Created: 07/Jul/22 11:41
• Updated: 07/Jul/22 11:53

Description

repro steps:

Configure the following parameters in yarn-site.xml

  <property>
    <name>yarn.nodemanager.log-aggregation.policy.class</name>
 <value>org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.SampleContainerLogAggregationPolicy</value>
  </property>
  <property>
    <name>yarn.nodemanager.log-aggregation.policy.parameters</name>
    <value>SR:0.3,MIN:20</value>
  </property>

 

run pi:

hadoop  org.apache.hadoop.examples.QuasiMonteCarlo 100 5

 

three nodemanagers in my cluster

 

Expected quantity：20+(100-20)*0.3+1=45 container logs uploaded

Actual quantity: 21 container logs uploaded

 

So why are there fewer logs uploaded?

 

I print debug log in SampleContainerLogAggregationPolicy.

shouldDoLogAggregation()

 

logContext.getContainerId().hashCode():-1455756646

logContext.getContainerId().hashCode() % (1/sampleRate):-0.041277885

 

So in most cases, it will not be uploaded

 

Issue Links

• relates to: YARN-221 NM should provide a way for AM to tell it not to aggregate logs.