[YARN-6259] Support pagination and optimize data transfer with zero-copy approach for containerlogs REST API in NMWebServices

• Type: Improvement
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.8.1
• Component/s: nodemanager
• Assignee: Tao Yang
• Reporter: Tao Yang
• Created: 01/Mar/17 11:10
• Updated: 25/Oct/19 20:25

Description

Currently containerlogs REST API in NMWebServices will read and send the entire content of container logs. Most of container logs are large and it's useful to support pagination.

• Add pagesize and pageindex parameters for containerlogs REST API

  URL: http://<nm_address>/ws/v1/node/containerlogs/<container_id>/<file_name>
  QueryParams:
    pagesize - max bytes of one page , default 1MB
    pageindex - index of required page, default 0, can be nagative(set -1 will get the last page content)
  

• Add containerlogs-info REST API since sometimes we need to know the totalSize/pageSize/pageCount info of log

  URL: http://<nm_address>/ws/v1/node/containerlogs-info/<container_id>/<file_name>
  QueryParams:
    pagesize - max bytes of one page , default 1MB
  Response example:
    {"logInfo":{"totalSize":2497280,"pageSize":1048576,"pageCount":3}}
  

Moreover, the data transfer pipeline (disk --> read buffer --> NM buffer --> socket buffer) can be optimized to pipeline(disk --> read buffer --> socket buffer) with zero-copy approach.