do the long running applications such as HBase on YARN using Slider need to do anything to make sure that partial logs are uploaded?
Sumit Mohanty Sorry for the late reply. Yes, we need to change some configurations/setting for ApplicationSubmissionContext.
Here is a scenario which can explain the purpose of this ticket:
In MapReduce, we will create stdout, stderr, and syslog for every containers. And since the MapReduce job is relatively short (compared with the long running applications), it does not make sense to upload those logs partially unless the users really want to.
So, the old include_pattern/exclude_pattern in ASC will be used to indicate which log files need to be aggregated explicitly at app finish.
and we introduce two additional parameter is ASC which is more related to long running applications, such as HBase on YARN.
If we want the logs be uploaded (partial logs) while the app is running, we should use these two newly instroduced parameters.
For the HBase on YARN using Slider case, after the patch, we need to switch the values from old include_pattern/exclude_pattern to new rolled_logs_include_pattern/rolled_logs_exclude_pattern