Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Invalid
-
None
-
None
Description
The StreamExecution maintains a write-ahead log of batch metadata in order to allow repeating previously in-flight batches if the driver is restarted. StreamExecution does not garbage-collect or compact this log in any way.
Since the log is implemented with HDFSMetadataLog, these files will consume memory on the HDFS NameNode. Specifically, each log file will consume about 300 bytes of NameNode memory (150 bytes for the inode and 150 bytes for the block of file contents; see https://www.cloudera.com/documentation/enterprise/latest/topics/admin_nn_memory_config.html. An application with a 100 msec batch interval will increase the NameNode's heap usage by about 250MB per day.
There is also the matter of recovery. StreamExecution reads its entire log when restarting. This read operation will be very expensive if the log contains millions of entries spread over millions of files.