Details
-
Improvement
-
Status: Open
-
Not a Priority
-
Resolution: Unresolved
-
1.12.0
-
None
-
flink 1.4.0 +
-
Patch
Description
in TwoPhaseCommitSinkFunctionOutput Maybe A large number of EventData will be output(log.info),which will cause IO bottleneck and disk waste.
my code in the attachment, A large number event data output in the log output by flink , e.g:
Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b, ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ), transactionStartTime=1610426158532} from checkpoint 4
in TwoPhaseCommitSinkFunction about LOG.info code is as follows:
LOG.info(
"{} - checkpoint {} complete, committing transaction {} from checkpoint {}",
name(),
checkpointId,
pendingTransaction,
pendingTransactionCheckpointId);
will be invoke pendingTransaction'toString method (pendingTransaction is TransactionHolder'instance)
TransactionHolder'toString method code is:
@Override public String toString() { return "TransactionHolder{" + "handle=" + handle + ", transactionStartTime=" + transactionStartTime + '}'; }
handle is the concrete realization of my Transaction! There is a parameter of List type in my Transaction, which is used to receive data. as a result, these data are printed out(log.info)