[FLINK-20972] TwoPhaseCommitSinkFunction Output a large amount of EventData

• Type: Improvement
• Status: Open
• Priority: Not a Priority
• Resolution: Unresolved
• Affects Version/s: 1.12.0
• Component/s: API / DataStream
• Labels: auto-deprioritized-minor easyfix pull-request-available

Environment

flink 1.4.0 +

• Flags: Patch
• Assignee: Unassigned
• Reporter: Huajie Wang
• Created: 14/Jan/21 10:16
• Updated: 03/Dec/21 22:38

Description

in TwoPhaseCommitSinkFunctionOutput Maybe A large number of EventData will be output(log.info),which will cause IO bottleneck and disk waste.

 
my code in the attachment, A large number event data output in the log output by flink , e.g: 

Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b, ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ), transactionStartTime=1610426158532} from checkpoint 4

in TwoPhaseCommitSinkFunction about LOG.info code is as follows:

LOG.info(
        "{} - checkpoint {} complete, committing transaction {} from checkpoint {}",
        name(),
        checkpointId,
        pendingTransaction,
        pendingTransactionCheckpointId); 

will be invoke pendingTransaction'toString method (pendingTransaction is TransactionHolder'instance) 

TransactionHolder'toString method code is:

@Override
public String toString() {
    return "TransactionHolder{"
            + "handle="
            +  handle
            + ", transactionStartTime="
            + transactionStartTime
            + '}';
}

 handle is the concrete realization of my Transaction! There is a parameter of List type in my Transaction, which is used to receive data. as a result, these data are printed out(log.info)
 
 

 

Issue Links

• links to: GitHub Pull Request #14646
• links to: GitHub Pull Request #14654
• links to: GitHub Pull Request #14657