Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-20972

TwoPhaseCommitSinkFunction Output a large amount of EventData

    XMLWordPrintableJSON

    Details

    • Flags:
      Patch

      Description

      in TwoPhaseCommitSinkFunctionOutput Maybe A large number of EventData will be output(log.info),which will cause IO bottleneck and disk waste.

       
      my code in the attachment, A large number event data output in the log output by flink , e.g: 

      Jdbc2PCSinkFunction 1/1 - checkpoint 4 complete, committing transaction TransactionHolde {handle=Transaction(b420c880a951403984f231dd7e33597b, ListBuffer(insert into table(field1,field2) value ('11','22') ... ... ), transactionStartTime=1610426158532} from checkpoint 4

      in TwoPhaseCommitSinkFunction about LOG.info code is as follows:

      LOG.info(
              "{} - checkpoint {} complete, committing transaction {} from checkpoint {}",
              name(),
              checkpointId,
              pendingTransaction,
              pendingTransactionCheckpointId); 

      will be invoke pendingTransaction'toString method (pendingTransaction is TransactionHolder'instance) 

      TransactionHolder'toString method code is:

      @Override
      public String toString() {
          return "TransactionHolder{"
                  + "handle="
                  +  handle
                  + ", transactionStartTime="
                  + transactionStartTime
                  + '}';
      }

       handle is the concrete realization of my Transaction! There is a parameter of List type in my Transaction, which is used to receive data. as a result, these data are printed out(log.info)
       
       

       

        Attachments

        1. 1610682498960.jpg
          327 kB
          Huajie Wang
        2. 1610682603148.jpg
          140 kB
          Huajie Wang
        3. Jdbc2PCSinkFunction.scala
          5 kB
          Huajie Wang

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              benjobs Huajie Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - 1h
                1h
                Remaining:
                Remaining Estimate - 1h
                1h
                Logged:
                Time Spent - Not Specified
                Not Specified