Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-269

Provide ability to throttle DeltaStreamer sync runs

    XMLWordPrintableJSON

    Details

      Description

      Copied from https://github.com/apache/incubator-hudi/issues/922

      In some scenario in our cluster, we may want delta streamer to slow down a bit.
      so it's nice to have a parameter to control the min sync interval of each sync in continuous mode.
      this param is default to 0, so this should not affect current logic.

      minor pr: #921

      the main reason we want to slow it down is that aws s3 is charged by s3 get/put/list requests. we don't want to pay for too many requests for a really slow change table.

        Attachments

        1. image-2019-09-26-09-02-24-761.png
          168 kB
          Xing Pan
        2. image-2019-09-25-08-51-19-686.png
          613 kB
          Xing Pan
        3. hudi_request_test.tar.gz
          10 kB
          Xing Pan

          Issue Links

            Activity

              People

              • Assignee:
                XingXPan Xing Pan
                Reporter:
                vbalaji Balaji Varadarajan
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 10m
                  10m