Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-5065

HTTP Site-to-Site fails sending data more than 2GB at once

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.7.0
    • Component/s: Core Framework
    • Labels:
      None

      Description

      NiFi Site-to-Site (S2S) is designed to support sending large files. While RAW transport protocol works fine with large files (I tested up to 8GB, but it can be more), HTTP transport protocol starts failing if more than 2GB data is sent at a time, meaning by sending a 2GB FlowFile, or batching multiple FlowFiles to make more than 2GB into one transaction.

       

      When it fails, following log messages are written:

      2018-04-10 16:05:45,006 ERROR [I/O dispatcher 25] o.a.n.r.util.SiteToSiteRestApiClient Failed to send data to http://localhost:8080/nifi-api/data-transfer/input-ports/ad9a3887-0162-1000-e312-dee642179
      c9c/transactions/608f1ce4-56da-4899-9348-d2864e364d40/flow-files due to java.lang.RuntimeException: Sending data to http://localhost:8080/nifi-api/data-transfer/input-ports/ad9a3887-0162-1000-e312-dee
      642179c9c/transactions/608f1ce4-56da-4899-9348-d2864e364d40/flow-files has reached to its end, but produced : read : wrote byte sizes (659704502 : 659704502 : 9249639094) were not equal. Something went wr
      ong.
      java.lang.RuntimeException: Sending data to http://localhost:8080/nifi-api/data-transfer/input-ports/ad9a3887-0162-1000-e312-dee642179c9c/transactions/608f1ce4-56da-4899-9348-d2864e364d40/flow-files h
      as reached to its end, but produced : read : wrote byte sizes (659704502 : 659704502 : 9249639094) were not equal. Something went wrong.
              at org.apache.nifi.remote.util.SiteToSiteRestApiClient$4.produceContent(SiteToSiteRestApiClient.java:848)
              at org.apache.http.impl.nio.client.MainClientExec.produceContent(MainClientExec.java:262)
              at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.produceContent(DefaultClientExchangeHandlerImpl.java:140)
              at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.outputReady(HttpAsyncRequestExecutor.java:241)
              at org.apache.http.impl.nio.DefaultNHttpClientConnection.produceOutput(DefaultNHttpClientConnection.java:290)
              at org.apache.http.impl.nio.client.InternalIODispatch.onOutputReady(InternalIODispatch.java:86)
              at org.apache.http.impl.nio.client.InternalIODispatch.onOutputReady(InternalIODispatch.java:39)
              at org.apache.http.impl.nio.reactor.AbstractIODispatch.outputReady(AbstractIODispatch.java:145)
              at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:188)
              at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:341)
              at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
              at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
              at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
              at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
              at java.lang.Thread.run(Thread.java:745)
      2018-04-10 16:06:25,009 ERROR [Timer-Driven Process Thread-3] o.a.nifi.remote.StandardRemoteGroupPort RemoteGroupPort[name=input,targets=http://localhost:8080/nifi] failed to communicate with remote NiFi instance due to java.io.IOException: Failed to confirm transaction with Peer[url=http://localhost:8080/nifi-api] due to java.io.IOException: Awaiting transferDataLatch has been timeout.
      2018-04-10 16:06:25,009 ERROR [Timer-Driven Process Thread-3] o.a.nifi.remote.StandardRemoteGroupPort
      java.io.IOException: Failed to confirm transaction with Peer[url=http://localhost:8080/nifi-api] due to java.io.IOException: Awaiting transferDataLatch has been timeout.
              at org.apache.nifi.remote.AbstractTransaction.confirm(AbstractTransaction.java:264)
              at org.apache.nifi.remote.StandardRemoteGroupPort.transferFlowFiles(StandardRemoteGroupPort.java:369)
              at org.apache.nifi.remote.StandardRemoteGroupPort.onTrigger(StandardRemoteGroupPort.java:285)
              at org.apache.nifi.controller.AbstractPort.onTrigger(AbstractPort.java:250)
              at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:175)
              at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      Caused by: java.io.IOException: Awaiting transferDataLatch has been timeout.
              at org.apache.nifi.remote.util.SiteToSiteRestApiClient.finishTransferFlowFiles(SiteToSiteRestApiClient.java:938)
              at org.apache.nifi.remote.protocol.http.HttpClientTransaction.readTransactionResponse(HttpClientTransaction.java:93)
              at org.apache.nifi.remote.AbstractTransaction.confirm(AbstractTransaction.java:239)
              ... 12 common frames omitted
      

      The cause of this issue is that SiteToSiteRestApiClient.openConnectionForSend method uses int variables to track bytes it sends. When the total size exceeds Integer.MAX_VALUE (2,147,483,647 = 2GB), the int variables get overflown. Then the check condition goes wrong.
      https://github.com/apache/nifi/blob/master/nifi-commons/nifi-site-to-site-client/src/main/java/org/apache/nifi/remote/util/SiteToSiteRestApiClient.java#L777

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ijokarumawak Koji Kawamura
                Reporter:
                ijokarumawak Koji Kawamura
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: