Synapse
  1. Synapse
  2. SYNAPSE-847

VFS transport throws exception, when try to pull large (>150kb) file from FTP location

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0
    • Component/s: Transports
    • Labels:
      None

      Description

      When try to pull larger file from an ftp location , i face the following issue[1]..I used following proxy configuration[2]

      The issue is here, vfs transport listener closes the stream before the vfs transport sender reads it.. As a workaround, i added a Stringbuffer @ listener with a limited buffer size to read the soap envelope..
      Attached the patch with this workaround..
      Please review and commit

      [1]
      r the specified or optional timeout
      [2012-02-29 15:39:19,321] ERROR - VFSTransportListener Error closing stream
      org.apache.commons.vfs2.FileSystemException: Could not close the input stream for file "ftp://anonymous@localhost/in/textfile.txt?vfs.passive=true".
      at org.apache.commons.vfs2.provider.DefaultFileContent$FileContentInputStream.close(DefaultFileContent.java:612)
      at org.apache.synapse.transport.vfs.VFSTransportListener.processFile(VFSTransportListener.java:566)
      at org.apache.synapse.transport.vfs.VFSTransportListener.scanFileOrDirectory(VFSTransportListener.java:306)
      at org.apache.synapse.transport.vfs.VFSTransportListener.poll(VFSTransportListener.java:157)
      at org.apache.synapse.transport.vfs.VFSTransportListener.poll(VFSTransportListener.java:106)
      at org.apache.axis2.transport.base.AbstractPollingTransportListener$1$1.run(AbstractPollingTransportListener.java:67)
      at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:173)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:619)
      Caused by: org.apache.commons.vfs2.FileSystemException: Could not get FTP file "ftp://anonymous@localhost/in/textfile.txt?vfs.passive=true".
      at org.apache.commons.vfs2.provider.ftp.FtpFileObject$FtpInputStream.onClose(FtpFileObject.java:731)
      at org.apache.commons.vfs2.util.MonitorInputStream.close(MonitorInputStream.java:140)
      at java.io.BufferedInputStream.close(BufferedInputStream.java:451)
      at org.apache.commons.vfs2.util.MonitorInputStream.close(MonitorInputStream.java:130)
      at org.apache.commons.vfs2.provider.DefaultFileContent$FileContentInputStream.close(DefaultFileContent.java:608)
      ... 9 more
      [2012-02-29 15:39:19,324] ERROR - VFSTransportListener Error processing File URI : ftp://anonymous@localhost/in/textfile.txt?vfs.passive=true
      org.apache.axis2.AxisFault: Error closing stream
      at org.apache.axis2.transport.base.AbstractTransportListener.handleException(AbstractTransportListener.java:343)
      at org.apache.synapse.transport.vfs.VFSTransportListener.processFile(VFSTransportListener.java:568)
      at org.apache.synapse.transport.vfs.VFSTransportListener.scanFileOrDirectory(VFSTransportListener.java:306)
      at org.apache.synapse.transport.vfs.VFSTransportListener.poll(VFSTransportListener.java:157)
      at org.apache.synapse.transport.vfs.VFSTransportListener.poll(VFSTransportListener.java:106)
      at org.apache.axis2.transport.base.AbstractPollingTransportListener$1$1.run(AbstractPollingTransportListener.java:67)
      at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:173)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:619)
      Caused by: org.apache.commons.vfs2.FileSystemException: Could not close the input stream for file "ftp://anonymous@localhost/in/textfile.txt?vfs.passi
      ve=true".
      at org.apache.commons.vfs2.provider.DefaultFileContent$FileContentInputStream.close(DefaultFileContent.java:612)
      at org.apache.synapse.transport.vfs.VFSTransportListener.processFile(VFSTransportListener.java:566)
      ... 8 more
      Caused by: org.apache.commons.vfs2.FileSystemException: Could not get FTP file "ftp://anonymous@localhost/in/textfile.txt?vfs.passive=true".
      at org.apache.commons.vfs2.provider.ftp.FtpFileObject$FtpInputStream.onClose(FtpFileObject.java:731)

      1. vfsTransport.patch
        3 kB
        Vijayaratha Vijayasingam

        Activity

        Vijayaratha Vijayasingam created issue -
        Vijayaratha Vijayasingam made changes -
        Field Original Value New Value
        Attachment vfsListenerPatch.patch [ 12516543 ]
        Hide
        Vijayaratha Vijayasingam added a comment -

        Sample proxy;
        <proxy name="StockQuoteProxy" transports="vfs" startOnLoad="true" trace="disable">
        <target faultSequence="fault">
        <inSequence>
        <property name="OUT_ONLY" value="true"/>
        <send>
        <endpoint>
        <address uri="vfs:file:C:\Users\TOSHIBA\Desktop\out1"/>
        </endpoint>
        </send>
        </inSequence>
        </target>
        <parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
        <parameter name="transport.PollInterval">15</parameter>
        <parameter name="transport.vfs.MoveAfterProcess">file:C:\Users\TOSHIBA\Desktop\original</parameter>
        <parameter name="transport.vfs.FileURI">vfs:ftp://anonymous@localhost/in?vfs.passive=true</parameter>
        <parameter name="transport.vfs.MoveAfterFailure">file:C:\Users\TOSHIBA\Desktop\error</parameter>
        <parameter name="transport.vfs.FileNamePattern">.*</parameter>
        <parameter name="transport.vfs.ContentType">text/plain</parameter>
        <parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
        <parameter name="serviceType">proxy</parameter>
        </proxy>

        Show
        Vijayaratha Vijayasingam added a comment - Sample proxy; <proxy name="StockQuoteProxy" transports="vfs" startOnLoad="true" trace="disable"> <target faultSequence="fault"> <inSequence> <property name="OUT_ONLY" value="true"/> <send> <endpoint> <address uri="vfs: file:C:\Users\TOSHIBA\Desktop\out1 "/> </endpoint> </send> </inSequence> </target> <parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter> <parameter name="transport.PollInterval">15</parameter> <parameter name="transport.vfs.MoveAfterProcess"> file:C:\Users\TOSHIBA\Desktop\original </parameter> <parameter name="transport.vfs.FileURI">vfs: ftp://anonymous@localhost/in?vfs.passive=true </parameter> <parameter name="transport.vfs.MoveAfterFailure"> file:C:\Users\TOSHIBA\Desktop\error </parameter> <parameter name="transport.vfs.FileNamePattern">.*</parameter> <parameter name="transport.vfs.ContentType">text/plain</parameter> <parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter> <parameter name="serviceType">proxy</parameter> </proxy>
        Vijayaratha Vijayasingam made changes -
        Description When try to pull larger file from an ftp location , i face the following issue[1]..I used following proxy configuration[2]


        The issue is here, vfs transport listener closes the stream before the vfs transport sender reads it.. As a workaround, i added a Stringbuffer @ listener with a limited buffer size to read the soap envelope..
        Attached the patch with this workaround..
        Please review and commit


        [1]
        2012-02-15 14:34:53,066 [-] [vfs-Worker-2] ERROR VFSTransportListener Error closing stream
        org.apache.commons.vfs2.FileSystemException: Could not close the input stream for file "ftp://sonat\eman:***@ftp.sonat.se/IPS/Shp_110201_050048.txt?vfs.passive=true".
               at org.apache.commons.vfs2.provider.DefaultFileContent$FileContentInputStream.close(DefaultFileContent.java:612)
               at org.apache.synapse.transport.vfs.VFSTransportListener.processFile(VFSTransportListener.java:566)
               at org.apache.synapse.transport.vfs.VFSTransportListener.scanFileOrDirectory(VFSTransportListener.java:306)
               at org.apache.synapse.transport.vfs.VFSTransportListener.poll(VFSTransportListener.java:157)
               at org.apache.synapse.transport.vfs.VFSTransportListener.poll(VFSTransportListener.java:106)
               at org.apache.axis2.transport.base.AbstractPollingTransportListener$1$1.run(AbstractPollingTransportListener.java:67)
               at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:173)
               at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
               at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
               at java.lang.Thread.run(Thread.java:662)
        Caused by: org.apache.commons.vfs2.FileSystemException: Could not get FTP file "ftp://sonat\eman:***@ftp.sonat.se/IPS/Shp_110201_050048.txt?vfs.passive=true".
               at org.apache.commons.vfs2.provider.ftp.FtpFileObject$FtpInputStream.onClose(FtpFileObject.java:731)
               at org.apache.commons.vfs2.util.MonitorInputStream.close(MonitorInputStream.java:140)
               at java.io.BufferedInputStream.close(BufferedInputStream.java:451)
               at org.apache.commons.vfs2.util.MonitorInputStream.close(MonitorInputStream.java:130)
               at org.apache.commons.vfs2.provider.DefaultFileContent$FileContentInputStream.close(DefaultFileContent.java:608)
               ... 9 more
        2012-02-15 14:34:53,067 [-] [vfs-Worker-2] ERROR VFSTransportListener Error processing File URI : ftp://sonat\eman:Someniller.,.,@ftp.sonat.se/IPS/Shp_110201_050048.txt?vfs.passive=true
        org.apache.axis2.AxisFault: Error closing stream
               at org.apache.axis2.transport.base.AbstractTransportListener.handleException(AbstractTransportListener.java:343)
               at org.apache.synapse.transport.vfs.VFSTransportListener.processFile(VFSTransportListener.java:568)
               at org.apache.synapse.transport.vfs.VFSTransportListener.scanFileOrDirectory(VFSTransportListener.java:306)
               at org.apache.synapse.transport.vfs.VFSTransportListener.poll(VFSTransportListener.java:157)
               at org.apache.synapse.transport.vfs.VFSTransportListener.poll(VFSTransportListener.java:106)
               at org.apache.axis2.transport.base.AbstractPollingTransportListener$1$1.run(AbstractPollingTransportListener.java:67)
               at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:173)
               at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
               at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
               at java.lang.Thread.run(Thread.java:662)
        When try to pull larger file from an ftp location , i face the following issue[1]..I used following proxy configuration[2]


        The issue is here, vfs transport listener closes the stream before the vfs transport sender reads it.. As a workaround, i added a Stringbuffer @ listener with a limited buffer size to read the soap envelope..
        Attached the patch with this workaround..
        Please review and commit


        [1]
        r the specified or optional timeout
        [2012-02-29 15:39:19,321] ERROR - VFSTransportListener Error closing stream
        org.apache.commons.vfs2.FileSystemException: Could not close the input stream for file "ftp://anonymous@localhost/in/textfile.txt?vfs.passive=true".
                at org.apache.commons.vfs2.provider.DefaultFileContent$FileContentInputStream.close(DefaultFileContent.java:612)
                at org.apache.synapse.transport.vfs.VFSTransportListener.processFile(VFSTransportListener.java:566)
                at org.apache.synapse.transport.vfs.VFSTransportListener.scanFileOrDirectory(VFSTransportListener.java:306)
                at org.apache.synapse.transport.vfs.VFSTransportListener.poll(VFSTransportListener.java:157)
                at org.apache.synapse.transport.vfs.VFSTransportListener.poll(VFSTransportListener.java:106)
                at org.apache.axis2.transport.base.AbstractPollingTransportListener$1$1.run(AbstractPollingTransportListener.java:67)
                at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:173)
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                at java.lang.Thread.run(Thread.java:619)
        Caused by: org.apache.commons.vfs2.FileSystemException: Could not get FTP file "ftp://anonymous@localhost/in/textfile.txt?vfs.passive=true".
                at org.apache.commons.vfs2.provider.ftp.FtpFileObject$FtpInputStream.onClose(FtpFileObject.java:731)
                at org.apache.commons.vfs2.util.MonitorInputStream.close(MonitorInputStream.java:140)
                at java.io.BufferedInputStream.close(BufferedInputStream.java:451)
                at org.apache.commons.vfs2.util.MonitorInputStream.close(MonitorInputStream.java:130)
                at org.apache.commons.vfs2.provider.DefaultFileContent$FileContentInputStream.close(DefaultFileContent.java:608)
                ... 9 more
        [2012-02-29 15:39:19,324] ERROR - VFSTransportListener Error processing File URI : ftp://anonymous@localhost/in/textfile.txt?vfs.passive=true
        org.apache.axis2.AxisFault: Error closing stream
                at org.apache.axis2.transport.base.AbstractTransportListener.handleException(AbstractTransportListener.java:343)
                at org.apache.synapse.transport.vfs.VFSTransportListener.processFile(VFSTransportListener.java:568)
                at org.apache.synapse.transport.vfs.VFSTransportListener.scanFileOrDirectory(VFSTransportListener.java:306)
                at org.apache.synapse.transport.vfs.VFSTransportListener.poll(VFSTransportListener.java:157)
                at org.apache.synapse.transport.vfs.VFSTransportListener.poll(VFSTransportListener.java:106)
                at org.apache.axis2.transport.base.AbstractPollingTransportListener$1$1.run(AbstractPollingTransportListener.java:67)
                at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:173)
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                at java.lang.Thread.run(Thread.java:619)
        Caused by: org.apache.commons.vfs2.FileSystemException: Could not close the input stream for file "ftp://anonymous@localhost/in/textfile.txt?vfs.passi
        ve=true".
                at org.apache.commons.vfs2.provider.DefaultFileContent$FileContentInputStream.close(DefaultFileContent.java:612)
                at org.apache.synapse.transport.vfs.VFSTransportListener.processFile(VFSTransportListener.java:566)
                ... 8 more
        Caused by: org.apache.commons.vfs2.FileSystemException: Could not get FTP file "ftp://anonymous@localhost/in/textfile.txt?vfs.passive=true".
                at org.apache.commons.vfs2.provider.ftp.FtpFileObject$FtpInputStream.onClose(FtpFileObject.java:731)
        Udayanga Wickramasinghe made changes -
        Assignee Udayanga Wickramasinghe [ udayangaw@wso2.com ]
        Hide
        Sadeep Jayasumana added a comment -

        Hi,

        This issue has been around for long time (see [1]). It's good to see that the root cause of the issue has been found.

        However, I'm not quite sure whether the attached patch should be checked in. With this patch, each message is forced to build within the memory, defeating the streaming nature of Syanpse. With this patch VFS transport can drive Synpase out of memory, if large files are picked and/or a large number of files are picked concurrently.

        There should be another method to prevent the transport listener from closing the stream before it's completely read. Maybe we can wrap the input stream we get from Commons VFS to add auto closing functionality (IIRC a similar concept is implemented in Axis2 Servlet Transport).

        [1] http://markmail.org/message/zphmtx3bi7mzbwzq

        Thanks,
        Sadeep

        Show
        Sadeep Jayasumana added a comment - Hi, This issue has been around for long time (see [1] ). It's good to see that the root cause of the issue has been found. However, I'm not quite sure whether the attached patch should be checked in. With this patch, each message is forced to build within the memory, defeating the streaming nature of Syanpse. With this patch VFS transport can drive Synpase out of memory, if large files are picked and/or a large number of files are picked concurrently. There should be another method to prevent the transport listener from closing the stream before it's completely read. Maybe we can wrap the input stream we get from Commons VFS to add auto closing functionality (IIRC a similar concept is implemented in Axis2 Servlet Transport). [1] http://markmail.org/message/zphmtx3bi7mzbwzq Thanks, Sadeep
        Hide
        Vijayaratha Vijayasingam added a comment - - edited

        Thanks sadeep for the input..I have updated the patch...
        We can use the AutoCloseInputstream implementation, which is available in commons.io library to close all the input streams...Anyway at transport listener, we need to close the file object..
        Where, commons-vfs lib tries to close inputstream and detach the file..So, there is a warning comes to synapse end when vfs lib tries to close the stream..

        Since we handover the inputstream closing job to another process at synapse end , I think we can avoid this warning coming from commons-vfs library..

        Show
        Vijayaratha Vijayasingam added a comment - - edited Thanks sadeep for the input..I have updated the patch... We can use the AutoCloseInputstream implementation, which is available in commons.io library to close all the input streams...Anyway at transport listener, we need to close the file object.. Where, commons-vfs lib tries to close inputstream and detach the file..So, there is a warning comes to synapse end when vfs lib tries to close the stream.. Since we handover the inputstream closing job to another process at synapse end , I think we can avoid this warning coming from commons-vfs library..
        Vijayaratha Vijayasingam made changes -
        Attachment vfsListenerPatch.patch [ 12516543 ]
        Hide
        Vijayaratha Vijayasingam added a comment -

        Attaching updated one

        Show
        Vijayaratha Vijayasingam added a comment - Attaching updated one
        Vijayaratha Vijayasingam made changes -
        Attachment vfsTransport.patch [ 12518010 ]
        Hide
        Hiranya Jayathilaka added a comment -

        Patch applied. Thanks Ratha for the contribution.

        Show
        Hiranya Jayathilaka added a comment - Patch applied. Thanks Ratha for the contribution.
        Hiranya Jayathilaka made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Synapse - Trunk #4621 (See https://builds.apache.org/job/Synapse%20-%20Trunk/4621/)
        Applying the patch for SYNAPSE-847; Getting rid of redundant code to close the file input stream. This is taken care of by the FileObject.close() method. Also using an AutoCloseInputStream so that the input stream will be closed early if Synapse consumes it entirely. (hiranya: rev 1504674)

        • /synapse/trunk/java/modules/transports/core/vfs/pom.xml
        • /synapse/trunk/java/modules/transports/core/vfs/src/main/java/org/apache/synapse/transport/vfs/VFSTransportListener.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Synapse - Trunk #4621 (See https://builds.apache.org/job/Synapse%20-%20Trunk/4621/ ) Applying the patch for SYNAPSE-847 ; Getting rid of redundant code to close the file input stream. This is taken care of by the FileObject.close() method. Also using an AutoCloseInputStream so that the input stream will be closed early if Synapse consumes it entirely. (hiranya: rev 1504674) /synapse/trunk/java/modules/transports/core/vfs/pom.xml /synapse/trunk/java/modules/transports/core/vfs/src/main/java/org/apache/synapse/transport/vfs/VFSTransportListener.java
        Hiranya Jayathilaka made changes -
        Fix Version/s 3.0 [ 12324903 ]
        Fix Version/s FUTURE [ 12313086 ]

          People

          • Assignee:
            Udayanga Wickramasinghe
            Reporter:
            Vijayaratha Vijayasingam
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development