Uploaded image for project: 'Camel'
  1. Camel
  2. CAMEL-4605

Camel ftp read file after close connection

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.8.1
    • 2.8.3, 2.9.0
    • camel-ftp
    • None
    • Unknown

    Description

      There is an error with ftp server.
      Many files on the server. All files have no time to be processed.
      When the processing of the next file, the process stops.
      Error - The request is a file after closing the connection.

      Camel config route:

      <?xml version="1.0" encoding="UTF-8"?>
      <blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
        xmlns:cm="http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.0.0"
        xmlns:camel="http://camel.apache.org/schema/blueprint"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        default-activation="lazy">
      
        <cm:property-placeholder persistent-id="ccwe.exchange.oos">
          <cm:default-properties>
            <cm:property name="ftp.doc.url" value="ftp://mail01.zakupki.gov.ru/auto/organization/all/"/>
            <cm:property name="ftp.doc.add_params" value=""/>
          </cm:default-properties>
        </cm:property-placeholder>
        
        <bean id="readProcess" class="mypackage.docread.ReadProcess"/>
      
        <bean id="routePolicy" 
          class="org.apache.camel.routepolicy.quartz.CronScheduledRoutePolicy">
          <property name="routeStartTime" value="* 0/10 * * * ?"/>
          <property name="routeStopTime" value="* 4/10 * * * ?"/>
        </bean>
      
        <camelContext id="exchange-read" 
           xmlns="http://camel.apache.org/schema/blueprint" autoStartup="false">
           
          <threadPoolProfile id="fooProfile" 
             poolSize="2" maxPoolSize="2" maxQueueSize="-1"/>
             
          <route id="readRoute" 
            routePolicyRef="routePolicy"
            shutdownRunningTask= "CompleteAllTasks">
            
            <from uri="{{ftp.doc.url}}?noop=true&amp;binary=true{{ftp.doc.add_params}}"/>
            <to uri="bean:readProcess"/>
          </route>
        </camelContext>
      </blueprint>
      

      We have two streams

      <threadPoolProfile id="fooProfile" poolSize="2" maxPoolSize="2" maxQueueSize="-1"/>

      The first thread (thread#0) reading file from ftp server

      2011-11-02 09:13:57,376 | TRACE | Camel (exchange-read) thread #0 - ftp://mail01.zakupki.gov.ru/auto/organization/all/ | FtpConsumer                      | mponent.file.GenericFileConsumer  261 | 94 - org.apache.camel.camel-core - 2.8.1 |
           Processing file: GenericFile[organization_all_20111023_010003_33.xml.zip]
      2011-11-02 09:13:57,376 | TRACE | Camel (exchange-read) thread #0 - ftp://mail01.zakupki.gov.ru/auto/organization/all/ | FtpConsumer                      | mponent.file.GenericFileConsumer  293 | 94 - org.apache.camel.camel-core - 2.8.1 |
           Retrieving file: auto/organization/all/organization_all_20111023_010003_33.xml.zip from: Endpoint[ftp://mail01.zakupki.gov.ru/auto/organization/all/?
      binary=true&connectTimeout=10000&delay=600000&disconnect=true&localWorkDirectory=%2Fhome%2Fuser%2Ftemp&noop=true&passiveMode=true&soTimeout=10000&timeout=15000]
      

      The second stream (thread#1) is stopped

      2011-11-02 09:14:00,004 | DEBUG | Camel (exchange-read) thread #1 - ShutdownTask | ScheduledPollConsumer            | camel.impl.ScheduledPollConsumer  265 | 94 - org.apache.camel.camel-core - 2.8.1 | 
           This consumer is stopping, so cancelling scheduled task: java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@2bf8d0a4
      2011-11-02 09:14:00,005 | DEBUG | Camel (exchange-read) thread #1 - ShutdownTask | FtpConsumer                      | pache.camel.impl.DefaultConsumer   83 | 94 - org.apache.camel.camel-core - 2.8.1 | 
           Stopping consumer: FtpConsumer[ftp://mail01.zakupki.gov.ru/auto/organization/all/?binary=true&connectTimeout=10000&delay=600000&disconnect=true&localWorkDirectory=%2Fhome%2Fuser%2Ftemp&noop=true&passiveMode=true&soTimeout=10000&timeout=15000]
      2011-11-02 09:14:00,007 | DEBUG | Camel (exchange-read) thread #1 - ShutdownTask | ProcessorEndpoint$1              | pache.camel.impl.DefaultProducer   76 | 94 - org.apache.camel.camel-core - 2.8.1 | 
           Stopping producer: Producer[bean://readProcess]
      2011-11-02 09:14:00,010 | DEBUG | Camel (exchange-read) thread #1 - ShutdownTask | FtpConsumer                      | t.file.remote.RemoteFileConsumer  102 | 94 - org.apache.camel.camel-core - 2.8.1 | 
           Disconnecting from: ftp://anonymous@mail01.zakupki.gov.ru:21
      

      Warning!!!! ftp close connection. Completed method FTPClient.disconnect(). Parament controlInput = null.

      2011-11-02 09:14:03,182 | INFO  | Camel (exchange-read) thread #1 - ShutdownTask | DefaultShutdownStrategy          | ultShutdownStrategy$ShutdownTask  460 | 94 - org.apache.camel.camel-core - 2.8.1 | 
           Route: readRoute shutdown complete, was consuming from: Endpoint[...]
      2011-11-02 09:14:03,182 | INFO  | DefaultQuartzScheduler-exchange-read_Worker-1 | DefaultShutdownStrategy          | mel.impl.DefaultShutdownStrategy  158 | 94 - org.apache.camel.camel-core - 2.8.1 | 
           Graceful shutdown of 1 routes completed in 3 seconds
      

      The first thread (thread#0) continues reading the file [organization_all_20111023_010003_33.xml.zip].

      2011-11-02 09:14:04,172 | ERROR | Camel (exchange-read) thread #0 - ftp://mail01.zakupki.gov.ru/auto/organization/all/ | FtpConsumer                      | ache.camel.processor.CamelLogger  232 | 94 - org.apache.camel.camel-core - 2.8.1 | Caused by: [java.lang.NullPointerException - null]
      java.lang.NullPointerException
      	at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:295)[97:org.apache.commons.net:2.2]
      	at org.apache.commons.net.ftp.FTP.getReply(FTP.java:622)[97:org.apache.commons.net:2.2]
      	at org.apache.commons.net.ftp.FTPClient.completePendingCommand(FTPClient.java:1408)[97:org.apache.commons.net:2.2]
      	at org.apache.commons.net.ftp.FTPClient.retrieveFile(FTPClient.java:1467)[97:org.apache.commons.net:2.2]
      	at org.apache.camel.component.file.remote.FtpOperations.retrieveFileToFileInLocalWorkDirectory(FtpOperations.java:406)[99:org.apache.camel.camel-ftp:2.8.1]
      

      Attachments

        Activity

          People

            davsclaus Claus Ibsen
            alexey-s Aleksey Sushko
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: