Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: 2.0.0
    • Fix Version/s: Future
    • Component/s: None
    • Labels:
      None

      Description

      Add a new component camel-commons-vfs which utilises the commons-vfs library providing access to zip,jar,tgz,cifs,ftp and many other wonderful "file" type resources.

      The component will go into camel 2.0 as the new camel-file and camel-ftp components have been re-written to support a better model.

      1. commons-vfs-changes-part1.file-refactoring.diff
        245 kB
        Ramon Buckland
      2. CAMEL-1241.part_1_port_Generic_to_camel-core.diff
        72 kB
        Ramon Buckland
      3. CAMEL-1241_part2.zip
        73 kB
        Ramon Buckland
      4. CAMEL-1241_part2_all_files.tgz
        35 kB
        Ramon Buckland

        Issue Links

          Activity

          Hide
          Ramon Buckland added a comment -

          I have been investigating how this change will impact / change the recent refactoring work for camel-file and camel-ftp.

          There is a set of interfaces and some implementation largely named Remote* .. for example RemoteFileOperation.

          This work will refactor some of these classes, making interfaces and sorting out some other interfaces.

          I propose the following.

          Pull-up a more generic set of interface and base classes, called Generic*.

          • These Generic iface/classes will be 90%-95% of the current Remote.. so largely a renaming.
          • What is "Remote" specific, such as Hostname, will come to a lower "remote" implementation

          commons-vfs will hang off the Generic and the existing camel-file can also hang off this generic set of classes.

          Any comments ?

          Show
          Ramon Buckland added a comment - I have been investigating how this change will impact / change the recent refactoring work for camel-file and camel-ftp. There is a set of interfaces and some implementation largely named Remote* .. for example RemoteFileOperation. This work will refactor some of these classes, making interfaces and sorting out some other interfaces. I propose the following. Pull-up a more generic set of interface and base classes, called Generic*. These Generic iface/classes will be 90%-95% of the current Remote.. so largely a renaming. What is "Remote" specific, such as Hostname, will come to a lower "remote" implementation commons-vfs will hang off the Generic and the existing camel-file can also hang off this generic set of classes. Any comments ?
          Hide
          Claus Ibsen added a comment -

          +1

          Yeah when we poll the interfaces from camel-ftp into camel-core they need a rename to be generic, so GenericFileOperation will do.

          I just wanted to be sure that commons-vfs can integrate with this approach using the interfaces from camel-ftp allowing us to have much more shared code for the different file based transports

          • java.io.File
          • regular FTP
          • SFTP
          • and now commons-vfs
          Show
          Claus Ibsen added a comment - +1 Yeah when we poll the interfaces from camel-ftp into camel-core they need a rename to be generic, so GenericFileOperation will do. I just wanted to be sure that commons-vfs can integrate with this approach using the interfaces from camel-ftp allowing us to have much more shared code for the different file based transports java.io.File regular FTP SFTP and now commons-vfs
          Hide
          Ramon Buckland added a comment -

          Yes, it appears with the minor changes that the new 2.0 changes can be adapted without much hassle for more file goodness.

          Show
          Ramon Buckland added a comment - Yes, it appears with the minor changes that the new 2.0 changes can be adapted without much hassle for more file goodness.
          Hide
          Claus Ibsen added a comment -

          @Ramon

          That is fantastic. Makes me smile the rest of the day

          Show
          Claus Ibsen added a comment - @Ramon That is fantastic. Makes me smile the rest of the day
          Hide
          Ramon Buckland added a comment -

          Attached is the refactoring to "extract out" the "generic File stuff" from the RemoteFile* implementation.

          All the tests are working as they were with this refactoring.

          Now I will create a commons-vfs implementation underneath the Generic*

          Note, I am uploading this now so that other can see what I am currently doing, get an idea for it.

          Hopefully I will have commons-vfs completed by tomorrow, at which time I will supply the new patch that is EVERYTHING including this attached one.

          Show
          Ramon Buckland added a comment - Attached is the refactoring to "extract out" the "generic File stuff" from the RemoteFile* implementation. All the tests are working as they were with this refactoring. Now I will create a commons-vfs implementation underneath the Generic* Note, I am uploading this now so that other can see what I am currently doing, get an idea for it. Hopefully I will have commons-vfs completed by tomorrow, at which time I will supply the new patch that is EVERYTHING including this attached one.
          Hide
          Ramon Buckland added a comment -

          As per Claus' suggestion. This is part 1 of the list below ..

          So if you can provide diffs in multiple steps. A rough plan like this:

          1) Generic* that are used by the FTP component
          2) When #1 is commited to triunk and you code is aligned with trunk,
          then move on to
          3) Move Generic* from camel-ftp to camel-core
          4) When #3 is commited to triunk and you code is aligned with trunk,
          then move on to
          5) You can start on commons-vfs component
          6) Then we can take a look at the java generics stuff <T> and get that
          done in the right way
          7) Refactor File in camel-core to use the new generics
          8) Add the new commons-vfs component when its ready and done
          9) Add wiki doc for the new commons-vfs component

          Show
          Ramon Buckland added a comment - As per Claus' suggestion. This is part 1 of the list below .. So if you can provide diffs in multiple steps. A rough plan like this: 1) Generic* that are used by the FTP component 2) When #1 is commited to triunk and you code is aligned with trunk, then move on to 3) Move Generic* from camel-ftp to camel-core 4) When #3 is commited to triunk and you code is aligned with trunk, then move on to 5) You can start on commons-vfs component 6) Then we can take a look at the java generics stuff <T> and get that done in the right way 7) Refactor File in camel-core to use the new generics 8) Add the new commons-vfs component when its ready and done 9) Add wiki doc for the new commons-vfs component
          Hide
          Claus Ibsen added a comment -

          Thanks for the patch.

          I renamed the sorter to be GenericXXX as well so they all are starting with Generic. Then its easier to spot.
          And also fixed the checkstyle issues. Please dont auto wrap the code. It gets ugly.

          We should add generics to these interfaces, later. (the <T> stuff.)
          Its easier to do when we refactor the FTP component to use these base classes.

          Show
          Claus Ibsen added a comment - Thanks for the patch. I renamed the sorter to be GenericXXX as well so they all are starting with Generic. Then its easier to spot. And also fixed the checkstyle issues. Please dont auto wrap the code. It gets ugly. We should add generics to these interfaces, later. (the <T> stuff.) Its easier to do when we refactor the FTP component to use these base classes.
          Hide
          Claus Ibsen added a comment -

          Ramon I have done some modifications to the camel-core in trunk.

          I stated on #7 to get a due diligence test of the GenericXXX classes. I created a NewFileXXX that is the regular file integrating using the GenericXXX classes.
          I basically got it working with both the consumer and the producer but have only tested it using two new unit tests. At present time I want it to not interfer with the regular File component as its used heavily in entire Camel for unit testing. So the NewFileXX must be rock solid before we let it replace the old one.

          There a some TODOs in the code for stuff to consider and check upon. I do think the GenericFileOperations might could change to pass in the GenericFile instead of the T type I have now. But more work on that will tell.

          Also from the FTP point of with, as I have added generics to it, I would like to see especially the SFTPConsumer if it can use it. I kinda remember it had a sucky API representing the remote FTP file.

          Show
          Claus Ibsen added a comment - Ramon I have done some modifications to the camel-core in trunk. I stated on #7 to get a due diligence test of the GenericXXX classes. I created a NewFileXXX that is the regular file integrating using the GenericXXX classes. I basically got it working with both the consumer and the producer but have only tested it using two new unit tests. At present time I want it to not interfer with the regular File component as its used heavily in entire Camel for unit testing. So the NewFileXX must be rock solid before we let it replace the old one. There a some TODOs in the code for stuff to consider and check upon. I do think the GenericFileOperations might could change to pass in the GenericFile instead of the T type I have now. But more work on that will tell. Also from the FTP point of with, as I have added generics to it, I would like to see especially the SFTPConsumer if it can use it. I kinda remember it had a sucky API representing the remote FTP file.
          Hide
          Ramon Buckland added a comment -

          Thanks Claus, I will integrate your changes on my trunk now and get it all working.

          In my local working tree, I will be making sure that the FULL suit of tests for Camel pass before I push through the patch; as you say, File is used in a lot of places already. (as I also have seen).

          Yes, you are right the SFTP API is a little different. I did start on pushing common "code" up one level below a remote implementation. But I'll revise again shortly.

          Show
          Ramon Buckland added a comment - Thanks Claus, I will integrate your changes on my trunk now and get it all working. In my local working tree, I will be making sure that the FULL suit of tests for Camel pass before I push through the patch; as you say, File is used in a lot of places already. (as I also have seen). Yes, you are right the SFTP API is a little different. I did start on pushing common "code" up one level below a remote implementation. But I'll revise again shortly.
          Hide
          Claus Ibsen added a comment -

          I updated it yet again with a minor fix to add newfile as a componet scheme so its possible to test it by replacing "file://" with "newfile://".

          The reason is so file and newfile can reside side by side until its rock solid and can replace file.

          Ramon do you have any changes to the file component? If not I would like to get that sorted out. I would advice that you focus on the new commons-vfs component that you originally wanted to add. Or you can start refactoring the FTP/SFTP component to use GenericXXX.

          What do you say?

          Show
          Claus Ibsen added a comment - I updated it yet again with a minor fix to add newfile as a componet scheme so its possible to test it by replacing "file://" with "newfile://". The reason is so file and newfile can reside side by side until its rock solid and can replace file. Ramon do you have any changes to the file component? If not I would like to get that sorted out. I would advice that you focus on the new commons-vfs component that you originally wanted to add. Or you can start refactoring the FTP/SFTP component to use GenericXXX. What do you say?
          Hide
          Ramon Buckland added a comment -

          I am about another hour or two from having the ftp/sftp using the new GenericXX components. From there I will add in the commons-vfs. Expect a patch soon.

          Show
          Ramon Buckland added a comment - I am about another hour or two from having the ftp/sftp using the new GenericXX components. From there I will add in the commons-vfs. Expect a patch soon.
          Hide
          Ramon Buckland added a comment -

          This patch implements the GenericFile* classes / ifaces across the original Remote* classes.

          Specifically, the following occurs with this patch.

          (a) minor modifications to the new Generic* classes in camel-core (to support RemoteFile style endpoints)
          (b) added a method to DefaultEndpoint to obtain the "protocol" in the uri (so we can re-lookup our META-INF file to find default strategy)
          (c) refactored all the FTP tests to use target/ftpserver/home as the ftp server root, instead of res/

          This patch readies the camel-core / camel-ftp (file based) so we can now implement commons-vfs into the fray.

          All Tests are working as of diff revision.

          Show
          Ramon Buckland added a comment - This patch implements the GenericFile* classes / ifaces across the original Remote* classes. Specifically, the following occurs with this patch. (a) minor modifications to the new Generic* classes in camel-core (to support RemoteFile style endpoints) (b) added a method to DefaultEndpoint to obtain the "protocol" in the uri (so we can re-lookup our META-INF file to find default strategy) (c) refactored all the FTP tests to use target/ftpserver/home as the ftp server root, instead of res/ This patch readies the camel-core / camel-ftp (file based) so we can now implement commons-vfs into the fray. All Tests are working as of diff revision.
          Hide
          Ramon Buckland added a comment -

          COMPLETE 1) Generic* that are used by the FTP component
          COMPLETE 2) When #1 is commited to triunk and you code is aligned with trunk, then move on to
          COMPLETE 3) Move Generic* from camel-ftp to camel-core
          WAITING COMMIT 4) When #3 is commited to triunk and you code is aligned with trunk, then move on to
          INPROGRESS 5) You can start on commons-vfs component
          INPROGRESS 6) Then we can take a look at the java generics stuff <T> and get that done in the right way
          Claus STARTED 7) Refactor File in camel-core to use the new generics
          8) Add the new commons-vfs component when its ready and done
          9) Add wiki doc for the new commons-vfs component

          Show
          Ramon Buckland added a comment - COMPLETE 1) Generic* that are used by the FTP component COMPLETE 2) When #1 is commited to triunk and you code is aligned with trunk, then move on to COMPLETE 3) Move Generic* from camel-ftp to camel-core WAITING COMMIT 4) When #3 is commited to triunk and you code is aligned with trunk, then move on to INPROGRESS 5) You can start on commons-vfs component INPROGRESS 6) Then we can take a look at the java generics stuff <T> and get that done in the right way Claus STARTED 7) Refactor File in camel-core to use the new generics 8) Add the new commons-vfs component when its ready and done 9) Add wiki doc for the new commons-vfs component
          Hide
          Ramon Buckland added a comment -

          Fixed the patch (I had local uber pom.xml changes for future work not yet required) .. removed these.

          Show
          Ramon Buckland added a comment - Fixed the patch (I had local uber pom.xml changes for future work not yet required) .. removed these.
          Hide
          Ramon Buckland added a comment -

          Fixed the getUriEndpointProtocol (removed from DefaultEndpoint)
          and changed to GenericFileEndpoint.getScheme()

          Broke patch into two
          camel-core
          camel-ftp (absolute or relative directory patch supplied)

          Both need to be applied

          Show
          Ramon Buckland added a comment - Fixed the getUriEndpointProtocol (removed from DefaultEndpoint) and changed to GenericFileEndpoint.getScheme() Broke patch into two camel-core camel-ftp (absolute or relative directory patch supplied) Both need to be applied
          Hide
          Ramon Buckland added a comment -

          In this patch release (CAMEL-1241_part2.zip) is as follows

          • Included now is the patch fixing CAMEL-1290
          • Patch files

          ============== The patch Files ===============
          1. For camel-core (/trunk$ svn diff camel-core)
          File: CAMEL-1241_part2_camel-core.patch

          === For 2, use either ===
          Claus, you mentioned patching in through IDEA and I know eclipse roots projects at the pom.xml so I figured the patch is needed
          at that level (2b). Both are supplied. but have identical changes (of course)

          2a. For camel-ftp Generated at the project root.
          (/trunk$ svn diff components/camel-ftp)
          File: CAMEL-1241_part2_camel-ftp.patch
          OR

          2b. For camel-ftp Generated at components/camel-ftp
          (/trunk/components$ svn diff camel-ftp)
          File: CAMEL-1241_part2_camel-ftp_relative_dir.patch

          Show
          Ramon Buckland added a comment - In this patch release ( CAMEL-1241 _part2.zip) is as follows Included now is the patch fixing CAMEL-1290 Patch files ============== The patch Files =============== 1. For camel-core (/trunk$ svn diff camel-core) File: CAMEL-1241 _part2_camel-core.patch === For 2, use either === Claus, you mentioned patching in through IDEA and I know eclipse roots projects at the pom.xml so I figured the patch is needed at that level (2b). Both are supplied. but have identical changes (of course) 2a. For camel-ftp Generated at the project root. (/trunk$ svn diff components/camel-ftp) File: CAMEL-1241 _part2_camel-ftp.patch OR 2b. For camel-ftp Generated at components/camel-ftp (/trunk/components$ svn diff camel-ftp) File: CAMEL-1241 _part2_camel-ftp_relative_dir.patch
          Hide
          Claus Ibsen added a comment -

          @Ramon

          Patch hell for camel-ftp.
          When you do delete/rename and all sort of stuff it gets messy.

          Can you send the src files in a .zip as plain .java files? Only for camel-ftp

          The camel-core is okay, only the AntPathMatcherGenericFileFilter is missing. Could you send it as well?

          Sorry but svn patches gets messy when there are to much add/delete/rename going on.
          And I guess the patch file should be ordered as well, I have a kinda deadlock where files patches look for a new class that does not exists since its being renamed and patched later in the file.

          Show
          Claus Ibsen added a comment - @Ramon Patch hell for camel-ftp . When you do delete/rename and all sort of stuff it gets messy. Can you send the src files in a .zip as plain .java files? Only for camel-ftp The camel-core is okay, only the AntPathMatcherGenericFileFilter is missing. Could you send it as well? Sorry but svn patches gets messy when there are to much add/delete/rename going on. And I guess the patch file should be ordered as well, I have a kinda deadlock where files patches look for a new class that does not exists since its being renamed and patched later in the file.
          Hide
          Claus Ibsen added a comment -

          Oh btw IDEA can do the relative/base stuff. There is a strip trailing directories you can adjust.

          Show
          Claus Ibsen added a comment - Oh btw IDEA can do the relative/base stuff. There is a strip trailing directories you can adjust.
          Hide
          Ramon Buckland added a comment -

          Yes.. patch hell

          Attached is all relevant new files or changes tar'd up.

          And below is the deletions needed

          svn rm components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/RemoteFileFilter.java
          svn rm components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/strategy/DeleteRemoteFileProcessStrategy.java
          svn rm components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/strategy/RemoteFileRenameExclusiveReadLockStrategy.java
          svn rm components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/strategy/RemoteFileExpressionRenamer.java
          svn rm components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/strategy/RemoteFileProcessStrategyFactory.java
          svn rm components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/strategy/RemoteFileRenamer.java
          svn rm components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/strategy/RenameRemoteFileProcessStrategy.java
          svn rm components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/strategy/NoOpRemoteFileProcessStrategy.java
          svn rm components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/strategy/DefaultRemoteFileRenamer.java
          svn rm components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/strategy/RemoteFileProcessStrategySupport.java
          svn rm components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/AntPathMatcherRemoteFileFilter.java

          Show
          Ramon Buckland added a comment - Yes.. patch hell Attached is all relevant new files or changes tar'd up. And below is the deletions needed svn rm components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/RemoteFileFilter.java svn rm components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/strategy/DeleteRemoteFileProcessStrategy.java svn rm components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/strategy/RemoteFileRenameExclusiveReadLockStrategy.java svn rm components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/strategy/RemoteFileExpressionRenamer.java svn rm components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/strategy/RemoteFileProcessStrategyFactory.java svn rm components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/strategy/RemoteFileRenamer.java svn rm components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/strategy/RenameRemoteFileProcessStrategy.java svn rm components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/strategy/NoOpRemoteFileProcessStrategy.java svn rm components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/strategy/DefaultRemoteFileRenamer.java svn rm components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/strategy/RemoteFileProcessStrategySupport.java svn rm components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/AntPathMatcherRemoteFileFilter.java
          Hide
          Chris Love added a comment -

          What work has been done on the vfs component? Any source for cifs writes or reads?

          Thanks in advance

          Show
          Chris Love added a comment - What work has been done on the vfs component? Any source for cifs writes or reads? Thanks in advance
          Hide
          Claus Ibsen added a comment -

          I do not think Ramon is working on this anymore.

          Anyone feel free to take over and implement a camel-vfs component

          Show
          Claus Ibsen added a comment - I do not think Ramon is working on this anymore. Anyone feel free to take over and implement a camel-vfs component
          Hide
          Claus Ibsen added a comment -

          commons-vfs is a dead project

          Show
          Claus Ibsen added a comment - commons-vfs is a dead project
          Hide
          Claus Ibsen added a comment -

          Closing all resolved tickets from 2010 or older

          Show
          Claus Ibsen added a comment - Closing all resolved tickets from 2010 or older

            People

            • Assignee:
              Unassigned
              Reporter:
              Ramon Buckland
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 12h
                12h

                  Development