Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.5.0
    • Component/s: start
    • Labels:
      None
    1. ACCUMULO-927-2.patch
      64 kB
      Dave Marion
    2. ACCUMULO-927-1.patch
      70 kB
      Dave Marion

      Activity

      Dave Marion created issue -
      Dave Marion made changes -
      Field Original Value New Value
      Fix Version/s 1.5.0 [ 12318645 ]
      Dave Marion made changes -
      Component/s start [ 12316603 ]
      Dave Marion made changes -
      Attachment ACCUMULO-927-1.patch [ 12563219 ]
      Hide
      Dave Marion added a comment -

      Attached patch that removes the ReadOnlyHdfs* VFS classes and replaces them with the accepted, but not yet released, classes from Commons VFS. I had to change them a little due to interface changes in VFS 2.1 which is not yet released. This change should allow us to remove these classes when Commons VFS 2.1 is released with no code change.

      I also removed the test case associated with the ReadOnlyHdfs* classes as the new classes have their own tests in Commons VFS and pass those tests. There was one behavior change with the recent changes to HdfsFileSystem.resolveFile(). It appears that when a file is deleted from HDFS the resolveFile() method still returns true instead of false because the file is still in the local VFS cache. I don't think this will present a problem as the fileChanged event still fires instead of the fileDeleted event. I don't think it will be common that folks delete jars from HDFS without restarting Accumulo as the classes that are loaded from that jar will still be resident in the VM.

      Show
      Dave Marion added a comment - Attached patch that removes the ReadOnlyHdfs* VFS classes and replaces them with the accepted, but not yet released, classes from Commons VFS. I had to change them a little due to interface changes in VFS 2.1 which is not yet released. This change should allow us to remove these classes when Commons VFS 2.1 is released with no code change. I also removed the test case associated with the ReadOnlyHdfs* classes as the new classes have their own tests in Commons VFS and pass those tests. There was one behavior change with the recent changes to HdfsFileSystem.resolveFile(). It appears that when a file is deleted from HDFS the resolveFile() method still returns true instead of false because the file is still in the local VFS cache. I don't think this will present a problem as the fileChanged event still fires instead of the fileDeleted event. I don't think it will be common that folks delete jars from HDFS without restarting Accumulo as the classes that are loaded from that jar will still be resident in the VM.
      Dave Marion made changes -
      Status Open [ 1 ] Patch Available [ 10002 ]
      Hide
      Josh Elser added a comment -

      What about a case in which I have my-code-1.0.jar deployed. I then find a problem in my code, cut a new version and deploy my-code-1.1.jar. Say in version 1.1, I have a class in which I added a new method that did not exist in 1.0. Wouldn't I have undefined behavior (my class in 1.0 or 1.1 could be used by Accumulo) if the 1.0 classes aren't removed completely?

      I don't think it will be common that folks delete jars from HDFS without restarting Accumulo as the classes that are loaded from that jar will still be resident in the VM.

      Show
      Josh Elser added a comment - What about a case in which I have my-code-1.0.jar deployed. I then find a problem in my code, cut a new version and deploy my-code-1.1.jar. Say in version 1.1, I have a class in which I added a new method that did not exist in 1.0. Wouldn't I have undefined behavior (my class in 1.0 or 1.1 could be used by Accumulo) if the 1.0 classes aren't removed completely? I don't think it will be common that folks delete jars from HDFS without restarting Accumulo as the classes that are loaded from that jar will still be resident in the VM.
      Hide
      Dave Marion added a comment -

      But your v1.0 class would still be loaded in the VM. I don't think there is a guarantee as to when classes get unloaded from the VM assuming that you have enabled those options. Your scenario would only work if the class was in a different package.

      You could however deploy the jar to a new location and set the table context to the new location.

      Sent from my Motorola ATRIX™ 4G on AT&T

      ----Original message----
      From: "Josh Elser (JIRA)" <jira@apache.org>
      To: notifications@accumulo.apache.org
      Sent: Fri, Jan 4, 2013 02:54:12 GMT+00:00
      Subject: [jira] [Commented] (ACCUMULO-927) Replace HDFS VFS FileSystem objects with the ones from Commons VFS

      [ https://issues.apache.org/jira/browse/ACCUMULO-927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13543557#comment-13543557 ]

      Josh Elser commented on ACCUMULO-927:
      -------------------------------------

      What about a case in which I have my-code-1.0.jar deployed. I then find a problem in my code, cut a new version and deploy my-code-1.1.jar. Say in version 1.1, I have a class in which I added a new method that did not exist in 1.0. Wouldn't I have undefined behavior (my class in 1.0 or 1.1 could be used by Accumulo) if the 1.0 classes aren't removed completely?

      I don't think it will be common that folks delete jars from HDFS without restarting Accumulo as the classes that are loaded from that jar will still be resident in the VM.


      This message is automatically generated by JIRA.
      If you think it was sent incorrectly, please contact your JIRA administrators
      For more information on JIRA, see: http://www.atlassian.com/software/jira

      Show
      Dave Marion added a comment - But your v1.0 class would still be loaded in the VM. I don't think there is a guarantee as to when classes get unloaded from the VM assuming that you have enabled those options. Your scenario would only work if the class was in a different package. You could however deploy the jar to a new location and set the table context to the new location. Sent from my Motorola ATRIX™ 4G on AT&T ---- Original message ---- From: "Josh Elser (JIRA)" <jira@apache.org> To: notifications@accumulo.apache.org Sent: Fri, Jan 4, 2013 02:54:12 GMT+00:00 Subject: [jira] [Commented] ( ACCUMULO-927 ) Replace HDFS VFS FileSystem objects with the ones from Commons VFS [ https://issues.apache.org/jira/browse/ACCUMULO-927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13543557#comment-13543557 ] Josh Elser commented on ACCUMULO-927 : ------------------------------------- What about a case in which I have my-code-1.0.jar deployed. I then find a problem in my code, cut a new version and deploy my-code-1.1.jar. Say in version 1.1, I have a class in which I added a new method that did not exist in 1.0. Wouldn't I have undefined behavior (my class in 1.0 or 1.1 could be used by Accumulo) if the 1.0 classes aren't removed completely? I don't think it will be common that folks delete jars from HDFS without restarting Accumulo as the classes that are loaded from that jar will still be resident in the VM. – This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
      Hide
      Dave Marion added a comment -

      I take that back. You raise a valid point. In any case of a change to the monitored directory, the context classloader is recreated. I will look into this more to see if the deleted jar is ever removed from the local cache directory.

      Sent from my Motorola ATRIX™ 4G on AT&T

      ----Original message----
      From: "Dave Marion (JIRA)" <jira@apache.org>
      To: notifications@accumulo.apache.org
      Sent: Fri, Jan 4, 2013 03:04:13 GMT+00:00
      Subject: [jira] [Commented] (ACCUMULO-927) Replace HDFS VFS FileSystem objects with the ones from Commons VFS

      [ https://issues.apache.org/jira/browse/ACCUMULO-927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13543564#comment-13543564 ]

      Dave Marion commented on ACCUMULO-927:
      --------------------------------------

      But your v1.0 class would still be loaded in the VM. I don't think there is a guarantee as to when classes get unloaded from the VM assuming that you have enabled those options. Your scenario would only work if the class was in a different package.

      You could however deploy the jar to a new location and set the table context to the new location.

      Sent from my Motorola ATRIX™ 4G on AT&T

      ----Original message----
      From: "Josh Elser (JIRA)" <jira@apache.org>
      To: notifications@accumulo.apache.org
      Sent: Fri, Jan 4, 2013 02:54:12 GMT+00:00
      Subject: [jira] [Commented] (ACCUMULO-927) Replace HDFS VFS FileSystem objects with the ones from Commons VFS

      [ https://issues.apache.org/jira/browse/ACCUMULO-927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13543557#comment-13543557 ]

      Josh Elser commented on ACCUMULO-927:
      -------------------------------------

      What about a case in which I have my-code-1.0.jar deployed. I then find a problem in my code, cut a new version and deploy my-code-1.1.jar. Say in version 1.1, I have a class in which I added a new method that did not exist in 1.0. Wouldn't I have undefined behavior (my class in 1.0 or 1.1 could be used by Accumulo) if the 1.0 classes aren't removed completely?

      I don't think it will be common that folks delete jars from HDFS without restarting Accumulo as the classes that are loaded from that jar will still be resident in the VM.


      This message is automatically generated by JIRA.
      If you think it was sent incorrectly, please contact your JIRA administrators
      For more information on JIRA, see: http://www.atlassian.com/software/jira


      This message is automatically generated by JIRA.
      If you think it was sent incorrectly, please contact your JIRA administrators
      For more information on JIRA, see: http://www.atlassian.com/software/jira

      Show
      Dave Marion added a comment - I take that back. You raise a valid point. In any case of a change to the monitored directory, the context classloader is recreated. I will look into this more to see if the deleted jar is ever removed from the local cache directory. Sent from my Motorola ATRIX™ 4G on AT&T ---- Original message ---- From: "Dave Marion (JIRA)" <jira@apache.org> To: notifications@accumulo.apache.org Sent: Fri, Jan 4, 2013 03:04:13 GMT+00:00 Subject: [jira] [Commented] ( ACCUMULO-927 ) Replace HDFS VFS FileSystem objects with the ones from Commons VFS [ https://issues.apache.org/jira/browse/ACCUMULO-927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13543564#comment-13543564 ] Dave Marion commented on ACCUMULO-927 : -------------------------------------- But your v1.0 class would still be loaded in the VM. I don't think there is a guarantee as to when classes get unloaded from the VM assuming that you have enabled those options. Your scenario would only work if the class was in a different package. You could however deploy the jar to a new location and set the table context to the new location. Sent from my Motorola ATRIX™ 4G on AT&T ---- Original message ---- From: "Josh Elser (JIRA)" <jira@apache.org> To: notifications@accumulo.apache.org Sent: Fri, Jan 4, 2013 02:54:12 GMT+00:00 Subject: [jira] [Commented] ( ACCUMULO-927 ) Replace HDFS VFS FileSystem objects with the ones from Commons VFS [ https://issues.apache.org/jira/browse/ACCUMULO-927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13543557#comment-13543557 ] Josh Elser commented on ACCUMULO-927 : ------------------------------------- What about a case in which I have my-code-1.0.jar deployed. I then find a problem in my code, cut a new version and deploy my-code-1.1.jar. Say in version 1.1, I have a class in which I added a new method that did not exist in 1.0. Wouldn't I have undefined behavior (my class in 1.0 or 1.1 could be used by Accumulo) if the 1.0 classes aren't removed completely? I don't think it will be common that folks delete jars from HDFS without restarting Accumulo as the classes that are loaded from that jar will still be resident in the VM. – This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira – This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
      Hide
      Dave Marion added a comment -

      Do not apply patch #1

      Show
      Dave Marion added a comment - Do not apply patch #1
      Dave Marion made changes -
      Status Patch Available [ 10002 ] Open [ 1 ]
      Dave Marion made changes -
      Attachment ACCUMULO-927-2.patch [ 12563431 ]
      Dave Marion made changes -
      Status Open [ 1 ] Patch Available [ 10002 ]
      Hide
      Dave Marion added a comment -

      Uploaded patch #2. I fixed the issue in the Commons VFS HDFS provider where the files in the local cache were not deleted when the file was deleted in HDFS. The associated ticket is VFS-449.

      Copied the classes from Commons VFS to start package and removed the Accumulo versions. Updated tests accordingly due to package and name changes. We should be able to remove these copies when a version of Commons VFS is released that includes VFS-442 and VFS-449.

      Show
      Dave Marion added a comment - Uploaded patch #2. I fixed the issue in the Commons VFS HDFS provider where the files in the local cache were not deleted when the file was deleted in HDFS. The associated ticket is VFS-449 . Copied the classes from Commons VFS to start package and removed the Accumulo versions. Updated tests accordingly due to package and name changes. We should be able to remove these copies when a version of Commons VFS is released that includes VFS-442 and VFS-449 .
      Hide
      Keith Turner added a comment -

      I applied this patch in revision 1431079. I changed the package name so that we do not use the commons package name in accumulo code. Also moved where refresh is called in resolve. Made it always call it instead of just when the file object is created.

      Show
      Keith Turner added a comment - I applied this patch in revision 1431079. I changed the package name so that we do not use the commons package name in accumulo code. Also moved where refresh is called in resolve. Made it always call it instead of just when the file object is created.
      Keith Turner made changes -
      Status Patch Available [ 10002 ] Resolved [ 5 ]
      Resolution Fixed [ 1 ]
      Hide
      Hudson added a comment -

      Integrated in Accumulo-Trunk #623 (See https://builds.apache.org/job/Accumulo-Trunk/623/)
      ACCUMULO-927 applied patch from Dave Marion with modifications (Revision 1431079)

      Result = SUCCESS
      kturner :
      Files :

      • /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/AccumuloVFSClassLoader.java
      • /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsFileAttributes.java
      • /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsFileContentInfoFactory.java
      • /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsFileObject.java
      • /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsFileProvider.java
      • /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsFileSystem.java
      • /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsFileSystemConfigBuilder.java
      • /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsRandomAccessContent.java
      • /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsReadOnlyFileContentInfoFactory.java
      • /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsReadOnlyRandomAccessContent.java
      • /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/ReadOnlyHdfsFileProvider.java
      • /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/ReadOnlyHdfsFileSystem.java
      • /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/package.html
      • /accumulo/trunk/start/src/main/java/org/apache/commons/vfs2
      • /accumulo/trunk/start/src/main/java/org/apache/commons/vfs2/provider
      • /accumulo/trunk/start/src/main/java/org/apache/commons/vfs2/provider/hdfs
      • /accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/providers/ReadOnlyHdfsFileProviderTest.java
      • /accumulo/trunk/start/src/test/java/org/apache/accumulo/test/AccumuloDFSBase.java
      Show
      Hudson added a comment - Integrated in Accumulo-Trunk #623 (See https://builds.apache.org/job/Accumulo-Trunk/623/ ) ACCUMULO-927 applied patch from Dave Marion with modifications (Revision 1431079) Result = SUCCESS kturner : Files : /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/AccumuloVFSClassLoader.java /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsFileAttributes.java /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsFileContentInfoFactory.java /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsFileObject.java /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsFileProvider.java /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsFileSystem.java /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsFileSystemConfigBuilder.java /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsRandomAccessContent.java /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsReadOnlyFileContentInfoFactory.java /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsReadOnlyRandomAccessContent.java /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/ReadOnlyHdfsFileProvider.java /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/ReadOnlyHdfsFileSystem.java /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/package.html /accumulo/trunk/start/src/main/java/org/apache/commons/vfs2 /accumulo/trunk/start/src/main/java/org/apache/commons/vfs2/provider /accumulo/trunk/start/src/main/java/org/apache/commons/vfs2/provider/hdfs /accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/providers/ReadOnlyHdfsFileProviderTest.java /accumulo/trunk/start/src/test/java/org/apache/accumulo/test/AccumuloDFSBase.java

        People

        • Assignee:
          Dave Marion
          Reporter:
          Dave Marion
        • Votes:
          0 Vote for this issue
          Watchers:
          5 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development