Accumulo
  1. Accumulo
  2. ACCUMULO-708

Modify ClassLoader to support different applications / multi-tenancy

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.5.0
    • Component/s: start
    • Labels:

      Description

      I'd like to expand the current classloader to support loading classes from HDFS and different application contexts. I'll be modifying the ticket as the idea matures.

      1. HelloWorld.jar
        1.0 kB
        Dave Marion
      2. ACCUMULO-708-3.patch
        140 kB
        Josh Elser
      3. ACCUMULO-708-2.patch
        140 kB
        Dave Marion
      4. ACCUMULO-708-1.patch
        139 kB
        Dave Marion

        Issue Links

          Activity

          Dave Marion created issue -
          Hide
          Dave Marion added a comment -

          Documenting thoughts: Seems that there are only two contexts in which this applies: table operations (in which there is no user) and client operations (in which there is a user perform scan or update). The current classloader (be default) uses the boot classloader with a child classloader reading from $ACCUMULO_HOME/lib and a grandchild classloader reading from $ACCUMULO_HOME/lib/ext.

          We would need a way for the classloader to discover either the table name or the current user when loadClass is called in the thread. This is currently the part that I am not sure of. Need input from others and to look at the code more.

          Show
          Dave Marion added a comment - Documenting thoughts: Seems that there are only two contexts in which this applies: table operations (in which there is no user) and client operations (in which there is a user perform scan or update). The current classloader (be default) uses the boot classloader with a child classloader reading from $ACCUMULO_HOME/lib and a grandchild classloader reading from $ACCUMULO_HOME/lib/ext. We would need a way for the classloader to discover either the table name or the current user when loadClass is called in the thread. This is currently the part that I am not sure of. Need input from others and to look at the code more.
          Hide
          Dave Marion added a comment -

          Marc Parisi said:

          I think we should have a scan time class loader context that enables fully
          hot swappable analysis and iteration. Security would need to be enforced,
          but if we did that we could simply set the jar ala hadoop map/reduce and
          have an iterator hot loaded in a different way for a different scan
          session.

          Show
          Dave Marion added a comment - Marc Parisi said: I think we should have a scan time class loader context that enables fully hot swappable analysis and iteration. Security would need to be enforced, but if we did that we could simply set the jar ala hadoop map/reduce and have an iterator hot loaded in a different way for a different scan session.
          Hide
          Dave Marion added a comment -

          So, I think that this makes sense iff scan sessions run in their own process. You can specify a custom classpath for map reduce because it spins up a new jvm for your map reduce tasks, its not a long running server process. I would be concerned doing that in the tablet server right now.

          Show
          Dave Marion added a comment - So, I think that this makes sense iff scan sessions run in their own process. You can specify a custom classpath for map reduce because it spins up a new jvm for your map reduce tasks, its not a long running server process. I would be concerned doing that in the tablet server right now.
          Christopher Tubbs made changes -
          Field Original Value New Value
          Affects Version/s 1.5.0 [ 12318645 ]
          Dave Marion logged work - 18/Sep/12 02:57
          • Time Spent:
            12h
             
            Currently working from https://github.com/dlmarion/ACCUMULO-708, in case anyone wants to review progress. I have created:
            - a read only HDFS provider for the Commons VFS library and unit tests for it.
            - a VFSClassLoader test to load a class from a jar file that is in HDFS.
            - added a test to try the monitor mechanism.
            - started on a new AccumuloClassLoader that will create a VFSClassLoader per context.

            Currently the unit tests require a running HDFS instance because I load a jar file into HDFS. You will need to modify the unit test to run it locally on your machine.

            I believe that this approach would allow the start package to be versioned separately from the rest of the components. The start package and its dependencies would run on the local file system, the rest of the jars could be in HDFS.

            Currently looking for some feedback as this would require that Accumulo classes be modified to call this classloader method with a context name. The context name could be something like 'SYSTEM' as a default, or the name of the user performing the operation. This would allow us to define the location of the classpath for that context.
          Dave Marion made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Dave Marion made changes -
          Remaining Estimate 24h [ 86400 ] 12h [ 43200 ]
          Time Spent 12h [ 43200 ]
          Worklog Id 14038 [ 14038 ]
          Dave Marion made changes -
          Status In Progress [ 3 ] Open [ 1 ]
          Dave Marion made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Dave Marion made changes -
          Status In Progress [ 3 ] Open [ 1 ]
          Dave Marion made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Dave Marion made changes -
          Status In Progress [ 3 ] Open [ 1 ]
          Dave Marion logged work - 20/Sep/12 02:29
          • Time Spent:
            3h
             
            worked on more tests and top level class loader. Believe I am ready to start reviewing.
          Dave Marion made changes -
          Remaining Estimate 12h [ 43200 ] 9h [ 32400 ]
          Time Spent 12h [ 43200 ] 15h [ 54000 ]
          Worklog Id 14039 [ 14039 ]
          Hide
          Dave Marion added a comment -

          I'm ready for folks to start reviewing what I have done. I have been working from https://github.com/dlmarion/ACCUMULO-708. One thing to note is that the unit tests are probably really functional tests as they require a running HDFS instance. To run the tests, you will need to modify the HDFS_URI variable in each test. Currently its using hdfs://localhost:8020/.

          Here is the design:

          The new classloader creates the following structure:

                         SystemClassLoader
                                |
                                |
                          URLClassLoader
                                |
                                |
                     AccumuloContextClassLoader  
                                |
                                |
              Map<String, AccumuloReloadingVFSClassLoader>
          
          

          The top two levels of the structure are the same as the current class loader. The differences are:

          AccumuloReloadingVFSClassLoader: This class contains an Apache Commons VFSClassLoader and implements FileListener to get callbacks from the Apache Commons VFS file system. The VFSClassLoader will load classes from a specific set of locations, and the AccumuloReloadingVFSClassLoader will recreate the VFSClassLoader member when a change is detected. This is similar to the current FileSystemAlterationMonitor mechanism in the current classloader.

          AccumuloContextClassLoader: This class maintains a map of context names to AccumuloReloadingVFSClassLoader objects. There is a default context, which is primed using the property default.context.classpath. Additional contexts can be named and the classpath for the context is user defined.

          This implementation uses Apache Commons VFS which allows the classpath to specify locations other than the local filesystem. I have created a read only hdfs file system provider to allow loading of classes from HDFS. I think this may allow the start package to be versioned separately from the rest of the Accumulo modules. The start package and its dependencies would need to be on the local filesystem, and the rest of the Accumulo distribution could be in HDFS (i.e. /accumulo/classpath).

          Show
          Dave Marion added a comment - I'm ready for folks to start reviewing what I have done. I have been working from https://github.com/dlmarion/ACCUMULO-708 . One thing to note is that the unit tests are probably really functional tests as they require a running HDFS instance. To run the tests, you will need to modify the HDFS_URI variable in each test. Currently its using hdfs://localhost:8020/. Here is the design: The new classloader creates the following structure: SystemClassLoader | | URLClassLoader | | AccumuloContextClassLoader | | Map<String, AccumuloReloadingVFSClassLoader> The top two levels of the structure are the same as the current class loader. The differences are: AccumuloReloadingVFSClassLoader: This class contains an Apache Commons VFSClassLoader and implements FileListener to get callbacks from the Apache Commons VFS file system. The VFSClassLoader will load classes from a specific set of locations, and the AccumuloReloadingVFSClassLoader will recreate the VFSClassLoader member when a change is detected. This is similar to the current FileSystemAlterationMonitor mechanism in the current classloader. AccumuloContextClassLoader: This class maintains a map of context names to AccumuloReloadingVFSClassLoader objects. There is a default context, which is primed using the property default.context.classpath. Additional contexts can be named and the classpath for the context is user defined. This implementation uses Apache Commons VFS which allows the classpath to specify locations other than the local filesystem. I have created a read only hdfs file system provider to allow loading of classes from HDFS. I think this may allow the start package to be versioned separately from the rest of the Accumulo modules. The start package and its dependencies would need to be on the local filesystem, and the rest of the Accumulo distribution could be in HDFS (i.e. /accumulo/classpath).
          Drew Farris made changes -
          Link This issue relates to ACCUMULO-350 [ ACCUMULO-350 ]
          Dave Marion logged work - 18/Oct/12 01:52
          • Time Spent:
            6h
             
            <No comment>
          Dave Marion made changes -
          Remaining Estimate 9h [ 32400 ] 3h [ 10800 ]
          Time Spent 15h [ 54000 ] 21h [ 75600 ]
          Worklog Id 14134 [ 14134 ]
          Hide
          Dave Marion added a comment -

          For those interested, I am done with unit tests. Will begin working on integration with Accumulo and local testing. Code is up at https://github.com/dlmarion/ACCUMULO-708 if you want to review.

          Show
          Dave Marion added a comment - For those interested, I am done with unit tests. Will begin working on integration with Accumulo and local testing. Code is up at https://github.com/dlmarion/ACCUMULO-708 if you want to review.
          Dave Marion logged work - 28/Oct/12 23:51
          • Time Spent:
            12h
             
            <No comment>
          Dave Marion made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Dave Marion made changes -
          Status In Progress [ 3 ] Open [ 1 ]
          Dave Marion made changes -
          Remaining Estimate 3h [ 10800 ] 0h [ 0 ]
          Time Spent 21h [ 75600 ] 33h [ 118800 ]
          Worklog Id 14234 [ 14234 ]
          Hide
          Dave Marion added a comment -

          Known issues:

          • MiniDFSCluster FS permissions issues in AccumuloDFSBase.
          • MiniDFSCluster sets up on 8020, may conflict with running HDFS
          • NetUtils.getInputStream bug, ACCUMULO-837
          • Commented out ACCUMULO_HOME env setting in maven-surefire-plugin. Will need to add this property to Hudson instead.
          Show
          Dave Marion added a comment - Known issues: MiniDFSCluster FS permissions issues in AccumuloDFSBase. MiniDFSCluster sets up on 8020, may conflict with running HDFS NetUtils.getInputStream bug, ACCUMULO-837 Commented out ACCUMULO_HOME env setting in maven-surefire-plugin. Will need to add this property to Hudson instead.
          Dave Marion made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Dave Marion made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Dave Marion added a comment -

          Known issues:

          • MiniDFSCluster FS permissions issues in AccumuloDFSBase.
          • MiniDFSCluster sets up on 8020, may conflict with running HDFS
          • NetUtils.getInputStream bug, ACCUMULO-837
          • Commented out ACCUMULO_HOME env setting in maven-surefire-plugin. Will need to add this property to Hudson instead.
          Show
          Dave Marion added a comment - Known issues: MiniDFSCluster FS permissions issues in AccumuloDFSBase. MiniDFSCluster sets up on 8020, may conflict with running HDFS NetUtils.getInputStream bug, ACCUMULO-837 Commented out ACCUMULO_HOME env setting in maven-surefire-plugin. Will need to add this property to Hudson instead.
          Dave Marion made changes -
          Attachment ACCUMULO-708-1.patch [ 12551142 ]
          Hide
          Dave Marion added a comment -

          Steps I used to test locally:

          mvn clean package -P assemble
          tar zxf <distro> -C $ACCUMULO_HOME
          hadoop fs -rmr /accumulo/classpath
          hadoop fs -mkdir /accumulo/classpath
          hadoop fs -moveFromLocal $ACCUMULO_HOME/lib/* /accumulo/classpath
          hadoop fs -copyToLocal /accumulo/classpath/commons-vfs2-2.0.jar $ACCUMULO_HOME/lib/.
          hadoop fs -rmr /accumulo/classpath/commons-vfs2-2.0.jar
          hadoop fs -copyToLocal /accumulo/classpath/accumulo-start-1.5.0-SNAPSHOT.jar $ACCUMULO_HOME/lib/.
          hadoop fs -rmr /accumulo/classpath/accumulo-start-1.5.0-SNAPSHOT.jar
          #
          # You will need to copy the following jar out of HDFS also if you plan to run the old classloader.
          #
          hadoop fs -copyToLocal /accumulo/classpath/commons-jci-fam-1.0.jar $ACCUMULO_HOME/lib/.
          hadoop fs -rmr /accumulo/classpath/commons-jci-fam-1.0.jar
          
          cp $ACCUMULO_HOME/conf/examples/512MB/native-standalone/* $ACCUMULO_HOME/conf/.
          $ACCUMULO_HOME/bin/accumulo classpath
          
          # Add to accumulo-site.xml
                  <!-- VFS ClassLoader Settings -->
                  <property>
                          <name>classloader.vfs.enabled</name>
                          <value>true</value>
                          <description>set to true to enable the vfs classloader. If not enabled (default=false), then the old classloader is used</description>
                  </property>
          
          $ACCUMULO_HOME/bin/accumulo classpath
          
          #Then, add the default context classpath
                  <property>
                          <name>classloader.vfs.context.classpath.system</name>
                          <value>hdfs://localhost:8020/accumulo/classpath</value>
                          <description>location of the jars for the default (system) context</description>
                  </property>
          
          $ACCUMULO_HOME/bin/accumulo classpath
          
          
          Show
          Dave Marion added a comment - Steps I used to test locally: mvn clean package -P assemble tar zxf <distro> -C $ACCUMULO_HOME hadoop fs -rmr /accumulo/classpath hadoop fs -mkdir /accumulo/classpath hadoop fs -moveFromLocal $ACCUMULO_HOME/lib/* /accumulo/classpath hadoop fs -copyToLocal /accumulo/classpath/commons-vfs2-2.0.jar $ACCUMULO_HOME/lib/. hadoop fs -rmr /accumulo/classpath/commons-vfs2-2.0.jar hadoop fs -copyToLocal /accumulo/classpath/accumulo-start-1.5.0-SNAPSHOT.jar $ACCUMULO_HOME/lib/. hadoop fs -rmr /accumulo/classpath/accumulo-start-1.5.0-SNAPSHOT.jar # # You will need to copy the following jar out of HDFS also if you plan to run the old classloader. # hadoop fs -copyToLocal /accumulo/classpath/commons-jci-fam-1.0.jar $ACCUMULO_HOME/lib/. hadoop fs -rmr /accumulo/classpath/commons-jci-fam-1.0.jar cp $ACCUMULO_HOME/conf/examples/512MB/native-standalone/* $ACCUMULO_HOME/conf/. $ACCUMULO_HOME/bin/accumulo classpath # Add to accumulo-site.xml <!-- VFS ClassLoader Settings --> <property> <name>classloader.vfs.enabled</name> <value>true</value> <description>set to true to enable the vfs classloader. If not enabled (default=false), then the old classloader is used</description> </property> $ACCUMULO_HOME/bin/accumulo classpath #Then, add the default context classpath <property> <name>classloader.vfs.context.classpath.system</name> <value>hdfs://localhost:8020/accumulo/classpath</value> <description>location of the jars for the default (system) context</description> </property> $ACCUMULO_HOME/bin/accumulo classpath
          Dave Marion made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Fix Version/s 1.5.0 [ 12318645 ]
          Hide
          Josh Elser added a comment -

          Cool stuff, Dave. I assume the usage of the HDFS path "/accumulo" is based on the default value for the "instance.dfs.dir" parameter?

          Also, have you thought about modifying the assemble profile to load lib correctly and then modify `accumulo init` to auto-load the default jars into HDFS? In other words, have you thought about automating the above process through the maven profile and Accumulo initialization given the "classloader.vfs.enabled" and "classloader.vfs.context.classpath.system" parameters in accumulo-site.xml?

          Show
          Josh Elser added a comment - Cool stuff, Dave. I assume the usage of the HDFS path "/accumulo" is based on the default value for the "instance.dfs.dir" parameter? Also, have you thought about modifying the assemble profile to load lib correctly and then modify `accumulo init` to auto-load the default jars into HDFS? In other words, have you thought about automating the above process through the maven profile and Accumulo initialization given the "classloader.vfs.enabled" and "classloader.vfs.context.classpath.system" parameters in accumulo-site.xml?
          Hide
          Dave Marion added a comment -


          Thanks Josh. I used /accumulo/classpath as an example, it made sense to me. In reality though, it could be any location as long as it can be read by the OS user. I did not modify the assembly; by default 'classloader.vfs.enabled' is false and in this case the AccumuloClassLoader will be used (not the new AccumuloVFSClassLoader). In this default case, the jars will need to remain where they are now. I figure that this new classloader will need to be tested thoroughly (I expect a few issues will arise); so I think it makes sense to keep it experimental for now. Assuming all of the tests pass, Accumulo will still need to be modified to take advantage of the different contexts.

          Show
          Dave Marion added a comment - Thanks Josh. I used /accumulo/classpath as an example, it made sense to me. In reality though, it could be any location as long as it can be read by the OS user. I did not modify the assembly; by default 'classloader.vfs.enabled' is false and in this case the AccumuloClassLoader will be used (not the new AccumuloVFSClassLoader). In this default case, the jars will need to remain where they are now. I figure that this new classloader will need to be tested thoroughly (I expect a few issues will arise); so I think it makes sense to keep it experimental for now. Assuming all of the tests pass, Accumulo will still need to be modified to take advantage of the different contexts.
          Hide
          Dave Marion added a comment -

          I contributed the HDFS provider to the Commons VFS project (VFS-442). Once accepted and released we should be able to remove this from our codebase.

          Show
          Dave Marion added a comment - I contributed the HDFS provider to the Commons VFS project ( VFS-442 ). Once accepted and released we should be able to remove this from our codebase.
          Hide
          Josh Elser added a comment -

          I used /accumulo/classpath as an example, it made sense to me

          Cool, I'm making the assumption that this will be useful to a large number of people, and thus thinking forward to how users would want to use this. Using `instance.dfs.dir`/

          {classpath,lib,whatever}

          would allow multiple separate instances to be run over the same HDFS instance which could be very useful. That should also "guarantee" you a unique HDFS directory.

          Show
          Josh Elser added a comment - I used /accumulo/classpath as an example, it made sense to me Cool, I'm making the assumption that this will be useful to a large number of people, and thus thinking forward to how users would want to use this. Using `instance.dfs.dir`/ {classpath,lib,whatever} would allow multiple separate instances to be run over the same HDFS instance which could be very useful. That should also "guarantee" you a unique HDFS directory.
          Hide
          Dave Marion added a comment -

          Using `instance.dfs.dir`/{classpath,lib,whatever} would allow multiple separate instances to be run over the same HDFS instance which could be very useful.

          This classloader would allow it, just like the previous one. I'm not sure I can think of any good reasons to run multiple instances on the same cluster.

          Show
          Dave Marion added a comment - Using `instance.dfs.dir`/{classpath,lib,whatever} would allow multiple separate instances to be run over the same HDFS instance which could be very useful. This classloader would allow it, just like the previous one. I'm not sure I can think of any good reasons to run multiple instances on the same cluster.
          Hide
          Keith Turner added a comment -

          I scanned through the patch trying to get a sense of it at the high level. Tomorrow I will do a more in depth review. Have a few comments so far.

          • use general prefix for new config properties. general denotes its a configuration for all services (i.e. master, tserver, etc)
          • needs documentation. How does a user know this feature exists and what they can do with it?
          • probably should not use org/apache/commons/vfs2 package. If the commons project defines classes with same name and user loads Accumulo and Commons jar chaos could ensue.
          • I noticed there were a few accumulo site files introduced. Are these for testing? Some of them configured walog dirs which is no longer nesc in 1.5.
          Show
          Keith Turner added a comment - I scanned through the patch trying to get a sense of it at the high level. Tomorrow I will do a more in depth review. Have a few comments so far. use general prefix for new config properties. general denotes its a configuration for all services (i.e. master, tserver, etc) needs documentation. How does a user know this feature exists and what they can do with it? probably should not use org/apache/commons/vfs2 package. If the commons project defines classes with same name and user loads Accumulo and Commons jar chaos could ensue. I noticed there were a few accumulo site files introduced. Are these for testing? Some of them configured walog dirs which is no longer nesc in 1.5.
          Hide
          Eric Newton added a comment -

          Actually WALog dirs are needed in 1.5 for upgrade. They aren't needed for very long, but that's why they are still supported.

          Show
          Eric Newton added a comment - Actually WALog dirs are needed in 1.5 for upgrade. They aren't needed for very long, but that's why they are still supported.
          Hide
          Dave Marion added a comment -

          •use general prefix for new config properties. general denotes its a configuration for all services (i.e. master, tserver, etc)

          I'll include in another patch, I'll wait for your other comments and roll all the changes up into a new patch.

          •needs documentation. How does a user know this feature exists and what they can do with it?

          Where should this be documented? User guide, readme, etc?

          •probably should not use org/apache/commons/vfs2 package. If the commons project defines classes with same name and user loads Accumulo and Commons jar chaos could ensue.

          Understood. I know for a fact that these classes don't collide with ones in Commons VFS. Code has also been submitted to VFS for inclusion in a
          future release (VFS-442).

          •I noticed there were a few accumulo site files introduced. Are these for testing? Some of them configured walog dirs which is no longer nesc in 1.5.

          These are there for tests. I believe I copied the 512MB example site file and modified for my tests. The walog property is not used and the test will not fail if the directory does not exist as the servers are not started during the tests.

          Show
          Dave Marion added a comment - •use general prefix for new config properties. general denotes its a configuration for all services (i.e. master, tserver, etc) I'll include in another patch, I'll wait for your other comments and roll all the changes up into a new patch. •needs documentation. How does a user know this feature exists and what they can do with it? Where should this be documented? User guide, readme, etc? •probably should not use org/apache/commons/vfs2 package. If the commons project defines classes with same name and user loads Accumulo and Commons jar chaos could ensue. Understood. I know for a fact that these classes don't collide with ones in Commons VFS. Code has also been submitted to VFS for inclusion in a future release ( VFS-442 ). •I noticed there were a few accumulo site files introduced. Are these for testing? Some of them configured walog dirs which is no longer nesc in 1.5. These are there for tests. I believe I copied the 512MB example site file and modified for my tests. The walog property is not used and the test will not fail if the directory does not exist as the servers are not started during the tests.
          Hide
          Keith Turner added a comment -

          I applied the patch and have been unable to get the unit test to run. Keep seeing something like the following

          Caused by: java.lang.NullPointerException
          	at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422)
          	at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
          	at org.apache.accumulo.test.AccumuloDFSBase.<clinit>(AccumuloDFSBase.java:66)
          	... 21 more
          

          Where should this be documented? User guide, readme, etc?

          Maybe add something to the users guide and add an example that shows off what can be done.

          Show
          Keith Turner added a comment - I applied the patch and have been unable to get the unit test to run. Keep seeing something like the following Caused by: java.lang.NullPointerException at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422) at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280) at org.apache.accumulo.test.AccumuloDFSBase.<clinit>(AccumuloDFSBase.java:66) ... 21 more Where should this be documented? User guide, readme, etc? Maybe add something to the users guide and add an example that shows off what can be done.
          Hide
          Dave Marion added a comment -

          I updated my copy and now the tests are failing for me also with a different error than what you are getting. Fun.

          Show
          Dave Marion added a comment - I updated my copy and now the tests are failing for me also with a different error than what you are getting. Fun.
          Hide
          Dave Marion added a comment -

          Found my problem, had my $ACCUMULO_HOME/conf/accumulo-site.xml configured to use the vfs classloader and Hadoop was not running. I needed to comment out the vfs classloader settings in my site file. I did not get the same error you did, build was successful. I'll look at the MiniDFSCluster source and see if I see something.

          Show
          Dave Marion added a comment - Found my problem, had my $ACCUMULO_HOME/conf/accumulo-site.xml configured to use the vfs classloader and Hadoop was not running. I needed to comment out the vfs classloader settings in my site file. I did not get the same error you did, build was successful. I'll look at the MiniDFSCluster source and see if I see something.
          Hide
          Dave Marion added a comment -

          Keith, are you getting other errors about directory permissions? see: http://search-hadoop.com/m/5Te0lu8g0c1/hbase+windows/v=plain

          Show
          Dave Marion added a comment - Keith, are you getting other errors about directory permissions? see: http://search-hadoop.com/m/5Te0lu8g0c1/hbase+windows/v=plain
          Hide
          Keith Turner added a comment -

          Still do not have the unit test running. It may be a configuration issue on my workstation, but its not the umask issue, still investigating. A few more comments on the code.

          • In AccumuloVFSClassLoader and AccumuloClassLoader the methods replaceEnvVars(), addUrl(), findAccumuloURLs(), etc. look very similar. Are these methods just copies? If not, what is the essence of the diffs?
          • Seems like unit test AccumuloContextClassLoaderTest would benefit from using two seprate jars. The same jar is loaded into two different context. Seems like it would be better to load two different jars with different classes. Then verify that each context contains only the expected classes and nothing else.
          • AccumuloReloadingVFSClassLoaderTest would also benefit from another jar. Test are needed for a new class coming into existence, a new version of an existing class coming into existence, and an existing class going out of existence.

          Moving forward on this we need to start thinking about having a table prop for configuring per table classloaders. If you would like help I can work on that after we get this checked in, just let me know.

          I was thinking of a scenario where we have the following two classloader contexts :

          • system class loader context : contains accumulo server and core jars and all deps in HDFS
          • table foo class loader context : contains user iterator jars in HDFS

          It seems like the two context above are siblings. Is this correct? Therefore the foo class loader will not have accumulo-core jars in itself or its parent class loaders. Will this cause problems when loading an iterator from the foo classloader?

          Show
          Keith Turner added a comment - Still do not have the unit test running. It may be a configuration issue on my workstation, but its not the umask issue, still investigating. A few more comments on the code. In AccumuloVFSClassLoader and AccumuloClassLoader the methods replaceEnvVars(), addUrl(), findAccumuloURLs(), etc. look very similar. Are these methods just copies? If not, what is the essence of the diffs? Seems like unit test AccumuloContextClassLoaderTest would benefit from using two seprate jars. The same jar is loaded into two different context. Seems like it would be better to load two different jars with different classes. Then verify that each context contains only the expected classes and nothing else. AccumuloReloadingVFSClassLoaderTest would also benefit from another jar. Test are needed for a new class coming into existence, a new version of an existing class coming into existence, and an existing class going out of existence. Moving forward on this we need to start thinking about having a table prop for configuring per table classloaders. If you would like help I can work on that after we get this checked in, just let me know. I was thinking of a scenario where we have the following two classloader contexts : system class loader context : contains accumulo server and core jars and all deps in HDFS table foo class loader context : contains user iterator jars in HDFS It seems like the two context above are siblings. Is this correct? Therefore the foo class loader will not have accumulo-core jars in itself or its parent class loaders. Will this cause problems when loading an iterator from the foo classloader?
          Hide
          Josh Elser added a comment -

          Dave, Keith,

          I did run into the same not-helpful Exception Keith posted above and traced it back to the same issue Dave linked to. After I realized AccumuloDFSBase.java:25 was sending the logging to /dev/null, I saw the actual error message, changed the default umask to 022 (755) as my system is configured and then was able to successfully run the tests via the command line.

          Show
          Josh Elser added a comment - Dave, Keith, I did run into the same not-helpful Exception Keith posted above and traced it back to the same issue Dave linked to. After I realized AccumuloDFSBase.java:25 was sending the logging to /dev/null, I saw the actual error message, changed the default umask to 022 (755) as my system is configured and then was able to successfully run the tests via the command line.
          Hide
          Keith Turner added a comment -

          I did run into the same not-helpful Exception Keith posted above and traced it back to the same issue Dave linked to. After I realized AccumuloDFSBase.java:25 was sending the logging to /dev/null

          I think my umask is correct

          $ umask
          0022
          

          AccumuloDFSBase.java line 25 is an import statement. Did you mean line 40 or line 49?

          Show
          Keith Turner added a comment - I did run into the same not-helpful Exception Keith posted above and traced it back to the same issue Dave linked to. After I realized AccumuloDFSBase.java:25 was sending the logging to /dev/null I think my umask is correct $ umask 0022 AccumuloDFSBase.java line 25 is an import statement. Did you mean line 40 or line 49?
          Hide
          Josh Elser added a comment -

          AccumuloDFSBase.java line 25 is an import statement. Did you mean line 40 or line 49?

          Uhh, my code still says line 25. The actual code I commented out was the following:

            //Turn off the Jetty and MiniDFSCluster logging
            static {
              System.setProperty("org.apache.commons.logging.Log", "org.apache.commons.logging.impl.NoOpLog");
            }
          

          If your umask is 022, then you need to change the code from

          conf.set("dfs.datanode.data.dir.perm", "775");
          

          to

          conf.set("dfs.datanode.data.dir.perm", "755");
          
          Show
          Josh Elser added a comment - AccumuloDFSBase.java line 25 is an import statement. Did you mean line 40 or line 49? Uhh, my code still says line 25. The actual code I commented out was the following: //Turn off the Jetty and MiniDFSCluster logging static { System.setProperty("org.apache.commons.logging.Log", "org.apache.commons.logging.impl.NoOpLog"); } If your umask is 022, then you need to change the code from conf.set("dfs.datanode.data.dir.perm", "775"); to conf.set("dfs.datanode.data.dir.perm", "755");
          Hide
          Josh Elser added a comment -

          Moving forward on this we need to start thinking about having a table prop for configuring per table classloaders.

          I was thinking more along the lines that a (Batch)Scanner could actually be tied to a ClassLoader. I could potentially have one table in which some standard query iterators are used for scanning, but I also want to allow other developers to load/use their own code to scan that same table without messing up my production query code. That most certainly complicates things, but perhaps there is an elegant way around not that without mucking with the existing API too much.

          You still need some default ClassLoader like you pointed out, Keith, which provides the accumulo-core and hadoop-core dependencies. Looking back at the older comments, perhaps tying ClassLoaders to authenticated users would be easiest? I'd have to defer to Keith's larger understanding of the code as well.

          Dave, I still owe you a look at the code/integration more in-depth; I didn't get far past the test cases running successfully.

          Show
          Josh Elser added a comment - Moving forward on this we need to start thinking about having a table prop for configuring per table classloaders. I was thinking more along the lines that a (Batch)Scanner could actually be tied to a ClassLoader. I could potentially have one table in which some standard query iterators are used for scanning, but I also want to allow other developers to load/use their own code to scan that same table without messing up my production query code. That most certainly complicates things, but perhaps there is an elegant way around not that without mucking with the existing API too much. You still need some default ClassLoader like you pointed out, Keith, which provides the accumulo-core and hadoop-core dependencies. Looking back at the older comments, perhaps tying ClassLoaders to authenticated users would be easiest? I'd have to defer to Keith's larger understanding of the code as well. Dave, I still owe you a look at the code/integration more in-depth; I didn't get far past the test cases running successfully.
          Hide
          Dave Marion added a comment -

          Josh,

          Please back out your change of setting dfs.datanode.data.dir.perm to 755. I believe this issue is fixed in HDFS-1560. Instead, please change the hadoop-test dependency in start/pom.xml to 1.1.0. I talked with Eric about changing the hadoop dependency from 0.20.205 to 1.1.0 for Accumulo 1.5. I talked with Keith yesterday and I believe he is past the directory permission issue, he is now running into another issue that has to do with the configuration of his computer.

          Keith,

          •In AccumuloVFSClassLoader and AccumuloClassLoader the methods replaceEnvVars(), addUrl(), findAccumuloURLs(), etc. look very similar. Are these methods just copies? If not, what is the essence of the diffs?

          If I remember correctly, they are copies of methods in AccumuloClassLoader. I did not want to extend AccumuloClassLoader as I do not know its lifetime.

          •Seems like unit test AccumuloContextClassLoaderTest would benefit from using two seprate jars. The same jar is loaded into two different context. Seems like it would be better to load two different jars with different classes. Then verify that each context contains only the expected classes and nothing else.

          I was testing that the same jar/class was in fact being loaded by two different classloaders, and were therefore different classes. I will add your test suggestions to my to-do list.

          All,

          I believe that the discussion around which contexts are supported is a separate discussion. I can think of several different scenarios in which contexts could be defined: per user, per table, per scan session, per minc/majc session, per application, etc. Regarding the heirarchy of the context classloader, I can see Keiths point above, and would also counter with a different scenario. Say for example that I wanted to run different versions (api compatible of course) of the Accumulo iterators on my table. The current setup would allow me to run a 1.5.0 tablet server and 1.5.1 iterators on my table. I can do this because the context classloader for my application is not a child of the system classloader. Changing the context classloader to be a child of the system classloader is a very small change.

          Show
          Dave Marion added a comment - Josh, Please back out your change of setting dfs.datanode.data.dir.perm to 755. I believe this issue is fixed in HDFS-1560 . Instead, please change the hadoop-test dependency in start/pom.xml to 1.1.0. I talked with Eric about changing the hadoop dependency from 0.20.205 to 1.1.0 for Accumulo 1.5. I talked with Keith yesterday and I believe he is past the directory permission issue, he is now running into another issue that has to do with the configuration of his computer. Keith, •In AccumuloVFSClassLoader and AccumuloClassLoader the methods replaceEnvVars(), addUrl(), findAccumuloURLs(), etc. look very similar. Are these methods just copies? If not, what is the essence of the diffs? If I remember correctly, they are copies of methods in AccumuloClassLoader. I did not want to extend AccumuloClassLoader as I do not know its lifetime. •Seems like unit test AccumuloContextClassLoaderTest would benefit from using two seprate jars. The same jar is loaded into two different context. Seems like it would be better to load two different jars with different classes. Then verify that each context contains only the expected classes and nothing else. I was testing that the same jar/class was in fact being loaded by two different classloaders, and were therefore different classes. I will add your test suggestions to my to-do list. All, I believe that the discussion around which contexts are supported is a separate discussion. I can think of several different scenarios in which contexts could be defined: per user, per table, per scan session, per minc/majc session, per application, etc. Regarding the heirarchy of the context classloader, I can see Keiths point above, and would also counter with a different scenario. Say for example that I wanted to run different versions (api compatible of course) of the Accumulo iterators on my table. The current setup would allow me to run a 1.5.0 tablet server and 1.5.1 iterators on my table. I can do this because the context classloader for my application is not a child of the system classloader. Changing the context classloader to be a child of the system classloader is a very small change.
          Hide
          Keith Turner added a comment -

          I set dfs.datanode.data.dir.perm to 755 and the test got alot further. I had set my maven deps to 1.1.0, so that does not seem to fix the issue. Its now failing because HelloWorld.jar resource is missing.

          Show
          Keith Turner added a comment - I set dfs.datanode.data.dir.perm to 755 and the test got alot further. I had set my maven deps to 1.1.0, so that does not seem to fix the issue. Its now failing because HelloWorld.jar resource is missing.
          Hide
          Keith Turner added a comment -

          I modified the commented out code that was trying to set the persmissions and I think I have that working. The existing code was dealing with base 10 numbers instead of base 8. In java prefixing a number with 0 makes it base 8, so it needs to be 0777 instead of 777. Also need to format the result as base 8 instead of base 10. Could have changed Intereger.toString(newPermission) to Integer.toString(newPermission, 8) but I used string.format instead.

              //HDFS-1560
              //Need to set permission to 777 xor umask
              FsPermission fsp = FsPermission.getUMask(conf);
              short umask = fsp.toShort();
              // leading zero makes java interpret as base 8
              int newPermission = 0777 ^ umask;
              conf.set("dfs.datanode.data.dir.perm", String.format("%03o", newPermission));
          
          Show
          Keith Turner added a comment - I modified the commented out code that was trying to set the persmissions and I think I have that working. The existing code was dealing with base 10 numbers instead of base 8. In java prefixing a number with 0 makes it base 8, so it needs to be 0777 instead of 777. Also need to format the result as base 8 instead of base 10. Could have changed Intereger.toString(newPermission) to Integer.toString(newPermission, 8) but I used string.format instead. //HDFS-1560 //Need to set permission to 777 xor umask FsPermission fsp = FsPermission.getUMask(conf); short umask = fsp.toShort(); // leading zero makes java interpret as base 8 int newPermission = 0777 ^ umask; conf.set( "dfs.datanode.data.dir.perm" , String .format( "%03o" , newPermission));
          Hide
          Dave Marion added a comment -

          Awesome! I must have forgot to add binaries to the patch. I'll try and upload tonight

          Show
          Dave Marion added a comment - Awesome! I must have forgot to add binaries to the patch. I'll try and upload tonight
          Hide
          Dave Marion added a comment -

          Unbelievable! I applied your changes and now I am getting the exact same error you were getting before (NPE on line 422 above).

          Show
          Dave Marion added a comment - Unbelievable! I applied your changes and now I am getting the exact same error you were getting before (NPE on line 422 above).
          Hide
          Keith Turner added a comment -

          Whats dfs.datanode.data.dir.perm being set to and what is your umask?

          Show
          Keith Turner added a comment - Whats dfs.datanode.data.dir.perm being set to and what is your umask?
          Hide
          Dave Marion added a comment -

          umask on cmd prompt yields: 0002
          umask in code above is 022
          newPermission in 755.

          Show
          Dave Marion added a comment - umask on cmd prompt yields: 0002 umask in code above is 022 newPermission in 755.
          Hide
          Dave Marion added a comment -

          Ah, FsPermission.getUMask() doesn't get the umask from the operating system, it gets it from the configuration file, doh.

          Show
          Dave Marion added a comment - Ah, FsPermission.getUMask() doesn't get the umask from the operating system, it gets it from the configuration file, doh.
          Hide
          Dave Marion added a comment -

          Fixed directory permission issue in AccumuloDFSBase.
          Set system property so that the MiniDFSCluster data directories are inside the target directory.
          Removed the maven-clean-plugin override in start/pom.xml

          Show
          Dave Marion added a comment - Fixed directory permission issue in AccumuloDFSBase. Set system property so that the MiniDFSCluster data directories are inside the target directory. Removed the maven-clean-plugin override in start/pom.xml
          Dave Marion made changes -
          Attachment ACCUMULO-708-2.patch [ 12553892 ]
          Hide
          Dave Marion added a comment -

          Could not upload the jar in the patch. It belongs in start/src/test/resources

          Show
          Dave Marion added a comment - Could not upload the jar in the patch. It belongs in start/src/test/resources
          Dave Marion made changes -
          Attachment HelloWorld.jar [ 12553893 ]
          Hide
          Josh Elser added a comment -

          Dave,

          An issue with your most recent patch:

          Short umask = Short.parseShort(line.trim());
          

          As Keith pointed out, we need base 8, but the default radix used by parseShort is base 10. Thus, I get the following incorrectly calculated umask set in the config.

          umask response: 0022
          Umask is: 026
          Perm is: 751
          

          Changing that Short.parseShort method call creates the expected output and the tests then pass.

          Short umask = Short.parseShort(line.trim(), 8);
          

          Output:

          umask response: 0022
          Umask is: 022
          Perm is: 755
          
          Show
          Josh Elser added a comment - Dave, An issue with your most recent patch: Short umask = Short.parseShort(line.trim()); As Keith pointed out, we need base 8, but the default radix used by parseShort is base 10. Thus, I get the following incorrectly calculated umask set in the config. umask response: 0022 Umask is: 026 Perm is: 751 Changing that Short.parseShort method call creates the expected output and the tests then pass. Short umask = Short.parseShort(line.trim(), 8); Output: umask response: 0022 Umask is: 022 Perm is: 755
          Hide
          Josh Elser added a comment -

          Rolling in the trival change from my last comment into a new patch.

          Show
          Josh Elser added a comment - Rolling in the trival change from my last comment into a new patch.
          Josh Elser made changes -
          Attachment ACCUMULO-708-3.patch [ 12554287 ]
          Hide
          Keith Turner added a comment -

          I tested the latest patch. Had to add back setting ACCUMULO_HOME in root pom inorder to get it to run. I also tested running Accumulo out of HDFS and that worked beautifully.

          Show
          Keith Turner added a comment - I tested the latest patch. Had to add back setting ACCUMULO_HOME in root pom inorder to get it to run. I also tested running Accumulo out of HDFS and that worked beautifully.
          Hide
          Dave Marion added a comment -

          Keith,

          I don't think that works. I checked out a clean copy of TRUNK, applied the patch, and uncommented the ACCUMULO_HOME system env setting in the pom.xml file. A couple of tests fail looking for the conf/accumulo-site.xml file. The conf/accumulo-site.xml file doesn't exist on a clean checkout.

          Dave

          Show
          Dave Marion added a comment - Keith, I don't think that works. I checked out a clean copy of TRUNK, applied the patch, and uncommented the ACCUMULO_HOME system env setting in the pom.xml file. A couple of tests fail looking for the conf/accumulo-site.xml file. The conf/accumulo-site.xml file doesn't exist on a clean checkout. Dave
          Hide
          Dave Marion added a comment -

          I think I'll have to set up the AccumuloVFSClassLoader in the failing tests using PowerMock like I did in my tests.

          Show
          Dave Marion added a comment - I think I'll have to set up the AccumuloVFSClassLoader in the failing tests using PowerMock like I did in my tests.
          Hide
          Keith Turner added a comment -

          I don't think that works. I checked out a clean copy of TRUNK, applied the patch, and uncommented the ACCUMULO_HOME system env setting in the pom.xml file. A couple of tests fail looking for the conf/accumulo-site.xml file. The conf/accumulo-site.xml file doesn't exist on a clean checkout.

          Good catch. If ACCUMULO_HOME is only needed by a unit test for the old classloader, and we are going to get rid of the old class loader, then maybe you could just drop that unit test before checking the code in. Just need to make sure that what that old unit test does is also done by the new unit test.

          Show
          Keith Turner added a comment - I don't think that works. I checked out a clean copy of TRUNK, applied the patch, and uncommented the ACCUMULO_HOME system env setting in the pom.xml file. A couple of tests fail looking for the conf/accumulo-site.xml file. The conf/accumulo-site.xml file doesn't exist on a clean checkout. Good catch. If ACCUMULO_HOME is only needed by a unit test for the old classloader, and we are going to get rid of the old class loader, then maybe you could just drop that unit test before checking the code in. Just need to make sure that what that old unit test does is also done by the new unit test.
          Hide
          Dave Marion added a comment -

          Here's my solution:

          I copied an example site file to the test module:

           cp conf/examples/512MB/standalone/accumulo-site.xml test/src/test/resources/conf/.
          

          Then I set the property to:

                       <ACCUMULO_HOME>../test/src/test/resources</ACCUMULO_HOME>
          
          Show
          Dave Marion added a comment - Here's my solution: I copied an example site file to the test module: cp conf/examples/512MB/standalone/accumulo-site.xml test/src/test/resources/conf/. Then I set the property to: <ACCUMULO_HOME>../test/src/test/resources</ACCUMULO_HOME>
          Hide
          Dave Marion added a comment -

          The above works and I believe it should only affect tests. I can put up another patch if you want.

          Show
          Dave Marion added a comment - The above works and I believe it should only affect tests. I can put up another patch if you want.
          Hide
          Keith Turner added a comment -

          The above works and I believe it should only affect tests. I can put up another patch if you want.

          If mvn package runs on a clean checkout, I think you should check it in.

          Show
          Keith Turner added a comment - The above works and I believe it should only affect tests. I can put up another patch if you want. If mvn package runs on a clean checkout, I think you should check it in.
          Hide
          Dave Marion added a comment -

          Done. You should be able to update and build.

          Show
          Dave Marion added a comment - Done. You should be able to update and build.
          Hide
          Hudson added a comment -

          Integrated in Accumulo-Trunk #562 (See https://builds.apache.org/job/Accumulo-Trunk/562/)
          ACCUMULO-708 initial implementation of VFS class loader (Revision 1412398)

          Result = SUCCESS
          dlmarion :
          Files :

          • /accumulo/trunk/bin/accumulo
          • /accumulo/trunk/conf/examples/vfs-classloader
          • /accumulo/trunk/conf/examples/vfs-classloader/accumulo-site.xml
          • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/client/mock/MockInstanceOperations.java
          • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/conf/Property.java
          • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/file/BloomFilterLayer.java
          • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/iterators/AggregatingIterator.java
          • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/iterators/IteratorUtil.java
          • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/iterators/TypedValueCombiner.java
          • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/iterators/conf/ColumnToClassMapping.java
          • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/util/shell/commands/ClasspathCommand.java
          • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/util/shell/commands/ScanCommand.java
          • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/util/shell/commands/SetIterCommand.java
          • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/util/shell/commands/ShellPluginConfigurationCommand.java
          • /accumulo/trunk/core/src/test/resources/disabled
          • /accumulo/trunk/core/src/test/resources/disabled/conf
          • /accumulo/trunk/core/src/test/resources/disabled/conf/accumulo-site.xml
          • /accumulo/trunk/pom.xml
          • /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/client/ClientServiceHandler.java
          • /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/constraints/ConstraintLoader.java
          • /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/master/Master.java
          • /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/master/balancer/TableLoadBalancer.java
          • /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/TabletServerResourceManager.java
          • /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/test/functional/FunctionalTest.java
          • /accumulo/trunk/start/pom.xml
          • /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/Main.java
          • /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs
          • /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/AccumuloContextClassLoader.java
          • /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/AccumuloReloadingVFSClassLoader.java
          • /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/AccumuloVFSClassLoader.java
          • /accumulo/trunk/start/src/main/java/org/apache/commons
          • /accumulo/trunk/start/src/main/java/org/apache/commons/vfs2
          • /accumulo/trunk/start/src/main/java/org/apache/commons/vfs2/provider
          • /accumulo/trunk/start/src/main/java/org/apache/commons/vfs2/provider/HdfsFileObject.java
          • /accumulo/trunk/start/src/main/java/org/apache/commons/vfs2/provider/HdfsFileSystemConfigBuilder.java
          • /accumulo/trunk/start/src/main/java/org/apache/commons/vfs2/provider/HdfsReadOnlyFileContentInfoFactory.java
          • /accumulo/trunk/start/src/main/java/org/apache/commons/vfs2/provider/HdfsReadOnlyRandomAccessContent.java
          • /accumulo/trunk/start/src/main/java/org/apache/commons/vfs2/provider/ReadOnlyHdfsFileProvider.java
          • /accumulo/trunk/start/src/main/java/org/apache/commons/vfs2/provider/ReadOnlyHdfsFileSystem.java
          • /accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader
          • /accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs
          • /accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/AccumuloContextClassLoaderTest.java
          • /accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/AccumuloReloadingVFSClassLoaderTest.java
          • /accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/AccumuloVFSClassLoaderTest.java
          • /accumulo/trunk/start/src/test/java/org/apache/accumulo/test
          • /accumulo/trunk/start/src/test/java/org/apache/accumulo/test/AccumuloDFSBase.java
          • /accumulo/trunk/start/src/test/java/org/apache/commons
          • /accumulo/trunk/start/src/test/java/org/apache/commons/vfs2
          • /accumulo/trunk/start/src/test/java/org/apache/commons/vfs2/provider
          • /accumulo/trunk/start/src/test/java/org/apache/commons/vfs2/provider/ReadOnlyHdfsFileProviderTest.java
          • /accumulo/trunk/start/src/test/java/org/apache/commons/vfs2/provider/VfsClassLoaderTest.java
          • /accumulo/trunk/start/src/test/resources/HelloWorld.jar
          • /accumulo/trunk/start/src/test/resources/application1
          • /accumulo/trunk/start/src/test/resources/application1/conf
          • /accumulo/trunk/start/src/test/resources/application1/conf/accumulo-site.xml
          • /accumulo/trunk/start/src/test/resources/default
          • /accumulo/trunk/start/src/test/resources/default/conf
          • /accumulo/trunk/start/src/test/resources/default/conf/accumulo-site.xml
          • /accumulo/trunk/start/src/test/resources/disabled
          • /accumulo/trunk/start/src/test/resources/disabled/conf
          • /accumulo/trunk/start/src/test/resources/disabled/conf/accumulo-site.xml
          • /accumulo/trunk/test/src/test/resources
          • /accumulo/trunk/test/src/test/resources/conf
          • /accumulo/trunk/test/src/test/resources/conf/accumulo-site.xml
          • /accumulo/trunk/test/src/test/resources/lib
          • /accumulo/trunk/test/src/test/resources/lib/ext
          Show
          Hudson added a comment - Integrated in Accumulo-Trunk #562 (See https://builds.apache.org/job/Accumulo-Trunk/562/ ) ACCUMULO-708 initial implementation of VFS class loader (Revision 1412398) Result = SUCCESS dlmarion : Files : /accumulo/trunk/bin/accumulo /accumulo/trunk/conf/examples/vfs-classloader /accumulo/trunk/conf/examples/vfs-classloader/accumulo-site.xml /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/client/mock/MockInstanceOperations.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/conf/Property.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/file/BloomFilterLayer.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/iterators/AggregatingIterator.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/iterators/IteratorUtil.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/iterators/TypedValueCombiner.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/iterators/conf/ColumnToClassMapping.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/util/shell/commands/ClasspathCommand.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/util/shell/commands/ScanCommand.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/util/shell/commands/SetIterCommand.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/util/shell/commands/ShellPluginConfigurationCommand.java /accumulo/trunk/core/src/test/resources/disabled /accumulo/trunk/core/src/test/resources/disabled/conf /accumulo/trunk/core/src/test/resources/disabled/conf/accumulo-site.xml /accumulo/trunk/pom.xml /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/client/ClientServiceHandler.java /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/constraints/ConstraintLoader.java /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/master/Master.java /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/master/balancer/TableLoadBalancer.java /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/TabletServerResourceManager.java /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/test/functional/FunctionalTest.java /accumulo/trunk/start/pom.xml /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/Main.java /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/AccumuloContextClassLoader.java /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/AccumuloReloadingVFSClassLoader.java /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/AccumuloVFSClassLoader.java /accumulo/trunk/start/src/main/java/org/apache/commons /accumulo/trunk/start/src/main/java/org/apache/commons/vfs2 /accumulo/trunk/start/src/main/java/org/apache/commons/vfs2/provider /accumulo/trunk/start/src/main/java/org/apache/commons/vfs2/provider/HdfsFileObject.java /accumulo/trunk/start/src/main/java/org/apache/commons/vfs2/provider/HdfsFileSystemConfigBuilder.java /accumulo/trunk/start/src/main/java/org/apache/commons/vfs2/provider/HdfsReadOnlyFileContentInfoFactory.java /accumulo/trunk/start/src/main/java/org/apache/commons/vfs2/provider/HdfsReadOnlyRandomAccessContent.java /accumulo/trunk/start/src/main/java/org/apache/commons/vfs2/provider/ReadOnlyHdfsFileProvider.java /accumulo/trunk/start/src/main/java/org/apache/commons/vfs2/provider/ReadOnlyHdfsFileSystem.java /accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader /accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs /accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/AccumuloContextClassLoaderTest.java /accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/AccumuloReloadingVFSClassLoaderTest.java /accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/AccumuloVFSClassLoaderTest.java /accumulo/trunk/start/src/test/java/org/apache/accumulo/test /accumulo/trunk/start/src/test/java/org/apache/accumulo/test/AccumuloDFSBase.java /accumulo/trunk/start/src/test/java/org/apache/commons /accumulo/trunk/start/src/test/java/org/apache/commons/vfs2 /accumulo/trunk/start/src/test/java/org/apache/commons/vfs2/provider /accumulo/trunk/start/src/test/java/org/apache/commons/vfs2/provider/ReadOnlyHdfsFileProviderTest.java /accumulo/trunk/start/src/test/java/org/apache/commons/vfs2/provider/VfsClassLoaderTest.java /accumulo/trunk/start/src/test/resources/HelloWorld.jar /accumulo/trunk/start/src/test/resources/application1 /accumulo/trunk/start/src/test/resources/application1/conf /accumulo/trunk/start/src/test/resources/application1/conf/accumulo-site.xml /accumulo/trunk/start/src/test/resources/default /accumulo/trunk/start/src/test/resources/default/conf /accumulo/trunk/start/src/test/resources/default/conf/accumulo-site.xml /accumulo/trunk/start/src/test/resources/disabled /accumulo/trunk/start/src/test/resources/disabled/conf /accumulo/trunk/start/src/test/resources/disabled/conf/accumulo-site.xml /accumulo/trunk/test/src/test/resources /accumulo/trunk/test/src/test/resources/conf /accumulo/trunk/test/src/test/resources/conf/accumulo-site.xml /accumulo/trunk/test/src/test/resources/lib /accumulo/trunk/test/src/test/resources/lib/ext
          John Vines made changes -
          Link This issue is related too ACCUMULO-926 [ ACCUMULO-926 ]
          Hide
          Hudson added a comment -

          Integrated in Accumulo-Trunk #603 (See https://builds.apache.org/job/Accumulo-Trunk/603/)
          ACCUMULO-708 ACCUMULO-866 ACCUMULO-869 ACCUMULO-870 many changes to accumulo-start (Revision 1428556)

          Result = SUCCESS
          kturner :
          Files :

          • /accumulo/trunk/bin/accumulo
          • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/conf/Property.java
          • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/master/thrift/TabletServerStatus.java
          • /accumulo/trunk/pom.xml
          • /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/TabletServer.java
          • /accumulo/trunk/start/pom.xml
          • /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/Main.java
          • /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/AccumuloClassLoader.java
          • /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/AccumuloFilesystemAlterationMonitor.java
          • /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/AccumuloReloadingVFSClassLoader.java
          • /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/AccumuloVFSClassLoader.java
          • /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/ContextManager.java
          • /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers
          • /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsFileObject.java
          • /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsFileSystemConfigBuilder.java
          • /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsReadOnlyFileContentInfoFactory.java
          • /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsReadOnlyRandomAccessContent.java
          • /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/ReadOnlyHdfsFileProvider.java
          • /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/ReadOnlyHdfsFileSystem.java
          • /accumulo/trunk/start/src/main/java/org/apache/commons/vfs2
          • /accumulo/trunk/start/src/test/java/org/apache/accumulo/start/Test.java
          • /accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/AccumuloReloadingVFSClassLoaderTest.java
          • /accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/AccumuloVFSClassLoaderTest.java
          • /accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/ContextManagerTest.java
          • /accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/providers
          • /accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/providers/ReadOnlyHdfsFileProviderTest.java
          • /accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/providers/VfsClassLoaderTest.java
          • /accumulo/trunk/start/src/test/java/org/apache/accumulo/test/AccumuloDFSBase.java
          • /accumulo/trunk/start/src/test/java/org/apache/commons/vfs2
          • /accumulo/trunk/start/src/test/resources/application1
          • /accumulo/trunk/start/src/test/resources/default/conf/accumulo-site.xml
          • /accumulo/trunk/start/src/test/resources/disabled/conf/accumulo-site.xml
          Show
          Hudson added a comment - Integrated in Accumulo-Trunk #603 (See https://builds.apache.org/job/Accumulo-Trunk/603/ ) ACCUMULO-708 ACCUMULO-866 ACCUMULO-869 ACCUMULO-870 many changes to accumulo-start (Revision 1428556) Result = SUCCESS kturner : Files : /accumulo/trunk/bin/accumulo /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/conf/Property.java /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/master/thrift/TabletServerStatus.java /accumulo/trunk/pom.xml /accumulo/trunk/server/src/main/java/org/apache/accumulo/server/tabletserver/TabletServer.java /accumulo/trunk/start/pom.xml /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/Main.java /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/AccumuloClassLoader.java /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/AccumuloFilesystemAlterationMonitor.java /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/AccumuloReloadingVFSClassLoader.java /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/AccumuloVFSClassLoader.java /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/ContextManager.java /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsFileObject.java /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsFileSystemConfigBuilder.java /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsReadOnlyFileContentInfoFactory.java /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/HdfsReadOnlyRandomAccessContent.java /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/ReadOnlyHdfsFileProvider.java /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/providers/ReadOnlyHdfsFileSystem.java /accumulo/trunk/start/src/main/java/org/apache/commons/vfs2 /accumulo/trunk/start/src/test/java/org/apache/accumulo/start/Test.java /accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/AccumuloReloadingVFSClassLoaderTest.java /accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/AccumuloVFSClassLoaderTest.java /accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/ContextManagerTest.java /accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/providers /accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/providers/ReadOnlyHdfsFileProviderTest.java /accumulo/trunk/start/src/test/java/org/apache/accumulo/start/classloader/vfs/providers/VfsClassLoaderTest.java /accumulo/trunk/start/src/test/java/org/apache/accumulo/test/AccumuloDFSBase.java /accumulo/trunk/start/src/test/java/org/apache/commons/vfs2 /accumulo/trunk/start/src/test/resources/application1 /accumulo/trunk/start/src/test/resources/default/conf/accumulo-site.xml /accumulo/trunk/start/src/test/resources/disabled/conf/accumulo-site.xml
          Hide
          Hudson added a comment -

          Integrated in Accumulo-Trunk #605 (See https://builds.apache.org/job/Accumulo-Trunk/605/)
          ACCUMULO-708 added warning if nothing is added for a classpath entry and removed some screwy code I recently added (Revision 1428622)

          Result = SUCCESS
          kturner :
          Files :

          • /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/AccumuloClassLoader.java
          Show
          Hudson added a comment - Integrated in Accumulo-Trunk #605 (See https://builds.apache.org/job/Accumulo-Trunk/605/ ) ACCUMULO-708 added warning if nothing is added for a classpath entry and removed some screwy code I recently added (Revision 1428622) Result = SUCCESS kturner : Files : /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/AccumuloClassLoader.java
          Hide
          Hudson added a comment -

          Integrated in Accumulo-Trunk #613 (See https://builds.apache.org/job/Accumulo-Trunk/613/)
          ACCUMULO-708 get functional tests working again (Revision 1430013)
          ACCUMULO-708 get functional tests working again (Revision 1430012)

          Result = SUCCESS
          ecn :
          Files :

          • /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/AccumuloClassLoader.java

          ecn :
          Files :

          • /accumulo/trunk/test/system/auto/TestUtils.py
          • /accumulo/trunk/test/system/auto/run.py
          Show
          Hudson added a comment - Integrated in Accumulo-Trunk #613 (See https://builds.apache.org/job/Accumulo-Trunk/613/ ) ACCUMULO-708 get functional tests working again (Revision 1430013) ACCUMULO-708 get functional tests working again (Revision 1430012) Result = SUCCESS ecn : Files : /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/AccumuloClassLoader.java ecn : Files : /accumulo/trunk/test/system/auto/TestUtils.py /accumulo/trunk/test/system/auto/run.py
          Josh Elser made changes -
          Link This issue relates to ACCUMULO-943 [ ACCUMULO-943 ]
          Josh Elser made changes -
          Link This issue breaks ACCUMULO-943 [ ACCUMULO-943 ]
          Josh Elser made changes -
          Link This issue relates to ACCUMULO-943 [ ACCUMULO-943 ]
          Hide
          Hudson added a comment -

          Integrated in Accumulo-Trunk #616 (See https://builds.apache.org/job/Accumulo-Trunk/616/)
          ACCUMULO-708 fix dynamic class loading test (Revision 1430407)

          Result = SUCCESS
          ecn :
          Files :

          • /accumulo/trunk/test/system/auto/simple/dynamic.py
          Show
          Hudson added a comment - Integrated in Accumulo-Trunk #616 (See https://builds.apache.org/job/Accumulo-Trunk/616/ ) ACCUMULO-708 fix dynamic class loading test (Revision 1430407) Result = SUCCESS ecn : Files : /accumulo/trunk/test/system/auto/simple/dynamic.py
          Hide
          Hudson added a comment -

          Integrated in Accumulo-Trunk #624 (See https://builds.apache.org/job/Accumulo-Trunk/624/)
          ACCUMULO-708 made per table classpaths work with mini accumulo cluster and added a test (Revision 1431474)

          Result = SUCCESS
          kturner :
          Files :

          • /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/AccumuloVFSClassLoader.java
          • /accumulo/trunk/test/src/test/java/org/apache/accumulo/test/MiniAccumuloClusterTest.java
          • /accumulo/trunk/test/src/test/resources/FooFilter.jar
          Show
          Hudson added a comment - Integrated in Accumulo-Trunk #624 (See https://builds.apache.org/job/Accumulo-Trunk/624/ ) ACCUMULO-708 made per table classpaths work with mini accumulo cluster and added a test (Revision 1431474) Result = SUCCESS kturner : Files : /accumulo/trunk/start/src/main/java/org/apache/accumulo/start/classloader/vfs/AccumuloVFSClassLoader.java /accumulo/trunk/test/src/test/java/org/apache/accumulo/test/MiniAccumuloClusterTest.java /accumulo/trunk/test/src/test/resources/FooFilter.jar
          John Vines made changes -
          Link This issue is duplicated by ACCUMULO-350 [ ACCUMULO-350 ]
          Hide
          Dave Marion added a comment -

          does it make sense to move the two open subtasks for this feature to the next release (not 1.5.0) and close this issue. The feature is in the 1.5.0 codebase. The two open tickets are not critical to 1.5.0.

          Show
          Dave Marion added a comment - does it make sense to move the two open subtasks for this feature to the next release (not 1.5.0) and close this issue. The feature is in the 1.5.0 codebase. The two open tickets are not critical to 1.5.0.
          Hide
          Eric Newton added a comment -

          Yes, please move them to 1.6.0 or unassign them based on your own willingness to work on it for 1.6.0.

          Show
          Eric Newton added a comment - Yes, please move them to 1.6.0 or unassign them based on your own willingness to work on it for 1.6.0.
          Hide
          Dave Marion added a comment -

          all subtasks completed, feature complete for 1.5.0

          Show
          Dave Marion added a comment - all subtasks completed, feature complete for 1.5.0
          Dave Marion made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Dave Marion made changes -
          Link This issue relates to ACCUMULO-1186 [ ACCUMULO-1186 ]
          Dave Marion made changes -
          Link This issue relates to ACCUMULO-886 [ ACCUMULO-886 ]
          Gavin made changes -
          Link This issue is related to ACCUMULO-926 [ ACCUMULO-926 ]
          Gavin made changes -
          Link This issue is related to ACCUMULO-926 [ ACCUMULO-926 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open In Progress In Progress
          89d 23h 42m 4 Dave Marion 28/Oct/12 23:51
          In Progress In Progress Open Open
          4m 30s 4 Dave Marion 28/Oct/12 23:51
          Patch Available Patch Available Open Open
          20s 1 Dave Marion 28/Oct/12 23:52
          Open Open Patch Available Patch Available
          4m 59s 2 Dave Marion 28/Oct/12 23:56
          Patch Available Patch Available Resolved Resolved
          149d 22h 57m 1 Dave Marion 27/Mar/13 22:54

            People

            • Assignee:
              Dave Marion
              Reporter:
              Dave Marion
            • Votes:
              2 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 24h Original Estimate - 24h
                24h
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 33h
                33h

                  Development