Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4.4, 1.5.0
    • Component/s: client
    • Labels:
      None
    • Environment:

      Windows 7, Oracle Java 7

      Description

      I set the log level up a bit, here's the result.

      [main] DEBUG org.apache.hadoop.conf.Configuration  - java.io.IOException: config()
      	at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:227)
      	at org.apache.accumulo.core.conf.SiteConfiguration.getXmlConfig(SiteConfiguration.java:49)
      	at org.apache.accumulo.core.conf.SiteConfiguration.iterator(SiteConfiguration.java:80)
      	at org.apache.accumulo.core.conf.ConfigSanityCheck.validate(ConfigSanityCheck.java:29)
      	at org.apache.accumulo.core.conf.SiteConfiguration.getInstance(SiteConfiguration.java:41)
      	at org.apache.accumulo.server.conf.ServerConfiguration.getSiteConfiguration(ServerConfiguration.java:38)
      	at org.apache.accumulo.test.MiniAccumuloCluster.init(MiniAccumuloCluster.java:192)
      	at org.apache.accumulo.test.MiniAccumuloCluster.start(MiniAccumuloCluster.java:320)
      	at org.apache.accumulo.instamo.ExampleAccumuloUnitTest.setupMiniCluster(ExampleAccumuloUnitTest.java:46)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      	at java.lang.reflect.Method.invoke(Unknown Source)
      	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
      	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
      	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
      	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
      	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
      	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
      	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
      	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
      	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
      	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
      	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
      
      [main] WARN  org.apache.accumulo.core.conf.SiteConfiguration  - accumulo-site.xml not found on classpath
      [main] DEBUG org.apache.hadoop.conf.Configuration  - java.io.IOException: config()
      	at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:227)
      	at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:214)
      	at org.apache.accumulo.core.util.CachedConfiguration.getInstance(CachedConfiguration.java:26)
      	at org.apache.accumulo.server.util.Initialize.main(Initialize.java:485)
      	at org.apache.accumulo.test.MiniAccumuloCluster.init(MiniAccumuloCluster.java:197)
      	at org.apache.accumulo.test.MiniAccumuloCluster.start(MiniAccumuloCluster.java:320)
      	at org.apache.accumulo.instamo.ExampleAccumuloUnitTest.setupMiniCluster(ExampleAccumuloUnitTest.java:46)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      	at java.lang.reflect.Method.invoke(Unknown Source)
      	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
      	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
      	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
      	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
      	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
      	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
      	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
      	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
      	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
      	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
      	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
      
      [main] DEBUG org.apache.hadoop.conf.Configuration  - java.io.IOException: config()
      	at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:227)
      	at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:214)
      	at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:187)
      	at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:239)
      	at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:438)
      	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:424)
      	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1494)
      	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1395)
      	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
      	at org.apache.accumulo.core.file.FileUtil.getFileSystem(FileUtil.java:553)
      	at org.apache.accumulo.server.util.Initialize.main(Initialize.java:487)
      	at org.apache.accumulo.test.MiniAccumuloCluster.init(MiniAccumuloCluster.java:197)
      	at org.apache.accumulo.test.MiniAccumuloCluster.start(MiniAccumuloCluster.java:320)
      	at org.apache.accumulo.instamo.ExampleAccumuloUnitTest.setupMiniCluster(ExampleAccumuloUnitTest.java:46)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      	at java.lang.reflect.Method.invoke(Unknown Source)
      	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
      	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
      	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
      	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
      	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
      	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
      	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
      	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
      	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
      	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
      	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
      
      [main] DEBUG org.apache.hadoop.security.Groups  -  Creating new Groups object
      [main] DEBUG org.apache.hadoop.security.Groups  - Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
      [main] DEBUG org.apache.hadoop.conf.Configuration  - java.io.IOException: config()
      	at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:227)
      	at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:214)
      	at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:187)
      	at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:239)
      	at org.apache.hadoop.security.KerberosName.<clinit>(KerberosName.java:83)
      	at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:212)
      	at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:187)
      	at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:239)
      	at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:438)
      	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:424)
      	at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1494)
      	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1395)
      	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
      	at org.apache.accumulo.core.file.FileUtil.getFileSystem(FileUtil.java:553)
      	at org.apache.accumulo.server.util.Initialize.main(Initialize.java:487)
      	at org.apache.accumulo.test.MiniAccumuloCluster.init(MiniAccumuloCluster.java:197)
      	at org.apache.accumulo.test.MiniAccumuloCluster.start(MiniAccumuloCluster.java:320)
      	at org.apache.accumulo.instamo.ExampleAccumuloUnitTest.setupMiniCluster(ExampleAccumuloUnitTest.java:46)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      	at java.lang.reflect.Method.invoke(Unknown Source)
      	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
      	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
      	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
      	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
      	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
      	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
      	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
      	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
      	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
      	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
      	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
      
      [main] DEBUG org.apache.hadoop.security.UserGroupInformation  - hadoop login
      [main] DEBUG org.apache.hadoop.security.UserGroupInformation  - hadoop login commit
      [main] DEBUG org.apache.hadoop.security.UserGroupInformation  - using local user:NTUserPrincipal: mike
      [main] DEBUG org.apache.hadoop.security.UserGroupInformation  - UGI loginUser:mike
      [main] DEBUG org.apache.hadoop.fs.FileSystem  - Creating filesystem for file:///
      [main] INFO  org.apache.accumulo.server.util.Initialize  - Hadoop Filesystem is file:///
      [main] ERROR org.apache.accumulo.core.conf.SiteConfiguration  - Using default value for instance.dfs.dir due to improperly formatted absolute path: D:\Users\mike\AppData\Local\Temp\junit1725534677520785696\accumulo
      [main] INFO  org.apache.accumulo.server.util.Initialize  - Accumulo data dir is /accumulo
      [main] INFO  org.apache.accumulo.server.util.Initialize  - Zookeeper server is localhost:36607
      [main] INFO  org.apache.accumulo.server.util.Initialize  - Checking if Zookeeper is available. If this hangs, then you need to make sure zookeeper is running
      [main] DEBUG org.apache.accumulo.fate.zookeeper.ZooSession  - Connecting to localhost:36607 with timeout 30000 with auth
      [main] ERROR org.apache.accumulo.core.conf.SiteConfiguration  - Using default value for instance.dfs.dir due to improperly formatted absolute path: D:\Users\mike\AppData\Local\Temp\junit1725534677520785696\accumulo
      [main] FATAL org.apache.accumulo.server.util.Initialize  - It appears this location was previously initialized, exiting ... 
      [Thread-4] DEBUG org.apache.hadoop.fs.FileSystem  - Starting clear of FileSystem cache with 1 elements.
      [Thread-4] DEBUG org.apache.hadoop.fs.FileSystem  - Removing filesystem for file:///
      [Thread-4] DEBUG org.apache.hadoop.fs.FileSystem  - Removing filesystem for file:///
      [Thread-4] DEBUG org.apache.hadoop.fs.FileSystem  - Done clearing cache
      

        Issue Links

          Activity

          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 1500320 from Mike Drob
          [ https://svn.apache.org/r1500320 ]

          ACCUMULO-895 Applying Nguessan Kouame's patch, modulo movement of the minicluster source.

          Show
          jira-bot ASF subversion and git services added a comment - Commit 1500320 from Mike Drob [ https://svn.apache.org/r1500320 ] ACCUMULO-895 Applying Nguessan Kouame's patch, modulo movement of the minicluster source.
          Hide
          kturner Keith Turner added a comment -

          this is ok in 1.5.0, still needs to be fixed for 1.4.4

          Show
          kturner Keith Turner added a comment - this is ok in 1.5.0, still needs to be fixed for 1.4.4
          Hide
          ctubbsii Christopher Tubbs added a comment -

          Looks fixed to me. Re-open or new ticket if still an issue.

          Show
          ctubbsii Christopher Tubbs added a comment - Looks fixed to me. Re-open or new ticket if still an issue.
          Hide
          elserj Josh Elser added a comment -

          I can double check later this week if that would add to your warm fuzzies.

          That would be wonderful. I know how much you can get caught up in the thrill of the hackathon-moment.

          Show
          elserj Josh Elser added a comment - I can double check later this week if that would add to your warm fuzzies. That would be wonderful. I know how much you can get caught up in the thrill of the hackathon-moment.
          Hide
          mdrob Mike Drob added a comment -

          Josh Elser - Those changes were a result of the May 2013 hackathon and were verified to work at that time. I can double check later this week if that would add to your warm fuzzies.

          Show
          mdrob Mike Drob added a comment - Josh Elser - Those changes were a result of the May 2013 hackathon and were verified to work at that time. I can double check later this week if that would add to your warm fuzzies.
          Hide
          elserj Josh Elser added a comment -

          These changes make MiniAccumuloCluster work on Windows. This is for Accumulo 1.4.

          Does anyone have an environment set up to build on Windows that can verify this patch fixes things? I'm a little scared to go down that path, personally.

          Show
          elserj Josh Elser added a comment - These changes make MiniAccumuloCluster work on Windows. This is for Accumulo 1.4. Does anyone have an environment set up to build on Windows that can verify this patch fixes things? I'm a little scared to go down that path, personally.
          Hide
          vines John Vines added a comment -

          Can we not just use File.pathSeperator?

          Show
          vines John Vines added a comment - Can we not just use File.pathSeperator?
          Hide
          kturner Keith Turner added a comment -

          seems MAC is not working under windows on 1.4

          Show
          kturner Keith Turner added a comment - seems MAC is not working under windows on 1.4
          Hide
          nguessan Nguessan Kouame added a comment -

          These changes make MiniAccumuloCluster work on Windows. This is for Accumulo 1.4.

          Show
          nguessan Nguessan Kouame added a comment - These changes make MiniAccumuloCluster work on Windows. This is for Accumulo 1.4.
          Hide
          billie.rinaldi Billie Rinaldi added a comment -

          I figured out why it fails in Hadoop 2.0. The change below in Path makes it only perform Windows slash normalization if you're on a Windows system. I'm going to remove the Windows path test from PropertyTest until this get fixed. Think this is worth opening a Hadoop ticket?

            static final boolean WINDOWS
              = System.getProperty("os.name").startsWith("Windows");
          
          -    if (path.indexOf("//") != -1) {
          -      path = path.replace("//", "/");
          -    }
          -    if (path.indexOf("\\") != -1) {    
          -      path = path.replace("\\", "/");
          +    path = StringUtils.replace(path, "//", "/");
          +    if (Path.WINDOWS) {
          +      path = StringUtils.replace(path, "\\", "/");
          
          Show
          billie.rinaldi Billie Rinaldi added a comment - I figured out why it fails in Hadoop 2.0. The change below in Path makes it only perform Windows slash normalization if you're on a Windows system. I'm going to remove the Windows path test from PropertyTest until this get fixed. Think this is worth opening a Hadoop ticket? static final boolean WINDOWS = System.getProperty("os.name").startsWith("Windows"); - if (path.indexOf("//") != -1) { - path = path.replace("//", "/"); - } - if (path.indexOf("\\") != -1) { - path = path.replace("\\", "/"); + path = StringUtils.replace(path, "//", "/"); + if (Path.WINDOWS) { + path = StringUtils.replace(path, "\\", "/");
          Hide
          billie.rinaldi Billie Rinaldi added a comment -

          PropertyTest fails when compiling against Hadoop 2 (mvn clean package -Dhadoop.profile=2.0):

          testTypes(org.apache.accumulo.core.conf.PropertyTest)  Time elapsed: 0.005 sec  <<< ERROR!
          java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: d:%5Cfoo12
                  at org.apache.hadoop.fs.Path.initialize(Path.java:157)
                  at org.apache.hadoop.fs.Path.<init>(Path.java:135)
                  at org.apache.accumulo.core.conf.PropertyType$1.isValidFormat(PropertyType.java:57)
                  at org.apache.accumulo.core.conf.PropertyTest.typeCheckValidFormat(PropertyTest.java:76)
                  at org.apache.accumulo.core.conf.PropertyTest.testTypes(PropertyTest.java:96)
          
          Show
          billie.rinaldi Billie Rinaldi added a comment - PropertyTest fails when compiling against Hadoop 2 (mvn clean package -Dhadoop.profile=2.0): testTypes(org.apache.accumulo.core.conf.PropertyTest) Time elapsed: 0.005 sec <<< ERROR! java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: d:%5Cfoo12 at org.apache.hadoop.fs.Path.initialize(Path.java:157) at org.apache.hadoop.fs.Path.<init>(Path.java:135) at org.apache.accumulo.core.conf.PropertyType$1.isValidFormat(PropertyType.java:57) at org.apache.accumulo.core.conf.PropertyTest.typeCheckValidFormat(PropertyTest.java:76) at org.apache.accumulo.core.conf.PropertyTest.testTypes(PropertyTest.java:96)
          Hide
          hudson Hudson added a comment -

          Integrated in Accumulo-Trunk #574 (See https://builds.apache.org/job/Accumulo-Trunk/574/)
          ACCUMULO-895 write zookeeper config using java.util.Properties to properly escape windows path (Revision 1420312)

          Result = SUCCESS
          kturner :
          Files :

          • /accumulo/trunk/test/src/main/java/org/apache/accumulo/test/MiniAccumuloCluster.java
          Show
          hudson Hudson added a comment - Integrated in Accumulo-Trunk #574 (See https://builds.apache.org/job/Accumulo-Trunk/574/ ) ACCUMULO-895 write zookeeper config using java.util.Properties to properly escape windows path (Revision 1420312) Result = SUCCESS kturner : Files : /accumulo/trunk/test/src/main/java/org/apache/accumulo/test/MiniAccumuloCluster.java
          Hide
          kturner Keith Turner added a comment -

          Not sure how to do this in a sane, cross-platform, way.

          I looked at the zookeeper source code to see how they read these props. They use java.util.Properties. So I wrote zoo.cfg using Properties. I did a test with a windows path and it escaped the back slashes and the colon for some reason. So I think this should work, the dataDir property is used to create a File object.

          Show
          kturner Keith Turner added a comment - Not sure how to do this in a sane, cross-platform, way. I looked at the zookeeper source code to see how they read these props. They use java.util.Properties. So I wrote zoo.cfg using Properties. I did a test with a windows path and it escaped the back slashes and the colon for some reason. So I think this should work, the dataDir property is used to create a File object.
          Hide
          mdrob Mike Drob added a comment -

          Windows pathing issue.

          Changing the file setup to

          fileWriter.append("dataDir=" + zooKeeperDir.getAbsolutePath().replace("\\", "\\\\") + "\n");
          

          worked for me. Not sure how to do this in a sane, cross-platform, way.

          Show
          mdrob Mike Drob added a comment - Windows pathing issue. Changing the file setup to fileWriter.append( "dataDir=" + zooKeeperDir.getAbsolutePath().replace( "\\" , "\\\\" ) + "\n" ); worked for me. Not sure how to do this in a sane, cross-platform, way.
          Hide
          mdrob Mike Drob added a comment -
          zoo.cfg
          tickTime=1000
          initLimit=10
          syncLimit=5
          clientPort=9191
          maxClientCnxns=100
          dataDir=D:\Users\mike\AppData\Local\Temp\junit8526291817458045947\zookeeper
          
          Show
          mdrob Mike Drob added a comment - zoo.cfg tickTime=1000 initLimit=10 syncLimit=5 clientPort=9191 maxClientCnxns=100 dataDir=D:\Users\mike\AppData\Local\Temp\junit8526291817458045947\zookeeper
          Hide
          kturner Keith Turner added a comment -

          For the zookeeper issue, can you look at %TEMP%/junitXXXXXXXX/conf/zoo.cfg and see what dataDir is set to. This file is generated by MiniAccumuloCluster, want to ensure its generating something reasonable.

          Show
          kturner Keith Turner added a comment - For the zookeeper issue, can you look at %TEMP%/junitXXXXXXXX/conf/zoo.cfg and see what dataDir is set to. This file is generated by MiniAccumuloCluster, want to ensure its generating something reasonable.
          Hide
          mdrob Mike Drob added a comment -

          Also, every time I run, I get a %TEMP%/junitXXXXXXXX directory, but those don't seem to be cleaning up correctly. Some of them are partially cleaned up, making me think that this is going to be hard to debug and I'm not even sure where to start.

          Even more interestingly, I get a corresponding UsersmikeAppDataLocalTempjunitXXXXXXXXzookeeper directory in my workspace root that never cleans up. It contains a version-2 directory with log.1 and snapshot.0 files. Zookeeper might not be behaving entirely correctly?

          Show
          mdrob Mike Drob added a comment - Also, every time I run, I get a %TEMP%/junitXXXXXXXX directory, but those don't seem to be cleaning up correctly. Some of them are partially cleaned up, making me think that this is going to be hard to debug and I'm not even sure where to start. Even more interestingly, I get a corresponding UsersmikeAppDataLocalTempjunitXXXXXXXXzookeeper directory in my workspace root that never cleans up. It contains a version-2 directory with log.1 and snapshot.0 files. Zookeeper might not be behaving entirely correctly?
          Hide
          mdrob Mike Drob added a comment -

          Can run the ExampleAccumuloUnitTest now!

          Having issues with MapReduceExample, though.

             ---- Running Mapred Against Accumulo
          
          UUID 1355244649903 81398c27-297e-4087-8ea8-ec0bdb976220
          FLUSH 1355244659049 9144 24 1000000 1000000
          FLUSH 1355244668456 9407 16 2000000 1000000
          FLUSH 1355244680016 11560 27 3000000 1000000
          FLUSH 1355244689021 9005 15 4000000 1000000
          FLUSH 1355244703781 14760 13 5000000 1000000
          [main] ERROR org.apache.hadoop.security.UserGroupInformation  - PriviledgedActionException as:mike cause:java.io.IOException: Failed to set permissions of path: \tmp\hadoop-mike\mapred\staging\mike-101115261\.staging to 0700
          Exception in thread "main" java.io.IOException: Failed to set permissions of path: \tmp\hadoop-mike\mapred\staging\mike-101115261\.staging to 0700
          	at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:682)
          	at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:655)
          	at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
          	at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
          	at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
          	at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116)
          	at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:856)
          	at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850)
          	at java.security.AccessController.doPrivileged(Native Method)
          	at javax.security.auth.Subject.doAs(Unknown Source)
          	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
          	at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850)
          	at org.apache.hadoop.mapreduce.Job.submit(Job.java:500)
          	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530)
          	at org.apache.accumulo.server.test.continuous.ContinuousVerify.run(ContinuousVerify.java:216)
          	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
          	at org.apache.accumulo.server.test.continuous.ContinuousVerify.main(ContinuousVerify.java:232)
          	at org.apache.accumulo.instamo.MapReduceExample.run(MapReduceExample.java:43)
          	at org.apache.accumulo.instamo.MapReduceExample.main(MapReduceExample.java:60)
          

          Perhaps need to specify a mapred work directory in some conf before launching?

          Show
          mdrob Mike Drob added a comment - Can run the ExampleAccumuloUnitTest now! Having issues with MapReduceExample , though. ---- Running Mapred Against Accumulo UUID 1355244649903 81398c27-297e-4087-8ea8-ec0bdb976220 FLUSH 1355244659049 9144 24 1000000 1000000 FLUSH 1355244668456 9407 16 2000000 1000000 FLUSH 1355244680016 11560 27 3000000 1000000 FLUSH 1355244689021 9005 15 4000000 1000000 FLUSH 1355244703781 14760 13 5000000 1000000 [main] ERROR org.apache.hadoop.security.UserGroupInformation - PriviledgedActionException as:mike cause:java.io.IOException: Failed to set permissions of path: \tmp\hadoop-mike\mapred\staging\mike-101115261\.staging to 0700 Exception in thread "main" java.io.IOException: Failed to set permissions of path: \tmp\hadoop-mike\mapred\staging\mike-101115261\.staging to 0700 at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:682) at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:655) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344) at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189) at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:856) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Unknown Source) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850) at org.apache.hadoop.mapreduce.Job.submit(Job.java:500) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530) at org.apache.accumulo.server.test.continuous.ContinuousVerify.run(ContinuousVerify.java:216) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.accumulo.server.test.continuous.ContinuousVerify.main(ContinuousVerify.java:232) at org.apache.accumulo.instamo.MapReduceExample.run(MapReduceExample.java:43) at org.apache.accumulo.instamo.MapReduceExample.main(MapReduceExample.java:60) Perhaps need to specify a mapred work directory in some conf before launching?
          Hide
          hudson Hudson added a comment -

          Integrated in Accumulo-Trunk #573 (See https://builds.apache.org/job/Accumulo-Trunk/573/)
          ACCUMULO-895 allowed windows path for instance.dfs.dir (Revision 1420142)

          Result = SUCCESS
          kturner :
          Files :

          • /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/conf/PropertyType.java
          • /accumulo/trunk/core/src/test/java/org/apache/accumulo/core/conf/PropertyTest.java
          Show
          hudson Hudson added a comment - Integrated in Accumulo-Trunk #573 (See https://builds.apache.org/job/Accumulo-Trunk/573/ ) ACCUMULO-895 allowed windows path for instance.dfs.dir (Revision 1420142) Result = SUCCESS kturner : Files : /accumulo/trunk/core/src/main/java/org/apache/accumulo/core/conf/PropertyType.java /accumulo/trunk/core/src/test/java/org/apache/accumulo/core/conf/PropertyTest.java
          Hide
          kturner Keith Turner added a comment -

          Is there a reason that instance.dfs.dir requires the value to start with "/"?

          I think the intent was to ensure it was an absolute path. It seems Path has an isAbsolute() method that handles windows paths. I will modify to use that.

          Show
          kturner Keith Turner added a comment - Is there a reason that instance.dfs.dir requires the value to start with "/"? I think the intent was to ensure it was an absolute path. It seems Path has an isAbsolute() method that handles windows paths. I will modify to use that.
          Hide
          mdrob Mike Drob added a comment -

          Is there a reason that instance.dfs.dir requires the value to start with "/"?

          Show
          mdrob Mike Drob added a comment - Is there a reason that instance.dfs.dir requires the value to start with "/" ?

            People

            • Assignee:
              kturner Keith Turner
              Reporter:
              mdrob Mike Drob
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development