Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.6.0
    • Fix Version/s: 1.6.0
    • Component/s: build
    • Labels:
      None
    • Environment:

      Mac OS X Mavericks

      Description

      The Makefile has bad includes for the test run, but even when I fix that, it still fails looking like it's not linking stdc++ properly. Josh Elser and I both looked at it for a bit and couldn't make it work.

      Since I last saw it succeed, I have both updated to Mavericks and merged in the change to have the pom manage make.

      I also haven't seen NativeMapIT pass since the updates, so I'm wondering if it's not just the test execution that's broken, but also the actual nativemap build.

        Issue Links

          Activity

          Michael Berman created issue -
          Hide
          Christopher Tubbs added a comment -

          NativeMapIT is a known issue and I can address that, but I don't have a Mac to reproduce the test failure in the Makefile.

          Show
          Christopher Tubbs added a comment - NativeMapIT is a known issue and I can address that, but I don't have a Mac to reproduce the test failure in the Makefile.
          Josh Elser made changes -
          Field Original Value New Value
          Link This issue is duplicated by ACCUMULO-1884 [ ACCUMULO-1884 ]
          Bill Havanki made changes -
          Assignee Bill Havanki [ bhavanki ]
          Bill Havanki made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Hide
          Bill Havanki added a comment -

          I have some success with running NativeMapIT. Here is some background.

          The suffix that Java uses to look for a library file changed from Java 6 to Java 7. Before, it was "jnilib". After, it is "dylib". The System.mapLibraryName() method is the originator of that difference; that method is used in NativeMap.getNativeLibPath(), whose result is used in the class's static initializer to try to load the library.

          So, if you are still running Java 6 (like me!), the "dylib" file is never looked for. If you are running Java 7 (the Mavericks upgrade apparently removes Java, so of course you'd re-install 7), it looks for "dylib".

          (By the way, the path also includes the result of Platform.getPlatform(), which adds the "Mac_OS_X-x86_64-64" component of the file name. See ACCUMULO-1843.

          There is code in MiniAccumuloCluster specifically designed to copy the native map library into the "mini-test" test directory, under lib/native/map. The code there looks under "server/src/main/c++", which is OK for 1.5.x, but for 1.6.x it's "server/native/target/<project-artifact-version>/<project-artifact-version>. Therefore, the copy never happens, and NativeMapIT fails since there is no library present.

          So, I have two changes. I'll post a review once Review Board is OK, and I've run the tests all the way through with mvn verify.

          1. Update the nativeMap Makefile to create .dylib and .jnilib copies of the library.
          2. Update the code that looks for the libraries in MiniAccumuloCluster to look in the right spot, and copy any library it finds over (so, both jnilib and dylib).

          Show
          Bill Havanki added a comment - I have some success with running NativeMapIT. Here is some background. The suffix that Java uses to look for a library file changed from Java 6 to Java 7. Before, it was "jnilib". After, it is "dylib". The System.mapLibraryName() method is the originator of that difference; that method is used in NativeMap.getNativeLibPath() , whose result is used in the class's static initializer to try to load the library. So, if you are still running Java 6 (like me!), the "dylib" file is never looked for. If you are running Java 7 (the Mavericks upgrade apparently removes Java, so of course you'd re-install 7), it looks for "dylib". (By the way, the path also includes the result of Platform.getPlatform() , which adds the "Mac_OS_X-x86_64-64" component of the file name. See ACCUMULO-1843 . There is code in MiniAccumuloCluster specifically designed to copy the native map library into the "mini-test" test directory, under lib/native/map. The code there looks under "server/src/main/c++", which is OK for 1.5.x, but for 1.6.x it's "server/native/target/<project-artifact-version>/<project-artifact-version>. Therefore, the copy never happens, and NativeMapIT fails since there is no library present. So, I have two changes. I'll post a review once Review Board is OK, and I've run the tests all the way through with mvn verify . 1. Update the nativeMap Makefile to create .dylib and .jnilib copies of the library. 2. Update the code that looks for the libraries in MiniAccumuloCluster to look in the right spot, and copy any library it finds over (so, both jnilib and dylib).
          Hide
          Bill Havanki added a comment -
          Show
          Bill Havanki added a comment - Review available: https://reviews.apache.org/r/15499/
          Bill Havanki made changes -
          Attachment ACCUMLUO-1852.patch [ 12613951 ]
          Hide
          Bill Havanki added a comment -

          Patch applies to 1.6.0-SNAPSHOT (and higher).

          Show
          Bill Havanki added a comment - Patch applies to 1.6.0-SNAPSHOT (and higher).
          Bill Havanki made changes -
          Status In Progress [ 3 ] Patch Available [ 10002 ]
          Fix Version/s 1.6.0 [ 12322468 ]
          Hide
          ASF subversion and git services added a comment -

          Commit 2aad7fe0af7701609df67dea3daf872a526761be in branch refs/heads/1.6.0-SNAPSHOT from Christopher Tubbs
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=2aad7fe ]

          ACCUMULO-1852 Fix native maps on OS X 10.9

          Show
          ASF subversion and git services added a comment - Commit 2aad7fe0af7701609df67dea3daf872a526761be in branch refs/heads/1.6.0-SNAPSHOT from Christopher Tubbs [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=2aad7fe ] ACCUMULO-1852 Fix native maps on OS X 10.9
          Hide
          Christopher Tubbs added a comment -

          Fixed, tested on OS X 10.8.2 and 10.9.

          Show
          Christopher Tubbs added a comment - Fixed, tested on OS X 10.8.2 and 10.9.
          Christopher Tubbs made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Assignee Bill Havanki [ bhavanki ] Christopher Tubbs [ ctubbsii ]
          Resolution Fixed [ 1 ]
          Hide
          ASF subversion and git services added a comment -

          Commit 2aad7fe0af7701609df67dea3daf872a526761be in branch refs/heads/master from Christopher Tubbs
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=2aad7fe ]

          ACCUMULO-1852 Fix native maps on OS X 10.9

          Show
          ASF subversion and git services added a comment - Commit 2aad7fe0af7701609df67dea3daf872a526761be in branch refs/heads/master from Christopher Tubbs [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=2aad7fe ] ACCUMULO-1852 Fix native maps on OS X 10.9
          Hide
          Bill Havanki added a comment -

          After pulling this ticket's commits, I'm still unable to run NativeMapIT on my Mac (10.8.5). I'm seeing UnsatisfiedLinkErrors. Am I verifying this ticket the wrong way, or perhaps I'm missing something in my environment?

          Show
          Bill Havanki added a comment - After pulling this ticket's commits, I'm still unable to run NativeMapIT on my Mac (10.8.5). I'm seeing UnsatisfiedLinkErrors. Am I verifying this ticket the wrong way, or perhaps I'm missing something in my environment?
          Hide
          Bill Havanki added a comment -

          I should follow up that it appears NativeMapIT passes when I run mvn verify, but not if I run mvn -Dtest=NativeMapIT -DfailIfNoTests=true test.

          Show
          Bill Havanki added a comment - I should follow up that it appears NativeMapIT passes when I run mvn verify , but not if I run mvn -Dtest=NativeMapIT -DfailIfNoTests=true test .
          Hide
          Josh Elser added a comment -

          That's odd. The mvn test... invocation works for me.

          Show
          Josh Elser added a comment - That's odd. The mvn test... invocation works for me.
          Hide
          Bill Havanki added a comment -

          ... and today it's working for me. shrug

          Show
          Bill Havanki added a comment - ... and today it's working for me. shrug
          Hide
          John Vines added a comment -
          Show
          John Vines added a comment - Hey Bill Havanki , do you mind closing https://reviews.apache.org/r/15499/
          Hide
          Bill Havanki added a comment -

          Done! Thanks for the bump.

          Show
          Bill Havanki added a comment - Done! Thanks for the bump.
          Hide
          Michael Allen added a comment -

          This did not work for me on my laptop until I added one more include to the native map test path. I'm submitting another patch, perhaps this bug needs to be reopened?

          Show
          Michael Allen added a comment - This did not work for me on my laptop until I added one more include to the native map test path. I'm submitting another patch, perhaps this bug needs to be reopened?
          Michael Allen made changes -
          John Vines made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Hide
          Josh Elser added a comment -

          Weird - it works on my mac (mavericks). What's your OSX and Java version?

          Show
          Josh Elser added a comment - Weird - it works on my mac (mavericks). What's your OSX and Java version?
          Hide
          Bill Havanki added a comment -

          I'm still working without that extra patch. I'm OSX 10.8.5, Java 1.6.0_65 (Apple's own). I found I needed to run mvn install before NativeMapIT would work (on 1.6.0-SNAPSHOT).

          mvn -DskipTests=true clean install
          mvn -Dtest=NativeMapIT -DfailIfNoTests=false test
          
          Show
          Bill Havanki added a comment - I'm still working without that extra patch. I'm OSX 10.8.5, Java 1.6.0_65 (Apple's own). I found I needed to run mvn install before NativeMapIT would work (on 1.6.0-SNAPSHOT). mvn -DskipTests=true clean install mvn -Dtest=NativeMapIT -DfailIfNoTests=false test
          Hide
          Josh Elser added a comment -

          Wait a second.. who is talking about which test now? Michael Allen is `make test` or the NativeMapIT failing for you? Or both?

          Show
          Josh Elser added a comment - Wait a second.. who is talking about which test now? Michael Allen is `make test` or the NativeMapIT failing for you? Or both?
          Hide
          Michael Berman added a comment -

          I think he was talking about `make test`. The error was about being unable to find jni_md.h. It works for me (mavericks, 1.7.0_25), and it's pulling that file from /System/Library/Frameworks/JavaVM.framework/Headers/jni_md.h (which through a series of symlinks is really /System/Library/Frameworks/JavaVM.framework/Versions/A/Headers/jni_md.h). Interestingly, the file is not identical to the one in $JAVA_HOME/include/darwin/jni_md.h, which maybe means we should be careful about which one we choose.

          In any case, I'm wondering which part of this isn't true for Michael Allen. I believe he was running 1.7.0_45, so maybe they rearranged the header files?

          Show
          Michael Berman added a comment - I think he was talking about `make test`. The error was about being unable to find jni_md.h. It works for me (mavericks, 1.7.0_25), and it's pulling that file from /System/Library/Frameworks/JavaVM.framework/Headers/jni_md.h (which through a series of symlinks is really /System/Library/Frameworks/JavaVM.framework/Versions/A/Headers/jni_md.h). Interestingly, the file is not identical to the one in $JAVA_HOME/include/darwin/jni_md.h, which maybe means we should be careful about which one we choose. In any case, I'm wondering which part of this isn't true for Michael Allen . I believe he was running 1.7.0_45, so maybe they rearranged the header files?
          Hide
          Michael Allen added a comment -

          It was definitely just the basic 'make test' part that was failing for me. On my Mac, for whatever reason, I do not have /System/Library/Frameworks/JavaVM.framework/Headers as a directory, so I'm not surprised it didn't work. I am running 1.7.0_45.

          Show
          Michael Allen added a comment - It was definitely just the basic 'make test' part that was failing for me. On my Mac, for whatever reason, I do not have /System/Library/Frameworks/JavaVM.framework/Headers as a directory, so I'm not surprised it didn't work. I am running 1.7.0_45.
          Hide
          Josh Elser added a comment -

          Interesting. I'm on 1.7.0_40, /System/Library/Frameworks/JavaVM.framework/Headers is symlinked to /System/Library/Frameworks/JavaVM.framework/Versions/Current/Headers and contains jni_md.h, but my JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_40.jdk/Contents/Home, not /System/Library/blah-blah.

          Honestly, I don't understand OSX java packaging one bit (/Library versus /System/Library), nor the differences between Apple java and Oracle Java.

          Show
          Josh Elser added a comment - Interesting. I'm on 1.7.0_40, /System/Library/Frameworks/JavaVM.framework/Headers is symlinked to /System/Library/Frameworks/JavaVM.framework/Versions/Current/Headers and contains jni_md.h, but my JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_40.jdk/Contents/Home, not /System/Library/blah-blah. Honestly, I don't understand OSX java packaging one bit (/Library versus /System/Library), nor the differences between Apple java and Oracle Java.
          Hide
          Michael Allen added a comment -

          Josh Elser I agree that OSX packaging is just kind of a mess. From a defensive "let's just make this work" perspective, I'm not sure there's any downside to just putting more include directories on that test line that cover the majority of cases, even if on certain systems those directories don't technically exist.

          Show
          Michael Allen added a comment - Josh Elser I agree that OSX packaging is just kind of a mess. From a defensive "let's just make this work" perspective, I'm not sure there's any downside to just putting more include directories on that test line that cover the majority of cases, even if on certain systems those directories don't technically exist.
          Hide
          Josh Elser added a comment -

          Agreed – just wanted to rule out the possibility of user configuration first.

          Show
          Josh Elser added a comment - Agreed – just wanted to rule out the possibility of user configuration first.
          Hide
          Bill Havanki added a comment -

          I think that Apple's Java (which discontinued with Java 7) lives under /System/Library, but Oracle's Java 7 and later live under /Library.

          Sorry about the earlier confusion.

          I noticed when I run make test that the Makefile picks up that I have Oracle's Java 7 installed and uses that as JAVA_HOME for its includes: /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home. I forced it to use Java 6's JAVA_HOME and it still worked: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home.

          Show
          Bill Havanki added a comment - I think that Apple's Java (which discontinued with Java 7) lives under /System/Library, but Oracle's Java 7 and later live under /Library. Sorry about the earlier confusion. I noticed when I run make test that the Makefile picks up that I have Oracle's Java 7 installed and uses that as JAVA_HOME for its includes: /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home. I forced it to use Java 6's JAVA_HOME and it still worked: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home.
          Hide
          Bill Havanki added a comment -

          Regardless, +1 to adding the include as a defensive measure anyway.

          Show
          Bill Havanki added a comment - Regardless, +1 to adding the include as a defensive measure anyway.
          Hide
          ASF subversion and git services added a comment -

          Commit f0c12cd04761b40b4642b9a03640c11ee72ce0e0 in branch refs/heads/1.6.0-SNAPSHOT from Michael Allen
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=f0c12cd ]

          ACCUMULO-1852 Further fixes nativemap test on OSX

          To make it work on my system I had to add this bit
          to the include path for executing the test.

          Signed-off-by: Josh Elser <elserj@apache.org>

          Show
          ASF subversion and git services added a comment - Commit f0c12cd04761b40b4642b9a03640c11ee72ce0e0 in branch refs/heads/1.6.0-SNAPSHOT from Michael Allen [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=f0c12cd ] ACCUMULO-1852 Further fixes nativemap test on OSX To make it work on my system I had to add this bit to the include path for executing the test. Signed-off-by: Josh Elser <elserj@apache.org>
          Hide
          ASF subversion and git services added a comment -

          Commit f0c12cd04761b40b4642b9a03640c11ee72ce0e0 in branch refs/heads/master from Michael Allen
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=f0c12cd ]

          ACCUMULO-1852 Further fixes nativemap test on OSX

          To make it work on my system I had to add this bit
          to the include path for executing the test.

          Signed-off-by: Josh Elser <elserj@apache.org>

          Show
          ASF subversion and git services added a comment - Commit f0c12cd04761b40b4642b9a03640c11ee72ce0e0 in branch refs/heads/master from Michael Allen [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=f0c12cd ] ACCUMULO-1852 Further fixes nativemap test on OSX To make it work on my system I had to add this bit to the include path for executing the test. Signed-off-by: Josh Elser <elserj@apache.org>
          Hide
          Josh Elser added a comment -

          New patch applied, thanks Michael Allen.

          Show
          Josh Elser added a comment - New patch applied, thanks Michael Allen .
          Josh Elser made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Christopher Tubbs made changes -
          Link This issue is duplicated by ACCUMULO-1845 [ ACCUMULO-1845 ]

            People

            • Assignee:
              Christopher Tubbs
              Reporter:
              Michael Berman
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development