Uploaded image for project: 'Nutch'
  1. Nutch
  2. NUTCH-2639

bin/nutch fails to set native library path on Cygwin causing jobs to fail with UnsatisfiedLinkError

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4, 1.15
    • Fix Version/s: 2.4, 1.16
    • Component/s: bin
    • Labels:
      None
    • Environment:

      Windows 10 x64 Cygwin

      Description

      It is impossible to run nutch under Cygwin, even when hadoop.dll is properly available in the PATH.

      The issue is two-fold:

      1. JAVA_PLATFORM detects as "Windows_NT-amd64-64\r" (notice the trailing \r)

      2. A non-existent directory lib/native/Windows_NT-amd64-64\r is being set as java.library.path, making Java ignore the system PATH when looking for hadoop.dll.

       

      As a result, the following exception is thrown on any Gora backend access:

      java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
              at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
              at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:570)
              at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:977)
              at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:173)
              at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:160)
              at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:94)
              at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:285)
              at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
              at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
              at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
              at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
              at org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:131)
              at org.apache.hadoop.mapred.LocalJobRunner$Job.<init>(LocalJobRunner.java:163)
              at org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:731)
              at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432)
              at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
              at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:422)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
              at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
              at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
              at org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:115)
              at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:249)
              at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:270)
              at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:293)
              at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
              at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:302)
      

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                snagel Sebastian Nagel
                Reporter:
                rustamabd Rustam Abdullaev
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: