[NUTCH-2639] bin/nutch fails to set native library path on Cygwin causing jobs to fail with UnsatisfiedLinkError - ASF JIRA

Voters

Watch issue

Watchers

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: 2.4, 1.15
Fix Version/s: 2.4, 1.16
Component/s: bin
Labels:
None
Environment:

Windows 10 x64 Cygwin

Description

It is impossible to run nutch under Cygwin, even when hadoop.dll is properly available in the PATH.

The issue is two-fold:

1. JAVA_PLATFORM detects as "Windows_NT-amd64-64\r" (notice the trailing \r)

2. A non-existent directory lib/native/Windows_NT-amd64-64\r is being set as java.library.path, making Java ignore the system PATH when looking for hadoop.dll.

As a result, the following exception is thrown on any Gora backend access:

java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
        at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
        at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:570)
        at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:977)
        at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:173)
        at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:160)
        at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:94)
        at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:285)
        at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
        at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
        at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
        at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
        at org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:131)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.<init>(LocalJobRunner.java:163)
        at org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:731)
        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
        at org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:115)
        at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:249)
        at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:270)
        at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:293)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:302)