[SOLR-14653] Replace usages of org.apache.hadoop.fs.Path that are used to construct Zookeeper paths

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 8.7
• Assignee: Erick Erickson
• Reporter: Erick Erickson
• Created: 16/Jul/20 12:52
• Updated: 04/Nov/20 15:22
• Resolved: 17/Jul/20 20:24

Description

See SOLR-13939. We construct Zookeeper paths at least in ShardLeaderElectionContextBase.java by using methods in org.apache.hadoop.fs.Path e.g. "String parent = new Path(leaderPath).getParent().toString()". This works because  the Hadoop method uses forward slashes even on Windows machines.

If those calls are changed to use nio Paths.get or similar, we can't run on Windows. Relying on hdfs classes to construct these paths in code unrelated to HDFS is suspicious at best.

We should examine all inclusions of org.apache.hadoop.fs.Path and replace them with something else. Uwe suggests java.net.URI as one possibility.

We should check to see what Zookeeper provides.

There are 5 files that import org.apache.hadoop... that do not have HDFS in their name, and 4 of those are probably legitimate (SolrSnapshotsTool for instance) so checking all of them won't take long.

From Uwe's comment in SOLR-13939:

"Hi,
Windows is happy again. So your fix helped. I responded to mailing list, here again:

In Zookeeper all paths are forward slashes (as it is a "path" inside the zookeeper environment). If you replace that with a NIO Windows Path it creates backslashes. This makes Zookeeper fail (of course).

Actually using the Hadoop class is broken workaround, it may be better to use the more abstract java.net.URI class for correct "path" transformations. The risk with java.nio.file.Path is also that if the path exists in file system it handles the resolving different. So if some user has for some reason a local path exactly looking like a zookeper node path it would use different behaviour.

So my suggestion: Use java.net.URI to build the path and use URI.getPath() or similar to make a String out of it. The alternative to replace all backslashes in the path to forwards slashes is a bad idea for the reasons I explained before.

Now we should replace the hadoop class IMHO by java.net.URI.

Issue Links

• relates to: SOLR-13939 Extract any non-gradle related patches (deprecations, URL fixes, etc.) from gradle effort