[SOLR-11473] Make HDFSDirectoryFactory support other prefixes (besides hdfs:/)

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 6.6.1
• Fix Version/s: 8.1, 9.0
• Component/s: Hadoop Integration, hdfs
• Assignee: Kevin Risden
• Reporter: Radu Gheorghe
• Created: 12/Oct/17 10:29
• Updated: 12/May/22 00:26
• Resolved: 30/Mar/19 19:05

Description

Not sure if it's a bug or a missing feature I'm trying to make Solr work on Alluxio, as described by Timothy Potter in https://www.slideshare.net/thelabdude/running-solr-in-the-cloud-at-memory-speed-with-alluxio/1

The problem I'm facing here is with autoAddReplicas. If I have replicationFactor=1 and the node with that replica dies, the node taking over incorrectly assigns the data directory. For example:

before

"dataDir":"alluxio://localhost:19998/solr/test/",

after

"dataDir":"alluxio://localhost:19998/solr/test/core_node1/alluxio://localhost:19998/solr/test/",

The same happens for ulogDir. Apparently, this has to do with this bit from HDFSDirectoryFactory:

  public boolean isAbsolute(String path) {
    return path.startsWith("hdfs:/");
  }

If I add "alluxio:/" in there, the paths are correct and the index is recovered.

I see a few options here:

• add "alluxio:/" to the list there
• add a regular expression in the lines of [a-z]*:/ I hope that's not too expensive, I'm not sure how often this method is called
• don't do anything and expect alluxio to work with an "hdfs:/" path? I actually tried that and didn't manage to make it work
• have a different DirectoryFactory or something else?

What do you think?

Issue Links

• is related to: SOLR-9958 The FileSystem used by HdfsBackupRepository gets closed before the backup completes.
• is related to: SOLR-11335 HdfsDirectory & Factory should not close the FileSystem object retrieved with get
• is related to: SOLR-13359 Make UpdateHandler support other prefixes (besides hdfs:/)
• relates to: SOLR-7301 HdfsDirectoryFactory does not support maprfs
• relates to: SOLR-12458 ADLS support for SOLR
• relates to: SOLR-9961 RestoreCore needs the option to download files in parallel.
• links to: GitHub Pull Request #628