Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-11335

HdfsDirectory & Factory should not close the FileSystem object retrieved with get

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Workaround
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Hadoop Integration, hdfs
    • Labels:
      None

      Description

      I'm seeing issues where the Hadoop FileSystem instance is closed out from under other objects. From what I understand, the Hadoop FileSystem object (org.apache.hadoop.fs.FileSystem) retrieved with FileSystem.get as is done in HdfsDirectory's ctor is a shared object that if closed, can affect other code using that same shared instance. You can see this is a cached, shared object here ->
      https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java#L455

      Thus, I suspect Solr should not be closing any FileSystem instance retrieved with get. It's important to mention that if I set the fs.$SCHEME.impl.disable.cache to true, then my problems go away, which seems to confirm that Solr is using the API incorrectly. That being said, I'm surprised this hasn't been raised before, so maybe I've missed something basic in Solr's use of HDFS?

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                thelabdude Timothy Potter
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: