Solr
  1. Solr
  2. SOLR-2563

Allow generic pluggable file system implementations

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 4.0-ALPHA
    • Fix Version/s: None
    • Component/s: update
    • Labels:
      None

      Description

      For things like configuration files, they can be loaded from places other than the local filesystem, such as Zookeeper or HDFS. In this issue I will abstract that functionality out.

        Issue Links

          Activity

          Hide
          Uwe Schindler added a comment - - edited

          For configuration files (works even with XIncludes) there is already ResourceLoader, which does exactly what you want. To support Zookeeper or HDFS just write a ClassLoader that loads from there.

          Since I fixed the XInclude stuff (to use solrres://-URIs, see SOLR-1656), even that works phantastic. Abstraction is ResourceLoader, what is your new idea?

          Show
          Uwe Schindler added a comment - - edited For configuration files (works even with XIncludes) there is already ResourceLoader, which does exactly what you want. To support Zookeeper or HDFS just write a ClassLoader that loads from there. Since I fixed the XInclude stuff (to use solrres://-URIs, see SOLR-1656 ), even that works phantastic. Abstraction is ResourceLoader, what is your new idea?
          Hide
          Jason Rutherglen added a comment -

          Uwe, thanks, I think though there was an issue even trying to use that. I'll take a look and report back!

          Show
          Jason Rutherglen added a comment - Uwe, thanks, I think though there was an issue even trying to use that. I'll take a look and report back!
          Hide
          Jason Rutherglen added a comment -

          I don't think CoreContainer is completely removed from the local file system. Checkout persist, persistFile, etc. Those should either be turned off, or should write to the underlying generic file system.

          It looks like libs are hard coded in CoreContainer?

          if (libDir != null) {
            File f = FileUtils.resolvePath(new File(dir), libDir);
            log.info( "loading shared library: "+f.getAbsolutePath() );
            libLoader = SolrResourceLoader.createClassLoader(f, null);
          }
          

          CoreDescriptor.getDataDir() is ambiguous.

          QueryElevationComponent is hardcoded:

          // check if using ZooKeeper
          ZkController zkController = core.getCoreDescriptor().getCoreContainer().getZkController();
          if(zkController != null) {
          

          IndexBasedSpellChecker.initSourceReader()

          SolrIndexWriter hardcodes writing the infoStream to the local file system.

          The benchmark code is as well however that's somewhat less of a priority.

          Show
          Jason Rutherglen added a comment - I don't think CoreContainer is completely removed from the local file system. Checkout persist, persistFile, etc. Those should either be turned off, or should write to the underlying generic file system. It looks like libs are hard coded in CoreContainer? if (libDir != null ) { File f = FileUtils.resolvePath( new File(dir), libDir); log.info( "loading shared library: " +f.getAbsolutePath() ); libLoader = SolrResourceLoader.createClassLoader(f, null ); } CoreDescriptor.getDataDir() is ambiguous. QueryElevationComponent is hardcoded: // check if using ZooKeeper ZkController zkController = core.getCoreDescriptor().getCoreContainer().getZkController(); if (zkController != null ) { IndexBasedSpellChecker.initSourceReader() SolrIndexWriter hardcodes writing the infoStream to the local file system. The benchmark code is as well however that's somewhat less of a priority.
          Hide
          Jason Rutherglen added a comment -

          One way to test this out would be to create a Solr unit test that tries to create a Solr instance on top of HDFS using an HDFSSolrResourceLoader. Then I think the problem areas would reveal themselves. It would be nice to run all of the Solr unit tests this way, however that seems much more complex.

          Show
          Jason Rutherglen added a comment - One way to test this out would be to create a Solr unit test that tries to create a Solr instance on top of HDFS using an HDFSSolrResourceLoader. Then I think the problem areas would reveal themselves. It would be nice to run all of the Solr unit tests this way, however that seems much more complex.

            People

            • Assignee:
              Unassigned
              Reporter:
              Jason Rutherglen
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:

                Development