Solr
  1. Solr
  2. SOLR-3648

The "/browse" Solritas GUI does not work under SolrCloud - 500 error

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 4.0-ALPHA
    • Fix Version/s: 4.0-BETA, 6.0
    • Labels:
    • Environment:

      RHEL 5.4, SolrCloud

      Description

      When run in a SolrCloud configuration the Velocity searchHandler responds to a /browse request with an HTTP error 500 and a stack trace indicating org.apache.solr.common.cloud.ZooKeeperException:
      ZkSolrResourceLoader does not support getConfigDir()

      1. SOLR-3648.patch
        5 kB
        Erik Hatcher
      2. SOLR-3648.patch
        5 kB
        Jan Høydahl

        Activity

        Hide
        Jan Høydahl added a comment -

        Looks as if we in general need more interaction tests between SolrCloud and various other Solr features. I set it as Blocker for now, since it is probably a quick fix..

        Show
        Jan Høydahl added a comment - Looks as if we in general need more interaction tests between SolrCloud and various other Solr features. I set it as Blocker for now, since it is probably a quick fix..
        Hide
        Mark Miller added a comment -

        There are a variety of extras that may not work with SolrCloud - I don't consider any of them blockers though. Initially, I would only expect core stuff to work well until others start digging into other components. I have never looked at how the UIMA update processor plays with SolrCloud as an example. Unless a lot of people start asking about something like that, it's just not high on my priority list.

        Of course, a lot of this could be done by anybody.

        I look at this the same as some search components not yet supporting distrib. Some things won't work with SolrCloud to begin with as well. I've focused my attention on the core features.

        Currently, anything anywhere that tries to get the local fs directory will fail when in SolrCloud mode. In the cases that you can accomplish the same thing using zookeeper, another code path can be added. I've done this for a couple other things. In some cases it might not be so easy though.

        That exception is telling you, in cloud mode, this feature is not currently support - perhaps we should tweak it to be more explicit about that.

        Since I'm not familiar with browse or what it tries to do internally, it's not high on my personal list, which is quite long.

        Show
        Mark Miller added a comment - There are a variety of extras that may not work with SolrCloud - I don't consider any of them blockers though. Initially, I would only expect core stuff to work well until others start digging into other components. I have never looked at how the UIMA update processor plays with SolrCloud as an example. Unless a lot of people start asking about something like that, it's just not high on my priority list. Of course, a lot of this could be done by anybody. I look at this the same as some search components not yet supporting distrib. Some things won't work with SolrCloud to begin with as well. I've focused my attention on the core features. Currently, anything anywhere that tries to get the local fs directory will fail when in SolrCloud mode. In the cases that you can accomplish the same thing using zookeeper, another code path can be added. I've done this for a couple other things. In some cases it might not be so easy though. That exception is telling you, in cloud mode, this feature is not currently support - perhaps we should tweak it to be more explicit about that. Since I'm not familiar with browse or what it tries to do internally, it's not high on my personal list, which is quite long.
        Hide
        Mark Miller added a comment -

        In this case I assume the problem is in the VelocityResponseWriter.

            VelocityEngine engine = new VelocityEngine();
            String template_root = request.getParams().get("v.base_dir");
            File baseDir = new File(request.getCore().getResourceLoader().getConfigDir(), "velocity");
            if (template_root != null) {
              baseDir = new File(template_root);
            }
            engine.setProperty(RuntimeConstants.FILE_RESOURCE_LOADER_PATH, baseDir.getAbsolutePath());
            engine.setProperty("params.resource.loader.instance", new SolrParamResourceLoader(request));
            SolrVelocityResourceLoader resourceLoader =
                new SolrVelocityResourceLoader(request.getCore().getSolrConfig().getResourceLoader());
            engine.setProperty("solr.resource.loader.instance", resourceLoader);
        

        It looks like its asking for the location of config files - they are in zookeeper - but this 'engine' wants a file path. Looks like some difficulties...

        Show
        Mark Miller added a comment - In this case I assume the problem is in the VelocityResponseWriter. VelocityEngine engine = new VelocityEngine(); String template_root = request.getParams().get( "v.base_dir" ); File baseDir = new File(request.getCore().getResourceLoader().getConfigDir(), "velocity" ); if (template_root != null ) { baseDir = new File(template_root); } engine.setProperty(RuntimeConstants.FILE_RESOURCE_LOADER_PATH, baseDir.getAbsolutePath()); engine.setProperty( "params.resource.loader.instance" , new SolrParamResourceLoader(request)); SolrVelocityResourceLoader resourceLoader = new SolrVelocityResourceLoader(request.getCore().getSolrConfig().getResourceLoader()); engine.setProperty( "solr.resource.loader.instance" , resourceLoader); It looks like its asking for the location of config files - they are in zookeeper - but this 'engine' wants a file path. Looks like some difficulties...
        Hide
        Jan Høydahl added a comment -

        Attaching first patch

        • Not using Velocity's file.resource.loader
        • Param v.base_dir now is always relative to "conf" and respected by SolrResourceLoader
        • SolrVelocityResourceLoader first checks the given paths from v.base_dir, default "velocity" before checking default locations
        • ShowFileRequestHandler#showFromZooKeeper patched to remove "/" prefix from requested file so that e.g. /velocity/main.css resolves for ZK.

        Still missing are some unit tests. Also perhaps a way to load files/resources outside of current core's "conf" via ZK. But ".." is disallowed, so how would this work? Perhaps we need a way/syntax to address files from another config or folder than the current, a way that works across all ResourceLoaders...

        Show
        Jan Høydahl added a comment - Attaching first patch Not using Velocity's file.resource.loader Param v.base_dir now is always relative to "conf" and respected by SolrResourceLoader SolrVelocityResourceLoader first checks the given paths from v.base_dir, default "velocity" before checking default locations ShowFileRequestHandler#showFromZooKeeper patched to remove "/" prefix from requested file so that e.g. /velocity/main.css resolves for ZK. Still missing are some unit tests. Also perhaps a way to load files/resources outside of current core's "conf" via ZK. But ".." is disallowed, so how would this work? Perhaps we need a way/syntax to address files from another config or folder than the current, a way that works across all ResourceLoaders...
        Hide
        Jan Høydahl added a comment -

        Would it be a good idea to let resource loaders support a simple URI syntax?

        myfile.txt                           # relative to conf (whether in FS or ZK)
        file:///var/myfile.txt               # absolute file from local file system
        http://localhost/myfile.txt          # absolute URL
        zk://solr/configs/otherconfig/myfile # absolute ZK ref, to e.g. reference a synonym dict residing in a shared config
        
        Show
        Jan Høydahl added a comment - Would it be a good idea to let resource loaders support a simple URI syntax? myfile.txt # relative to conf (whether in FS or ZK) file:///var/myfile.txt # absolute file from local file system http://localhost/myfile.txt # absolute URL zk://solr/configs/otherconfig/myfile # absolute ZK ref, to e.g. reference a synonym dict residing in a shared config
        Hide
        Erik Hatcher added a comment -

        Jan - thanks for your efforts here. I've carved out a couple of days (local Lucene/Solr Hackathon) later this week to work on Solr 4/trunk issues and will be looking into this one in more detail then.

        At first glance, I don't think we should lose the file resource loader completely, as it can be a very useful way to make sets of templates that are overridable. But for a first pass to get this working with SolrCloud properly I'm ok with it. Ultimately (and I'll aim to dig into this more myself soon) I think we should make the template resource loading settings pluggable/editable from configuration.

        Show
        Erik Hatcher added a comment - Jan - thanks for your efforts here. I've carved out a couple of days (local Lucene/Solr Hackathon) later this week to work on Solr 4/trunk issues and will be looking into this one in more detail then. At first glance, I don't think we should lose the file resource loader completely, as it can be a very useful way to make sets of templates that are overridable. But for a first pass to get this working with SolrCloud properly I'm ok with it. Ultimately (and I'll aim to dig into this more myself soon) I think we should make the template resource loading settings pluggable/editable from configuration.
        Hide
        Jan Høydahl added a comment -

        Erik, feel free to grab the current patch and polish it to your likings.

        A disadvantage is that we have very little test coverage. We could of course use something like Selenium...

        Show
        Jan Høydahl added a comment - Erik, feel free to grab the current patch and polish it to your likings. A disadvantage is that we have very little test coverage. We could of course use something like Selenium...
        Hide
        Erik Hatcher added a comment -

        Adapted Jan's work with some tweaks. Hard-coded "velocity/" in the custom resource loader (which is what gets used in SolrCloud mode). Kept file resource loader in the mix when not in SolrCloud mode.

        All is working now, so I'll commit and iterate on anything else remaining here.

        Show
        Erik Hatcher added a comment - Adapted Jan's work with some tweaks. Hard-coded "velocity/" in the custom resource loader (which is what gets used in SolrCloud mode). Kept file resource loader in the mix when not in SolrCloud mode. All is working now, so I'll commit and iterate on anything else remaining here.
        Hide
        Erik Hatcher added a comment -

        Committed my patch to r1366775 (4_x) and r1366776 (trunk).

        Show
        Erik Hatcher added a comment - Committed my patch to r1366775 (4_x) and r1366776 (trunk).
        Hide
        Erik Hatcher added a comment -

        TODO: update VelocityResponseWriter wiki (it needs a cleanup anyway) with pertinent details.

        What else do you think should be done here, Jan? The one thing I didn't incorporate from your patch was the "path" parameter to the SolrVelocityResourceLoader (but rather hardcoded it), mainly for simplicity. I don't think that multiple "path"'s capability should be done that way here. Either your way or my way here is a change in how it was done before by requiring it look in a velocity/ sub"directory". I doubt anyone has used it the way I was intending it to be used, so not a big deal to break it I don't think. My thinking with that, thankfully quite useful in SolrCloud mode resource loader, was that plugins/contrib modules could ship their own Velocity templates in their JAR files. Anyone using it that way will now have to put the templates in a velocity/ subdirectory inside the JAR.

        Show
        Erik Hatcher added a comment - TODO: update VelocityResponseWriter wiki (it needs a cleanup anyway) with pertinent details. What else do you think should be done here, Jan? The one thing I didn't incorporate from your patch was the "path" parameter to the SolrVelocityResourceLoader (but rather hardcoded it), mainly for simplicity. I don't think that multiple "path"'s capability should be done that way here. Either your way or my way here is a change in how it was done before by requiring it look in a velocity/ sub"directory". I doubt anyone has used it the way I was intending it to be used, so not a big deal to break it I don't think. My thinking with that, thankfully quite useful in SolrCloud mode resource loader, was that plugins/contrib modules could ship their own Velocity templates in their JAR files. Anyone using it that way will now have to put the templates in a velocity/ subdirectory inside the JAR.
        Hide
        Jan Høydahl added a comment -

        I'm cool with that, thanks for finalizing!

        Show
        Jan Høydahl added a comment - I'm cool with that, thanks for finalizing!

          People

          • Assignee:
            Erik Hatcher
            Reporter:
            Nick Cotton
          • Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development