Solr
  1. Solr
  2. SOLR-1970

need to customize location of dataimport.properties

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.4
    • Fix Version/s: 4.1, 5.0
    • Labels:
      None

      Description

      By default dataimport.properties is written to

      {solr.home}

      /conf/. However when using multiple solr cores, it is currently useful to use the same conf directory for all of the cores and use solr.xml to specify a different schema.xml. I can then specify a different data-config.xml for each core to define how the data gets from the database to each core's shema.

      However, all the solr cores will fight over writing to the dataimport.properties file. There should be an option in solrconfig.xml to specify the location or name of this file so that a different one can be used for each core.

        Issue Links

          Activity

          Hide
          David Smiley added a comment -

          This is a great point! My annoyance is that this file is created in the conf directory when I think the data directory would be a more suitable location. The conf directory might not be writable, and the file reflects the current indexed state and so what better place to put it than the data directory?

          Show
          David Smiley added a comment - This is a great point! My annoyance is that this file is created in the conf directory when I think the data directory would be a more suitable location. The conf directory might not be writable, and the file reflects the current indexed state and so what better place to put it than the data directory?
          Hide
          Andrea Polci added a comment -

          +1 to David: the data directory is a better place for dataimport.properties

          Show
          Andrea Polci added a comment - +1 to David: the data directory is a better place for dataimport.properties
          Hide
          Terrance A. Snyder added a comment - - edited

          +1 I am looking at implementing with current trunk. I'll hopefully submit a patch to this. Keep in mind this may have to be added to the core configuration area to keep backward compatibility.

          Something like:

          <core name="users_001" instanceDir="users" config="solrconfig.xml" dataDir="../users_001" dataImportPropertiesFile="../users_001/di.properties"/>
          <core name="users_002" instanceDir="users" config="solrconfig.xml" dataDir="../users_002" dataImportPropertiesFile="../users_002/di.properties"/>

          This would operation just like todays core config:

          $SOLR_HOME/users
          /users/conf
          /users/conf/solrconfig.xml
          /users/conf/schema.xml
          /users_001
          /users_001/data
          /users_001/di.properties
          /users_002
          /users_002/data
          /users_002/di.properties

          This allows the core configuration and sharding to work effectively. The core question is how this would play with zookeeper / cloud support.

          I would think this should already be baked into SolrCloud.... but I could be wrong. Any thoughts?

          Show
          Terrance A. Snyder added a comment - - edited +1 I am looking at implementing with current trunk. I'll hopefully submit a patch to this. Keep in mind this may have to be added to the core configuration area to keep backward compatibility. Something like: <core name="users_001" instanceDir="users" config="solrconfig.xml" dataDir="../users_001" dataImportPropertiesFile="../users_001/di.properties"/> <core name="users_002" instanceDir="users" config="solrconfig.xml" dataDir="../users_002" dataImportPropertiesFile="../users_002/di.properties"/> This would operation just like todays core config: $SOLR_HOME/users /users/conf /users/conf/solrconfig.xml /users/conf/schema.xml /users_001 /users_001/data /users_001/di.properties /users_002 /users_002/data /users_002/di.properties This allows the core configuration and sharding to work effectively. The core question is how this would play with zookeeper / cloud support. I would think this should already be baked into SolrCloud.... but I could be wrong. Any thoughts?
          Hide
          Guillaume Belrose added a comment -

          +1. I came across this issue today, and being able to spin new cores at run time, each with a dedicated .properties files would be very useful for my use case.

          Show
          Guillaume Belrose added a comment - +1. I came across this issue today, and being able to spin new cores at run time, each with a dedicated .properties files would be very useful for my use case.
          Hide
          Erik Andersson added a comment -

          We just solved this issue by not using $

          {dataimporter.last_index_time}

          for delta-imports. Instead we store the 'last_index_time' (one for each core) in a DB table and use a sub-query to pull it out.

          SELECT
          ...
          WHERE
          date_modified > (SELECT `value` FROM `meta` WHERE `key`='solr.dataimporter.last_index_time')

          We run delta imports from a script so we can simply update the 'last_index_time' in that process (add a sleep() to make sure delta kicks in before the 'last_index_time' is updated). I'm sure there are better solutions that could trigger the update within Solr.

          Show
          Erik Andersson added a comment - We just solved this issue by not using $ {dataimporter.last_index_time} for delta-imports. Instead we store the 'last_index_time' (one for each core) in a DB table and use a sub-query to pull it out. SELECT ... WHERE date_modified > (SELECT `value` FROM `meta` WHERE `key`='solr.dataimporter.last_index_time') We run delta imports from a script so we can simply update the 'last_index_time' in that process (add a sleep() to make sure delta kicks in before the 'last_index_time' is updated). I'm sure there are better solutions that could trigger the update within Solr.
          Hide
          James Dyer added a comment -

          Fixed as part of SOLR-4051.

          This adds a <propertyWriter /> element to DIH's data-config.xml file, allowing the user to specify the location, filename and Locale for the "data-config.properties" file. Alternatively, users can specify their own property writer implementation for greater control.

          Show
          James Dyer added a comment - Fixed as part of SOLR-4051 . This adds a <propertyWriter /> element to DIH's data-config.xml file, allowing the user to specify the location, filename and Locale for the "data-config.properties" file. Alternatively, users can specify their own property writer implementation for greater control.

            People

            • Assignee:
              James Dyer
              Reporter:
              Chris Book
            • Votes:
              10 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development