Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.3
    • Fix Version/s: 1.3
    • Component/s: update
    • Labels:
      None

      Description

      We need a RequestHandler Which can import data from a DB or other dataSources into the Solr index .Think of it as an advanced form of SqlUpload Plugin (SOLR-103).

      The way it works is as follows.

      • Provide a configuration file (xml) to the Handler which takes in the necessary SQL queries and mappings to a solr schema
      • It also takes in a properties file for the data source configuraution
      • Given the configuration it can also generate the solr schema.xml
      • It is registered as a RequestHandler which can take two commands do-full-import, do-delta-import
      • do-full-import - dumps all the data from the Database into the index (based on the SQL query in configuration)
      • do-delta-import - dumps all the data that has changed since last import. (We assume a modified-timestamp column in tables)
      • It provides a admin page
      • where we can schedule it to be run automatically at regular intervals
      • It shows the status of the Handler (idle, full-import, delta-import)
      1. SOLR-469.patch
        336 kB
        Shalin Shekhar Mangar
      2. SOLR-469.patch
        324 kB
        Shalin Shekhar Mangar
      3. SOLR-469.patch
        230 kB
        Shalin Shekhar Mangar
      4. SOLR-469.patch
        182 kB
        Shalin Shekhar Mangar
      5. SOLR-469.patch
        182 kB
        Shalin Shekhar Mangar
      6. SOLR-469.patch
        377 kB
        Noble Paul
      7. SOLR-469.patch
        102 kB
        Shalin Shekhar Mangar
      8. SOLR-469.patch
        101 kB
        Shalin Shekhar Mangar
      9. SOLR-469.patch
        114 kB
        Shalin Shekhar Mangar
      10. SOLR-469-contrib.patch
        393 kB
        Shalin Shekhar Mangar
      11. SOLR-469-contrib.patch
        385 kB
        Noble Paul
      12. SOLR-469-contrib.patch
        382 kB
        Noble Paul
      13. SOLR-469-contrib.patch
        382 kB
        Noble Paul
      14. SOLR-469-contrib.patch
        380 kB
        Noble Paul
      15. SOLR-469-contrib.patch
        379 kB
        Noble Paul
      16. SOLR-469-contrib.patch
        368 kB
        Shalin Shekhar Mangar
      17. SOLR-469-contrib.patch
        756 kB
        Noble Paul
      18. SOLR-469-contrib.patch
        367 kB
        Shalin Shekhar Mangar
      19. SOLR-469-contrib.patch
        741 kB
        Noble Paul
      20. SOLR-469-contrib.patch
        349 kB
        Shalin Shekhar Mangar
      21. SOLR-469-contrib.patch
        337 kB
        Shalin Shekhar Mangar
      22. xpath-stream.patch
        10 kB
        Noble Paul

        Issue Links

          Activity

          Gavin made changes -
          Link This issue depends upon SOLR-563 [ SOLR-563 ]
          Gavin made changes -
          Link This issue depends on SOLR-563 [ SOLR-563 ]
          Noble Paul made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Noble Paul made changes -
          Attachment xpath-stream.patch [ 12387326 ]
          Shalin Shekhar Mangar made changes -
          Resolution Fixed [ 1 ]
          Status In Progress [ 3 ] Resolved [ 5 ]
          Shalin Shekhar Mangar made changes -
          Assignee Shalin Shekhar Mangar [ shalinmangar ]
          Grant Ingersoll made changes -
          Assignee Grant Ingersoll [ gsingers ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469-contrib.patch [ 12387197 ]
          Shalin Shekhar Mangar made changes -
          Comment [ A new patch containing the following changes:

          # On further thinking about Interface vs. Abstract classes, we have decided to replace all interfaces with abstract classes. Transformer, Context, EntityProcessor, Evaluator, DataSource and VariableResolver are now abstract classes.
          # The bug reported by Jonathan has been fixed and the TestCachedEntityProcessor has been updated to catch it. This exception used to be thrown only if the first request to CachedEntityProcessor needs a row which is not in cache. Subsequent requests were not affected.
          # Javadoc improvements. In particular, all the API related classes are marked as experimental and subject to change.
          # Propset Id in all classes.

          Users who have written their own custom transformers using the API will need to change their code. Sorry for the inconvenience.

          Grant - Is there anything else we need to do to get it committed? ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469.patch [ 12387196 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469.patch [ 12387196 ]
          Shalin Shekhar Mangar made changes -
          Comment [ A new patch containing the following changes:

          # On further thinking about Interface vs. Abstract classes, we have decided to replace all interfaces with abstract classes. Transformer, Context, EntityProcessor, Evaluator, DataSource and VariableResolver are now abstract classes.
          # The bug reported by Jonathan has been fixed and the TestCachedEntityProcessor has been updated to catch it. This exception used to be thrown only if the first request to CachedEntityProcessor needs a row which is not in cache. Subsequent requests were not affected.
          # Javadoc improvements. In particular, all the API related classes are marked as experimental and subject to change.
          # Propset Id in all classes.

          Users who have written their own custom transformers using the API will need to change their code. Sorry for the inconvenience.

          Grant -- Is there anything else we need to do to get it committed? ]
          Noble Paul made changes -
          Attachment SOLR-469-contrib.patch [ 12386744 ]
          Noble Paul made changes -
          Attachment SOLR-469-contrib.patch [ 12386707 ]
          Noble Paul made changes -
          Attachment SOLR-469-contrib.patch [ 12386704 ]
          Noble Paul made changes -
          Attachment SOLR-469-contrib.patch [ 12386598 ]
          Noble Paul made changes -
          Attachment SOLR-469-contrib.patch [ 12386599 ]
          Noble Paul made changes -
          Attachment SOLR-469-contrib.patch [ 12386598 ]
          Noble Paul made changes -
          Attachment SOLR-469-contrib.patch [ 12386267 ]
          Grant Ingersoll made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469-contrib.patch [ 12384594 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469.patch [ 12384592 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469.patch [ 12384592 ]
          Noble Paul made changes -
          Attachment SOLR-469-contrib.patch [ 12384480 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469-contrib.patch [ 12384054 ]
          Noble Paul made changes -
          Attachment SOLR-469-contrib.patch [ 12384004 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469-contrib.patch [ 12383857 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469.patch [ 12383854 ]
          Hoss Man made changes -
          Link This issue depends on SOLR-563 [ SOLR-563 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469-contrib.patch [ 12381499 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469.patch [ 12380679 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469.patch [ 12380002 ]
          Noble Paul made changes -
          Priority Minor [ 4 ] Major [ 3 ]
          Noble Paul made changes -
          Summary DB Import RequestHandler Data Import RequestHandler
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469.patch [ 12378899 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469.patch [ 12378868 ]
          Grant Ingersoll made changes -
          Assignee Grant Ingersoll [ gsingers ]
          Noble Paul made changes -
          Attachment SOLR-469.patch [ 12378135 ]
          Noble Paul made changes -
          Attachment SOLR-469.patch [ 12378134 ]
          Noble Paul made changes -
          Attachment SOLR-469.patch [ 12378128 ]
          Noble Paul made changes -
          Attachment SOLR-469.patch [ 12378134 ]
          Noble Paul made changes -
          Attachment SOLR-469.patch [ 12378128 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469.patch [ 12375753 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469.patch [ 12375183 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469.patch [ 12375147 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469.patch [ 12374882 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469.patch [ 12375149 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469.patch [ 12375147 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469.patch [ 12374882 ]
          Noble Paul made changes -
          Field Original Value New Value
          Link This issue is related to SOLR-103 [ SOLR-103 ]
          Noble Paul created issue -

            People

            • Assignee:
              Shalin Shekhar Mangar
              Reporter:
              Noble Paul
            • Votes:
              7 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development