Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.3
    • Fix Version/s: 1.3
    • Component/s: update
    • Labels:
      None

      Description

      We need a RequestHandler Which can import data from a DB or other dataSources into the Solr index .Think of it as an advanced form of SqlUpload Plugin (SOLR-103).

      The way it works is as follows.

      • Provide a configuration file (xml) to the Handler which takes in the necessary SQL queries and mappings to a solr schema
      • It also takes in a properties file for the data source configuraution
      • Given the configuration it can also generate the solr schema.xml
      • It is registered as a RequestHandler which can take two commands do-full-import, do-delta-import
      • do-full-import - dumps all the data from the Database into the index (based on the SQL query in configuration)
      • do-delta-import - dumps all the data that has changed since last import. (We assume a modified-timestamp column in tables)
      • It provides a admin page
      • where we can schedule it to be run automatically at regular intervals
      • It shows the status of the Handler (idle, full-import, delta-import)
      1. xpath-stream.patch
        10 kB
        Noble Paul
      2. SOLR-469-contrib.patch
        337 kB
        Shalin Shekhar Mangar
      3. SOLR-469-contrib.patch
        349 kB
        Shalin Shekhar Mangar
      4. SOLR-469-contrib.patch
        741 kB
        Noble Paul
      5. SOLR-469-contrib.patch
        367 kB
        Shalin Shekhar Mangar
      6. SOLR-469-contrib.patch
        756 kB
        Noble Paul
      7. SOLR-469-contrib.patch
        368 kB
        Shalin Shekhar Mangar
      8. SOLR-469-contrib.patch
        379 kB
        Noble Paul
      9. SOLR-469-contrib.patch
        380 kB
        Noble Paul
      10. SOLR-469-contrib.patch
        382 kB
        Noble Paul
      11. SOLR-469-contrib.patch
        382 kB
        Noble Paul
      12. SOLR-469-contrib.patch
        385 kB
        Noble Paul
      13. SOLR-469-contrib.patch
        393 kB
        Shalin Shekhar Mangar
      14. SOLR-469.patch
        114 kB
        Shalin Shekhar Mangar
      15. SOLR-469.patch
        101 kB
        Shalin Shekhar Mangar
      16. SOLR-469.patch
        102 kB
        Shalin Shekhar Mangar
      17. SOLR-469.patch
        377 kB
        Noble Paul
      18. SOLR-469.patch
        182 kB
        Shalin Shekhar Mangar
      19. SOLR-469.patch
        182 kB
        Shalin Shekhar Mangar
      20. SOLR-469.patch
        230 kB
        Shalin Shekhar Mangar
      21. SOLR-469.patch
        324 kB
        Shalin Shekhar Mangar
      22. SOLR-469.patch
        336 kB
        Shalin Shekhar Mangar

        Issue Links

          Activity

          Noble Paul created issue -
          Noble Paul made changes -
          Field Original Value New Value
          Link This issue is related to SOLR-103 [ SOLR-103 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469.patch [ 12374882 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469.patch [ 12375147 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469.patch [ 12375149 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469.patch [ 12374882 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469.patch [ 12375147 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469.patch [ 12375183 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469.patch [ 12375753 ]
          Noble Paul made changes -
          Attachment SOLR-469.patch [ 12378128 ]
          Noble Paul made changes -
          Attachment SOLR-469.patch [ 12378134 ]
          Noble Paul made changes -
          Attachment SOLR-469.patch [ 12378128 ]
          Noble Paul made changes -
          Attachment SOLR-469.patch [ 12378134 ]
          Noble Paul made changes -
          Attachment SOLR-469.patch [ 12378135 ]
          Grant Ingersoll made changes -
          Assignee Grant Ingersoll [ gsingers ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469.patch [ 12378868 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469.patch [ 12378899 ]
          Noble Paul made changes -
          Summary DB Import RequestHandler Data Import RequestHandler
          Noble Paul made changes -
          Priority Minor [ 4 ] Major [ 3 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469.patch [ 12380002 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469.patch [ 12380679 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469-contrib.patch [ 12381499 ]
          Hoss Man made changes -
          Link This issue depends on SOLR-563 [ SOLR-563 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469.patch [ 12383854 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469-contrib.patch [ 12383857 ]
          Noble Paul made changes -
          Attachment SOLR-469-contrib.patch [ 12384004 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469-contrib.patch [ 12384054 ]
          Noble Paul made changes -
          Attachment SOLR-469-contrib.patch [ 12384480 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469.patch [ 12384592 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469.patch [ 12384592 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469-contrib.patch [ 12384594 ]
          Grant Ingersoll made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Noble Paul made changes -
          Attachment SOLR-469-contrib.patch [ 12386267 ]
          Noble Paul made changes -
          Attachment SOLR-469-contrib.patch [ 12386598 ]
          Noble Paul made changes -
          Attachment SOLR-469-contrib.patch [ 12386599 ]
          Noble Paul made changes -
          Attachment SOLR-469-contrib.patch [ 12386598 ]
          Noble Paul made changes -
          Attachment SOLR-469-contrib.patch [ 12386704 ]
          Noble Paul made changes -
          Attachment SOLR-469-contrib.patch [ 12386707 ]
          Noble Paul made changes -
          Attachment SOLR-469-contrib.patch [ 12386744 ]
          Shalin Shekhar Mangar made changes -
          Comment [ A new patch containing the following changes:

          # On further thinking about Interface vs. Abstract classes, we have decided to replace all interfaces with abstract classes. Transformer, Context, EntityProcessor, Evaluator, DataSource and VariableResolver are now abstract classes.
          # The bug reported by Jonathan has been fixed and the TestCachedEntityProcessor has been updated to catch it. This exception used to be thrown only if the first request to CachedEntityProcessor needs a row which is not in cache. Subsequent requests were not affected.
          # Javadoc improvements. In particular, all the API related classes are marked as experimental and subject to change.
          # Propset Id in all classes.

          Users who have written their own custom transformers using the API will need to change their code. Sorry for the inconvenience.

          Grant -- Is there anything else we need to do to get it committed? ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469.patch [ 12387196 ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469.patch [ 12387196 ]
          Shalin Shekhar Mangar made changes -
          Comment [ A new patch containing the following changes:

          # On further thinking about Interface vs. Abstract classes, we have decided to replace all interfaces with abstract classes. Transformer, Context, EntityProcessor, Evaluator, DataSource and VariableResolver are now abstract classes.
          # The bug reported by Jonathan has been fixed and the TestCachedEntityProcessor has been updated to catch it. This exception used to be thrown only if the first request to CachedEntityProcessor needs a row which is not in cache. Subsequent requests were not affected.
          # Javadoc improvements. In particular, all the API related classes are marked as experimental and subject to change.
          # Propset Id in all classes.

          Users who have written their own custom transformers using the API will need to change their code. Sorry for the inconvenience.

          Grant - Is there anything else we need to do to get it committed? ]
          Shalin Shekhar Mangar made changes -
          Attachment SOLR-469-contrib.patch [ 12387197 ]
          Grant Ingersoll made changes -
          Assignee Grant Ingersoll [ gsingers ]
          Shalin Shekhar Mangar made changes -
          Assignee Shalin Shekhar Mangar [ shalinmangar ]
          Shalin Shekhar Mangar made changes -
          Resolution Fixed [ 1 ]
          Status In Progress [ 3 ] Resolved [ 5 ]
          Noble Paul made changes -
          Attachment xpath-stream.patch [ 12387326 ]
          Noble Paul made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Gavin made changes -
          Link This issue depends on SOLR-563 [ SOLR-563 ]
          Gavin made changes -
          Link This issue depends upon SOLR-563 [ SOLR-563 ]

            People

            • Assignee:
              Shalin Shekhar Mangar
              Reporter:
              Noble Paul
            • Votes:
              7 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development