Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Won't Fix
    • Affects Version/s: 1.4
    • Fix Version/s: 3.2
    • Component/s: search
    • Labels:
      None

      Description

      What's the use case? Sometimes queries are expensive (such as
      regex) or one has indexes located in HDFS, that then need to be
      searched on. By leveraging Hadoop, these non-time sensitive
      queries may be executed without dynamically deploying the
      indexes to new Solr servers.

      We'll download the index out of HDFS (assuming they're zipped),
      perform the queries in a batch on the index shard, then merge
      the results either using a Solr query results priority queue, or
      simply using Hadoop's built in merge sorting.

      The query file will be encoded in JSON format, (ID, query,
      numresults,fields). The shards file will simply contain newline
      delimited paths (HDFS or otherwise). The output can be a Solr
      encoded results file per query.

      I'm hoping to add an actual Hadoop unit test.

        Activity

        Jason Rutherglen created issue -
        Hide
        Andrzej Bialecki added a comment -

        If query performance is not a concern, then why not execute it directly on HDFS (using e.g. Nutch FsDirectory to read indexes from HDFS)?

        Show
        Andrzej Bialecki added a comment - If query performance is not a concern, then why not execute it directly on HDFS (using e.g. Nutch FsDirectory to read indexes from HDFS)?
        Hide
        Hoss Man added a comment -

        Bulk updating 240 Solr issues to set the Fix Version to "next" per the process outlined in this email...

        http://mail-archives.apache.org/mod_mbox/lucene-dev/201005.mbox/%3Calpine.DEB.1.10.1005251052040.24672@radix.cryptio.net%3E

        Selection criteria was "Unresolved" with a Fix Version of 1.5, 1.6, 3.1, or 4.0. email notifications were suppressed.

        A unique token for finding these 240 issues in the future: hossversioncleanup20100527

        Show
        Hoss Man added a comment - Bulk updating 240 Solr issues to set the Fix Version to "next" per the process outlined in this email... http://mail-archives.apache.org/mod_mbox/lucene-dev/201005.mbox/%3Calpine.DEB.1.10.1005251052040.24672@radix.cryptio.net%3E Selection criteria was "Unresolved" with a Fix Version of 1.5, 1.6, 3.1, or 4.0. email notifications were suppressed. A unique token for finding these 240 issues in the future: hossversioncleanup20100527
        Hoss Man made changes -
        Field Original Value New Value
        Fix Version/s Next [ 12315093 ]
        Fix Version/s 1.5 [ 12313566 ]
        Hide
        Jason Rutherglen added a comment -

        Sorry if this spam's things, however it's unlikely that I'll work on these.

        Show
        Jason Rutherglen added a comment - Sorry if this spam's things, however it's unlikely that I'll work on these.
        Jason Rutherglen made changes -
        Status Open [ 1 ] Closed [ 6 ]
        Resolution Won't Fix [ 2 ]
        Hoss Man made changes -
        Fix Version/s 3.2 [ 12316172 ]
        Fix Version/s Next [ 12315093 ]

          People

          • Assignee:
            Unassigned
            Reporter:
            Jason Rutherglen
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development