Solr
  1. Solr
  2. SOLR-1197

memcached implement solr cache for queryresultCache

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 1.3
    • Fix Version/s: 4.9, 5.0
    • Component/s: None
    • Labels:
      None
    • Environment:

      multiple slave solr replication environment

      Description

      multiple slave create query result together, and slaves can share that.

      implement memcached cache instead LRUCache

      my implement:

      solr-memcache.zip http://solr-side.googlecode.com/files/solr-memcache.zip

      object transport by net need Serializable, so need patch solr 1.3, DocSetBase implements Serializable, see http://code.google.com/p/solr-side/issues/detail?id=1&can=1

      1. SOLR_1197-solr-memcache.patch
        0.7 kB
        Linbin Chen
      2. SOLR_1197-solr-memcache-0.2.patch
        1 kB
        Linbin Chen

        Activity

        Hide
        Linbin Chen added a comment - - edited

        patch for Solr 1.3

        this patch only DocSetBase implement Serializable

        solr-memcache

        =====================readme.txt=========================

        MemcachedCache instead of solr queryresultCache (default LRUCache)

        config in solrconfig.xml to use solr-memcache

        add newSearcher and firstSearcher Listener, such as:

        <listener event="newSearcher" class="solr.MemcachedCache" />
        <listener event="firstSearcher" class="solr.MemcachedCache" />

        use listener only for get index version, to create memcached key

        indexVersion is static long field of MemcachedCache.java.

        //originalKey is QueryResultKey
        memcached key = keyPrefix+indexVersion+"-"+originalKey.hashCode()

        <!--

        MemcachedCache params:

        memcachedHosts (required), "," split.
        name (optional) no default.
        expTime (optional) default 1800 s (= 30 minute)
        defaultPort (optional) default 11211
        keyPrefix (optional) default ""

        -->

        <queryResultCache

        class="solr.MemcachedCache"
        memcachedHosts="192.168.0.100,192.168.0.101:1234,192.168.0.103"
        expTime="21600"
        defaultPort="11511"
        keyPrefix="shard-1-"/>

        dep jar:

        memcached-2.2.jar
        spy-2.4.jar

        solr-memcache.patch for solr 1.3

        if download and unzip to d:/apache-solr-1.3.0

        copy patch-build.xml and solr-memcache.patch to (d:/apache-solr-1.3.0)

        D:\apache-solr-1.3.0>ant -f patch-build.xml -Dpatch.file=solr-memcache.patch
        Buildfile: patch-build.xml

        apply-patch:
        [patch] patching file src/java/org/apache/solr/search/DocSet.java

        BUILD SUCCESSFUL
        Total time: 0 seconds

        if exist d:/apache-solr-1.3.0/contrib/solr-memcache, if no exist you can unzip solr-memcache.zip to that dir

        and dist

        D:\apache-solr-1.3.0>ant dist
        ...

        look D:\apache-solr-1.3.0\dist\apache-solr-memcache-1.3.0.jar

        Show
        Linbin Chen added a comment - - edited patch for Solr 1.3 this patch only DocSetBase implement Serializable solr-memcache =====================readme.txt========================= MemcachedCache instead of solr queryresultCache (default LRUCache) config in solrconfig.xml to use solr-memcache add newSearcher and firstSearcher Listener, such as: <listener event="newSearcher" class="solr.MemcachedCache" /> <listener event="firstSearcher" class="solr.MemcachedCache" /> use listener only for get index version, to create memcached key indexVersion is static long field of MemcachedCache.java. //originalKey is QueryResultKey memcached key = keyPrefix+indexVersion+"-"+originalKey.hashCode() <!-- MemcachedCache params: memcachedHosts (required), "," split. name (optional) no default. expTime (optional) default 1800 s (= 30 minute) defaultPort (optional) default 11211 keyPrefix (optional) default "" --> <queryResultCache class="solr.MemcachedCache" memcachedHosts="192.168.0.100,192.168.0.101:1234,192.168.0.103" expTime="21600" defaultPort="11511" keyPrefix="shard-1-"/> dep jar: memcached-2.2.jar spy-2.4.jar solr-memcache.patch for solr 1.3 if download and unzip to d:/apache-solr-1.3.0 copy patch-build.xml and solr-memcache.patch to (d:/apache-solr-1.3.0) D:\apache-solr-1.3.0>ant -f patch-build.xml -Dpatch.file=solr-memcache.patch Buildfile: patch-build.xml apply-patch: [patch] patching file src/java/org/apache/solr/search/DocSet.java BUILD SUCCESSFUL Total time: 0 seconds if exist d:/apache-solr-1.3.0/contrib/solr-memcache, if no exist you can unzip solr-memcache.zip to that dir and dist D:\apache-solr-1.3.0>ant dist ... look D:\apache-solr-1.3.0\dist\apache-solr-memcache-1.3.0.jar
        Hide
        Noble Paul added a comment -

        moving to 1.5. it is not a good idea to accept new features for 1.4

        Show
        Noble Paul added a comment - moving to 1.5. it is not a good idea to accept new features for 1.4
        Hide
        Linbin Chen added a comment -

        solr sint hashcode conflict cause solr memcached bug

        solr-memcached-0.2

        solr sint fieldType sint.hashcode conflict very hight above 99%

        such as id:44232808 and id:44090533 hashcode conflict, so q=id:44232808
        maybe response id:44090533 doc result. error!

        http://code.google.com/p/solr-side/issues/detail?id=2&can=1
        solr-memcached-0.2 patch for create memcache key strategy:

        QueryResultKey add method

        public String toString()

        { return query.toString(); }

        for MemcachedCache md5(QueryResultKey.toString())

        now memcache key strategy = md5(keyPrefix+indexVersion+QueryResultKey.hashCode()+md5(QueryResultKey.toString()))

        double md5

        solr-memcache-0.2.zip http://solr-side.googlecode.com/files/solr-memcache-0.2.zip

        Show
        Linbin Chen added a comment - solr sint hashcode conflict cause solr memcached bug solr-memcached-0.2 solr sint fieldType sint.hashcode conflict very hight above 99% such as id:44232808 and id:44090533 hashcode conflict, so q=id:44232808 maybe response id:44090533 doc result. error! http://code.google.com/p/solr-side/issues/detail?id=2&can=1 solr-memcached-0.2 patch for create memcache key strategy: QueryResultKey add method public String toString() { return query.toString(); } for MemcachedCache md5(QueryResultKey.toString()) now memcache key strategy = md5(keyPrefix+indexVersion+QueryResultKey.hashCode()+md5(QueryResultKey.toString())) double md5 solr-memcache-0.2.zip http://solr-side.googlecode.com/files/solr-memcache-0.2.zip
        Hide
        Hoss Man added a comment -

        Bulk updating 240 Solr issues to set the Fix Version to "next" per the process outlined in this email...

        http://mail-archives.apache.org/mod_mbox/lucene-dev/201005.mbox/%3Calpine.DEB.1.10.1005251052040.24672@radix.cryptio.net%3E

        Selection criteria was "Unresolved" with a Fix Version of 1.5, 1.6, 3.1, or 4.0. email notifications were suppressed.

        A unique token for finding these 240 issues in the future: hossversioncleanup20100527

        Show
        Hoss Man added a comment - Bulk updating 240 Solr issues to set the Fix Version to "next" per the process outlined in this email... http://mail-archives.apache.org/mod_mbox/lucene-dev/201005.mbox/%3Calpine.DEB.1.10.1005251052040.24672@radix.cryptio.net%3E Selection criteria was "Unresolved" with a Fix Version of 1.5, 1.6, 3.1, or 4.0. email notifications were suppressed. A unique token for finding these 240 issues in the future: hossversioncleanup20100527
        Hide
        Robert Muir added a comment -

        Bulk move 3.2 -> 3.3

        Show
        Robert Muir added a comment - Bulk move 3.2 -> 3.3
        Hide
        Robert Muir added a comment -

        3.4 -> 3.5

        Show
        Robert Muir added a comment - 3.4 -> 3.5
        Hide
        Hoss Man added a comment -

        bulk fixing the version info for 4.0-ALPHA and 4.0 all affected issues have "hoss20120711-bulk-40-change" in comment

        Show
        Hoss Man added a comment - bulk fixing the version info for 4.0-ALPHA and 4.0 all affected issues have "hoss20120711-bulk-40-change" in comment
        Hide
        Robert Muir added a comment -

        rmuir20120906-bulk-40-change

        Show
        Robert Muir added a comment - rmuir20120906-bulk-40-change
        Hide
        Robert Muir added a comment -

        moving all 4.0 issues not touched in a month to 4.1

        Show
        Robert Muir added a comment - moving all 4.0 issues not touched in a month to 4.1
        Hide
        Steve Rowe added a comment -

        Bulk move 4.4 issues to 4.5 and 5.0

        Show
        Steve Rowe added a comment - Bulk move 4.4 issues to 4.5 and 5.0
        Hide
        Uwe Schindler added a comment -

        Move issue to Solr 4.9.

        Show
        Uwe Schindler added a comment - Move issue to Solr 4.9.

          People

          • Assignee:
            Noble Paul
            Reporter:
            Linbin Chen
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:

              Development