Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.6.0
    • Fix Version/s: 0.6.0
    • Component/s: Data Collection
    • Labels:
      None

      Description

      Solr Cloud can create full text index and query through REST API. This is useful to index Hadoop logs for root cause analysis. In the event that Hadoop cluster is unavailable, Solr Cloud will continue to have Hadoop logs searchable to provide a quicker way to analyze the problem than ssh to multiple nodes to find the logs.

      1. CHUKWA-722.patch
        426 kB
        Eric Yang
      2. CHUKWA-722-1.patch
        426 kB
        Eric Yang

        Activity

        Hide
        eyang Eric Yang added a comment -
        • Added SolrWriter to mirror data to Solr in realtime.
        • Integration contains Solr instance and schema configuration files.
        • See README.txt file in contrib/solr directory for instruction of how to setup Solr Cloud for Chukwa.
        Show
        eyang Eric Yang added a comment - Added SolrWriter to mirror data to Solr in realtime. Integration contains Solr instance and schema configuration files. See README.txt file in contrib/solr directory for instruction of how to setup Solr Cloud for Chukwa.
        Hide
        sree2k Sreepathi Prasanna added a comment -

        Eric,

        a. What exactly will be stored in HBase by this writer ?
        b. A minor correction in SolrWriter.java. should this be SolrWriter.class ?

        private static Logger log = Logger.getLogger(HBaseWriter.class);

        Show
        sree2k Sreepathi Prasanna added a comment - Eric, a. What exactly will be stored in HBase by this writer ? b. A minor correction in SolrWriter.java. should this be SolrWriter.class ? private static Logger log = Logger.getLogger(HBaseWriter.class);
        Hide
        eyang Eric Yang added a comment -

        Good catch. New patch updated HBase reference to reflect Solr.

        Show
        eyang Eric Yang added a comment - Good catch. New patch updated HBase reference to reflect Solr.
        Hide
        shreyass123 shreyas subramanya added a comment -

        +1

        Show
        shreyass123 shreyas subramanya added a comment - +1
        Hide
        eyang Eric Yang added a comment -

        Thanks Shreyas, I just committed this.

        Show
        eyang Eric Yang added a comment - Thanks Shreyas, I just committed this.
        Hide
        hudson Hudson added a comment -

        ABORTED: Integrated in Chukwa-trunk #507 (See https://builds.apache.org/job/Chukwa-trunk/507/)
        CHUKWA-722. Added SolrWriter to stream data to SolrCloud. (Eric Yang) (eyang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1614808)

        • /chukwa/trunk/CHANGES.txt
        • /chukwa/trunk/conf/chukwa-agent-conf.xml
        • /chukwa/trunk/contrib/solr
        • /chukwa/trunk/contrib/solr/logs
        • /chukwa/trunk/contrib/solr/logs/README.txt
        • /chukwa/trunk/contrib/solr/logs/conf
        • /chukwa/trunk/contrib/solr/logs/conf/_schema_analysis_stopwords_english.json
        • /chukwa/trunk/contrib/solr/logs/conf/_schema_analysis_synonyms_english.json
        • /chukwa/trunk/contrib/solr/logs/conf/admin-extra.html
        • /chukwa/trunk/contrib/solr/logs/conf/admin-extra.menu-bottom.html
        • /chukwa/trunk/contrib/solr/logs/conf/admin-extra.menu-top.html
        • /chukwa/trunk/contrib/solr/logs/conf/clustering
        • /chukwa/trunk/contrib/solr/logs/conf/clustering/carrot2
        • /chukwa/trunk/contrib/solr/logs/conf/clustering/carrot2/kmeans-attributes.xml
        • /chukwa/trunk/contrib/solr/logs/conf/clustering/carrot2/lingo-attributes.xml
        • /chukwa/trunk/contrib/solr/logs/conf/clustering/carrot2/stc-attributes.xml
        • /chukwa/trunk/contrib/solr/logs/conf/currency.xml
        • /chukwa/trunk/contrib/solr/logs/conf/elevate.xml
        • /chukwa/trunk/contrib/solr/logs/conf/lang
        • /chukwa/trunk/contrib/solr/logs/conf/lang/contractions_ca.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/contractions_fr.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/contractions_ga.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/contractions_it.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/hyphenations_ga.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/stemdict_nl.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/stoptags_ja.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_ar.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_bg.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_ca.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_ckb.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_cz.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_da.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_de.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_el.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_en.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_es.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_eu.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_fa.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_fi.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_fr.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_ga.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_gl.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_hi.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_hu.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_hy.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_id.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_it.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_ja.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_lv.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_nl.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_no.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_pt.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_ro.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_ru.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_sv.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_th.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_tr.txt
        • /chukwa/trunk/contrib/solr/logs/conf/lang/userdict_ja.txt
        • /chukwa/trunk/contrib/solr/logs/conf/mapping-FoldToASCII.txt
        • /chukwa/trunk/contrib/solr/logs/conf/mapping-ISOLatin1Accent.txt
        • /chukwa/trunk/contrib/solr/logs/conf/protwords.txt
        • /chukwa/trunk/contrib/solr/logs/conf/schema.xml
        • /chukwa/trunk/contrib/solr/logs/conf/scripts.conf
        • /chukwa/trunk/contrib/solr/logs/conf/solrconfig.xml
        • /chukwa/trunk/contrib/solr/logs/conf/spellings.txt
        • /chukwa/trunk/contrib/solr/logs/conf/stopwords.txt
        • /chukwa/trunk/contrib/solr/logs/conf/synonyms.txt
        • /chukwa/trunk/contrib/solr/logs/conf/update-script.js
        • /chukwa/trunk/contrib/solr/logs/conf/velocity
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/README.txt
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/VM_global_library.vm
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/browse.vm
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/cluster.vm
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/cluster_results.vm
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/debug.vm
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/did_you_mean.vm
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/error.vm
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/facet_fields.vm
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/facet_pivot.vm
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/facet_queries.vm
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/facet_ranges.vm
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/facets.vm
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/footer.vm
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/head.vm
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/header.vm
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/hit.vm
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/hit_grouped.vm
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/hit_plain.vm
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/join_doc.vm
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/jquery.autocomplete.css
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/jquery.autocomplete.js
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/layout.vm
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/main.css
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/mime_type_lists.vm
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/pagination_bottom.vm
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/pagination_top.vm
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/product_doc.vm
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/query.vm
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/query_form.vm
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/query_group.vm
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/query_spatial.vm
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/results_list.vm
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/richtext_doc.vm
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/suggest.vm
        • /chukwa/trunk/contrib/solr/logs/conf/velocity/tabs.vm
        • /chukwa/trunk/contrib/solr/logs/conf/xslt
        • /chukwa/trunk/contrib/solr/logs/conf/xslt/example.xsl
        • /chukwa/trunk/contrib/solr/logs/conf/xslt/example_atom.xsl
        • /chukwa/trunk/contrib/solr/logs/conf/xslt/example_rss.xsl
        • /chukwa/trunk/contrib/solr/logs/conf/xslt/luke.xsl
        • /chukwa/trunk/contrib/solr/logs/conf/xslt/updateXml.xsl
        • /chukwa/trunk/contrib/solr/logs/core.properties
        • /chukwa/trunk/pom.xml
        • /chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/writer/solr
        • /chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/writer/solr/SolrWriter.java
        • /chukwa/trunk/src/packages/tarball/all.xml
        • /chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/datacollection/writer/solr
        • /chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/datacollection/writer/solr/TestSolrWriter.java
        Show
        hudson Hudson added a comment - ABORTED: Integrated in Chukwa-trunk #507 (See https://builds.apache.org/job/Chukwa-trunk/507/ ) CHUKWA-722 . Added SolrWriter to stream data to SolrCloud. (Eric Yang) (eyang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1614808 ) /chukwa/trunk/CHANGES.txt /chukwa/trunk/conf/chukwa-agent-conf.xml /chukwa/trunk/contrib/solr /chukwa/trunk/contrib/solr/logs /chukwa/trunk/contrib/solr/logs/README.txt /chukwa/trunk/contrib/solr/logs/conf /chukwa/trunk/contrib/solr/logs/conf/_schema_analysis_stopwords_english.json /chukwa/trunk/contrib/solr/logs/conf/_schema_analysis_synonyms_english.json /chukwa/trunk/contrib/solr/logs/conf/admin-extra.html /chukwa/trunk/contrib/solr/logs/conf/admin-extra.menu-bottom.html /chukwa/trunk/contrib/solr/logs/conf/admin-extra.menu-top.html /chukwa/trunk/contrib/solr/logs/conf/clustering /chukwa/trunk/contrib/solr/logs/conf/clustering/carrot2 /chukwa/trunk/contrib/solr/logs/conf/clustering/carrot2/kmeans-attributes.xml /chukwa/trunk/contrib/solr/logs/conf/clustering/carrot2/lingo-attributes.xml /chukwa/trunk/contrib/solr/logs/conf/clustering/carrot2/stc-attributes.xml /chukwa/trunk/contrib/solr/logs/conf/currency.xml /chukwa/trunk/contrib/solr/logs/conf/elevate.xml /chukwa/trunk/contrib/solr/logs/conf/lang /chukwa/trunk/contrib/solr/logs/conf/lang/contractions_ca.txt /chukwa/trunk/contrib/solr/logs/conf/lang/contractions_fr.txt /chukwa/trunk/contrib/solr/logs/conf/lang/contractions_ga.txt /chukwa/trunk/contrib/solr/logs/conf/lang/contractions_it.txt /chukwa/trunk/contrib/solr/logs/conf/lang/hyphenations_ga.txt /chukwa/trunk/contrib/solr/logs/conf/lang/stemdict_nl.txt /chukwa/trunk/contrib/solr/logs/conf/lang/stoptags_ja.txt /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_ar.txt /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_bg.txt /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_ca.txt /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_ckb.txt /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_cz.txt /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_da.txt /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_de.txt /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_el.txt /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_en.txt /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_es.txt /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_eu.txt /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_fa.txt /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_fi.txt /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_fr.txt /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_ga.txt /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_gl.txt /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_hi.txt /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_hu.txt /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_hy.txt /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_id.txt /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_it.txt /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_ja.txt /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_lv.txt /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_nl.txt /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_no.txt /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_pt.txt /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_ro.txt /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_ru.txt /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_sv.txt /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_th.txt /chukwa/trunk/contrib/solr/logs/conf/lang/stopwords_tr.txt /chukwa/trunk/contrib/solr/logs/conf/lang/userdict_ja.txt /chukwa/trunk/contrib/solr/logs/conf/mapping-FoldToASCII.txt /chukwa/trunk/contrib/solr/logs/conf/mapping-ISOLatin1Accent.txt /chukwa/trunk/contrib/solr/logs/conf/protwords.txt /chukwa/trunk/contrib/solr/logs/conf/schema.xml /chukwa/trunk/contrib/solr/logs/conf/scripts.conf /chukwa/trunk/contrib/solr/logs/conf/solrconfig.xml /chukwa/trunk/contrib/solr/logs/conf/spellings.txt /chukwa/trunk/contrib/solr/logs/conf/stopwords.txt /chukwa/trunk/contrib/solr/logs/conf/synonyms.txt /chukwa/trunk/contrib/solr/logs/conf/update-script.js /chukwa/trunk/contrib/solr/logs/conf/velocity /chukwa/trunk/contrib/solr/logs/conf/velocity/README.txt /chukwa/trunk/contrib/solr/logs/conf/velocity/VM_global_library.vm /chukwa/trunk/contrib/solr/logs/conf/velocity/browse.vm /chukwa/trunk/contrib/solr/logs/conf/velocity/cluster.vm /chukwa/trunk/contrib/solr/logs/conf/velocity/cluster_results.vm /chukwa/trunk/contrib/solr/logs/conf/velocity/debug.vm /chukwa/trunk/contrib/solr/logs/conf/velocity/did_you_mean.vm /chukwa/trunk/contrib/solr/logs/conf/velocity/error.vm /chukwa/trunk/contrib/solr/logs/conf/velocity/facet_fields.vm /chukwa/trunk/contrib/solr/logs/conf/velocity/facet_pivot.vm /chukwa/trunk/contrib/solr/logs/conf/velocity/facet_queries.vm /chukwa/trunk/contrib/solr/logs/conf/velocity/facet_ranges.vm /chukwa/trunk/contrib/solr/logs/conf/velocity/facets.vm /chukwa/trunk/contrib/solr/logs/conf/velocity/footer.vm /chukwa/trunk/contrib/solr/logs/conf/velocity/head.vm /chukwa/trunk/contrib/solr/logs/conf/velocity/header.vm /chukwa/trunk/contrib/solr/logs/conf/velocity/hit.vm /chukwa/trunk/contrib/solr/logs/conf/velocity/hit_grouped.vm /chukwa/trunk/contrib/solr/logs/conf/velocity/hit_plain.vm /chukwa/trunk/contrib/solr/logs/conf/velocity/join_doc.vm /chukwa/trunk/contrib/solr/logs/conf/velocity/jquery.autocomplete.css /chukwa/trunk/contrib/solr/logs/conf/velocity/jquery.autocomplete.js /chukwa/trunk/contrib/solr/logs/conf/velocity/layout.vm /chukwa/trunk/contrib/solr/logs/conf/velocity/main.css /chukwa/trunk/contrib/solr/logs/conf/velocity/mime_type_lists.vm /chukwa/trunk/contrib/solr/logs/conf/velocity/pagination_bottom.vm /chukwa/trunk/contrib/solr/logs/conf/velocity/pagination_top.vm /chukwa/trunk/contrib/solr/logs/conf/velocity/product_doc.vm /chukwa/trunk/contrib/solr/logs/conf/velocity/query.vm /chukwa/trunk/contrib/solr/logs/conf/velocity/query_form.vm /chukwa/trunk/contrib/solr/logs/conf/velocity/query_group.vm /chukwa/trunk/contrib/solr/logs/conf/velocity/query_spatial.vm /chukwa/trunk/contrib/solr/logs/conf/velocity/results_list.vm /chukwa/trunk/contrib/solr/logs/conf/velocity/richtext_doc.vm /chukwa/trunk/contrib/solr/logs/conf/velocity/suggest.vm /chukwa/trunk/contrib/solr/logs/conf/velocity/tabs.vm /chukwa/trunk/contrib/solr/logs/conf/xslt /chukwa/trunk/contrib/solr/logs/conf/xslt/example.xsl /chukwa/trunk/contrib/solr/logs/conf/xslt/example_atom.xsl /chukwa/trunk/contrib/solr/logs/conf/xslt/example_rss.xsl /chukwa/trunk/contrib/solr/logs/conf/xslt/luke.xsl /chukwa/trunk/contrib/solr/logs/conf/xslt/updateXml.xsl /chukwa/trunk/contrib/solr/logs/core.properties /chukwa/trunk/pom.xml /chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/writer/solr /chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/writer/solr/SolrWriter.java /chukwa/trunk/src/packages/tarball/all.xml /chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/datacollection/writer/solr /chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/datacollection/writer/solr/TestSolrWriter.java

          People

          • Assignee:
            eyang Eric Yang
            Reporter:
            eyang Eric Yang
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development