[CHUKWA-749] Redefine Solr facets for Hadoop log monitoring

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.7.0
• Component/s: Data Collection

Environment

Linux, Java

• Assignee: Eric Yang
• Reporter: Eric Yang
• Created: 31/May/15 00:21
• Updated: 31/May/15 00:38

Description

For improving the user experience of analyze big data environment log files, we can use Solr to organize all log files in a logical order. Log files of big data environment are generally divided into application and system categories.

In Application perspective, the structure of an application look like this:

Application
  user
  coordinator-job
    date
    workflow
      action
        jobs
          scheduler (app master)
          tasks
            task-attempts
              container
                host

In system perspective,

System
  cluster
    service
      component
        host
          container

The two perspectives are join by host. If we have Application perspective and System perspective in Solr cores. We can join the activities happened in Application with System using Solr.

Sub-Tasks

• Create Oozie Solr core schema.xml: Open
• Create YARN Solr core schema.xml: Open
• Create System Solr schema: Open
• Inject tagging facility before chunk is sent to Solr: Resolved
• Add a reverse proxy to solr: Resolved
• Update Chukwa Solr UI integration: Resolved
• Supply example log4j.properties to stream Hadoop logs to Chukwa: Resolved