[CHUKWA-700] Revisit Chukwa metrics schema design for HBase

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.6.0
• Component/s: Data Collection

Environment

MacOSX, Java

• Assignee: Unassigned
• Reporter: Eric Yang
• Created: 02/Nov/13 21:08
• Updated: 09/Nov/13 05:27

Description

Current Chukwa HBase schema looks like this:

<timestamp>-<primaryKey>   <columnFamily>:<cell>...

Monotonic increasing timestamp can not evenly distribute across region servers without special handle and care periodically.

It is time to revise the schema, and proposed schema looks like this:

<hhddmmyyyy>-<primaryId>  cf:<cell>...

Timestamp is stored with cell, row key helps to split data by hour, and a full hour of metrics is stored on the same row. PrimaryKey is replaced with hash id of the primary key. Metrics tables to aggregate metrics:

chukwaMetrics -> chukwaMetricsMonthly -> chukwaMetricsYearly

Sub-Tasks

• Support custom hbase schema: Open