Resume progress on this issue. We have learnt a few lessons on metrics schema design in the last couple years. Monotonic increasing row key is bad for HBase region server. We have built an alternate HBase schema, like this:
[group name].[date].[metric]:[primary key], and column family "m" and cell name "m". This provides a way to prefix split of regions, but it also isn't great. The advantage is to have one table for all type of metrics. We observed that some group may have metrics growing faster than other group, therefore, the region split still need a lot of manual maintenance to prevent HBase from blowing up.
A new proposal is to change metrics schema design to partition by day of the month. More than often that time series database have two requirements, fast lookup by time and fast lookup for the same metrics. This means the row key need to have hints for partition by day and partition by primary key. A improved schema can be generated by:
Table: [group name]
Row Key: [day:primary_key]
Column Family: [subgroup name]
Column: [metric name]
Timestamp: [actual timestamp]
Example of a Hadoop table would look like:
Row Key: 13:host1.example.com
Column Family: HDFS
Units, and metrics type can be stored in a secondary table for rendering and metadata lookup to reduce storage space. Thoughts?