Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.5.0
    • Component/s: Data Processors
    • Labels:
      None
    • Environment:

      Redhat EL 5.1, Java 6

    • Release Note:
      Added HBaseWriter for storing time series data in HBase for faster random read/write.

      Description

      The current Chukwa Record format is not suitable for data visualization. It is more like an archive format which combines data from multiple sources (hosts), and group them into a sorted time partitioned sequence file. Most of people collected data for two reasons, archive and data analysis. The current chukwa record format is fine for archive, but it is not so great for data analysis. Data analysis could be further break down into two different types. 1) Data can be aggregated and summarized, such as metrics. 2) Data that can not be summarized, like job history. Type 1 data is useful for visualization by graph, and type 2 data is useful by plain text viewing or search for a particular event.

      By the above rational, it probably makes sense to restructure Chukwa Records for data analysis. Outside of Hadoop world, rrdtools is great for time series data storage, and optimized for metrics from a single source, i.e. a host. RRD data file fragments badly when there are hundred of thousands of sources. Chukwa time series data storage should be able to combine multiple data sources into one Chukwa file to combat file fragmentation problem.

        Issue Links

          Activity

          Hide
          eyang Eric Yang added a comment -

          I just committed this, thanks Ari.

          Show
          eyang Eric Yang added a comment - I just committed this, thanks Ari.
          Hide
          asrabkin Ari Rabkin added a comment -

          OK. Read through this. +1 to commit.

          Show
          asrabkin Ari Rabkin added a comment - OK. Read through this. +1 to commit.
          Hide
          eyang Eric Yang added a comment -

          The current patch is ready to check in if people are fine with [Time partition]-[primary key] approach.

          Show
          eyang Eric Yang added a comment - The current patch is ready to check in if people are fine with [Time partition] - [primary key] approach.
          Hide
          eyang Eric Yang added a comment -

          Ideally RowKey can use better serialization library like Avro, but it also seems excessive to bake avro schema into part of the RowKey. Hence, the current implementation is:

          [time partition]-[primary key]
          

          The user can choose to bake his own dimensions into the primary key section.

          If we use avro, it will look like:

          {
            "type" : "record",
            "name" : "ChukwaKey",
            "namespace" : "org.apache.chukwa.ipc",
            "fields" : [
              { "name" : "timePartition", "type" : "long" },
              { "name" : "cluster", "type" : "string" },
              { "name" : "host", "type" : "string" }
            ]
          },{ ... binary data ...}
          

          Using Avro maybe better at type checking, but each row key has the same schema repeated over and over. It doesn't seem efficient for storage and serialization of the key. One benefit to go with avro, is it provides better handling of flexible key dimensions. Using string split, it is more efficient for serving data to HICC. What is the preferred method for the community?

          Show
          eyang Eric Yang added a comment - Ideally RowKey can use better serialization library like Avro, but it also seems excessive to bake avro schema into part of the RowKey. Hence, the current implementation is: [time partition]-[primary key] The user can choose to bake his own dimensions into the primary key section. If we use avro, it will look like: { "type" : "record", "name" : "ChukwaKey", "namespace" : "org.apache.chukwa.ipc", "fields" : [ { "name" : "timePartition", "type" : "long" }, { "name" : "cluster", "type" : "string" }, { "name" : "host", "type" : "string" } ] },{ ... binary data ...} Using Avro maybe better at type checking, but each row key has the same schema repeated over and over. It doesn't seem efficient for storage and serialization of the key. One benefit to go with avro, is it provides better handling of flexible key dimensions. Using string split, it is more efficient for serving data to HICC. What is the preferred method for the community?
          Hide
          eyang Eric Yang added a comment -

          Include test case for HBaseWriter.

          Show
          eyang Eric Yang added a comment - Include test case for HBaseWriter.
          Hide
          eyang Eric Yang added a comment -

          The test case depends on availability of MiniHBaseCluster.

          Show
          eyang Eric Yang added a comment - The test case depends on availability of MiniHBaseCluster.
          Hide
          eyang Eric Yang added a comment -

          Revised patch to include:

          • Added annotations for working demux parsers.
          • Added pipelineWriter support, HBaseWriter can be use with other writers.
          • Support for using HTablePool to save data sink to different table by demux parser class annotation.
          Show
          eyang Eric Yang added a comment - Revised patch to include: Added annotations for working demux parsers. Added pipelineWriter support, HBaseWriter can be use with other writers. Support for using HTablePool to save data sink to different table by demux parser class annotation.
          Hide
          eyang Eric Yang added a comment -

          The current patch has limitation of using a single hbase table as data sink. Ideally, the table name and row key can be specified by the demux parser class via annotation. This will ensure backward compatibility and provide an easy method to structure data for hbase.

          Show
          eyang Eric Yang added a comment - The current patch has limitation of using a single hbase table as data sink. Ideally, the table name and row key can be specified by the demux parser class via annotation. This will ensure backward compatibility and provide an easy method to structure data for hbase.
          Hide
          eyang Eric Yang added a comment -

          HBase writer is optional, and it is currently compatible with other components of Chukwa data processing pipeline. In the future, if HBaseWriter is voted to be the standard, then we can optimize the demux parser to be more efficient for HBaseWriter. There is no stricted schema as long as parser must be loaded ahead of time and parser's ReducerType must match the column family in HBase table. HBase table will also help on range scan to retrieve data for hourly and daily report of map reduce job. This approach is more efficient than periodically roll up and manages spill files.

          If hbase is not available, it throws WriterException, hence, the chunk is uncommitted. It should not have any impact to collector other than reject the chunk.

          I like to update Chukwa architecture to reflect this change. This may help us steer clear of having relational database on the critical path.

          Show
          eyang Eric Yang added a comment - HBase writer is optional, and it is currently compatible with other components of Chukwa data processing pipeline. In the future, if HBaseWriter is voted to be the standard, then we can optimize the demux parser to be more efficient for HBaseWriter. There is no stricted schema as long as parser must be loaded ahead of time and parser's ReducerType must match the column family in HBase table. HBase table will also help on range scan to retrieve data for hourly and daily report of map reduce job. This approach is more efficient than periodically roll up and manages spill files. If hbase is not available, it throws WriterException, hence, the chunk is uncommitted. It should not have any impact to collector other than reject the chunk. I like to update Chukwa architecture to reflect this change. This may help us steer clear of having relational database on the critical path.
          Hide
          billgraham Bill Graham added a comment -

          I agree with Jerome, in that I think Chukwa should still be able to be used without HBase. If you have an HBase install and want real-time, it can be enabled. Ideally we would have the ability to configurable which data pipeline to follow. I like the idea of adding an HBase component to the mix though.

          Show
          billgraham Bill Graham added a comment - I agree with Jerome, in that I think Chukwa should still be able to be used without HBase. If you have an HBase install and want real-time, it can be enabled. Ideally we would have the ability to configurable which data pipeline to follow. I like the idea of adding an HBase component to the mix though.
          Hide
          asrabkin Ari Rabkin added a comment -

          Overall HBase-based approach seems to make sense.

          I think this might be a good time to do something we've been talking about for a while – unifying the writer and sender APIs. This unification would also advance Jerome's SDK vision, I think, by making the architecture more flexible.

          Show
          asrabkin Ari Rabkin added a comment - Overall HBase-based approach seems to make sense. I think this might be a good time to do something we've been talking about for a while – unifying the writer and sender APIs. This unification would also advance Jerome's SDK vision, I think, by making the architecture more flexible.
          Hide
          jboulon Jerome Boulon added a comment -

          Eric,

          • Details on memory/cpu usage if HBase not available?
          • Can you give more information on the HBase usage?
            --> are you sending everything to HBase
            --> schema
            --> queries
            --> latency ... etc

          Also, Demux as a M/R is very important for a lot of people as HBase is just for near real time analytics and not for hourly/daily reports so why do we want to change the architecture?
          Like I've said before, I think that chukwa should be an SDK where you can take/use any part as you need to build your own system. That's seems to be what you want and I'm fine with that but we also need to get all big parts to work together.

          So is it an architecture change (incompatible with the current demux) or a different assembly in order to get the near real time?

          Thanks,
          /Jerome.

          Show
          jboulon Jerome Boulon added a comment - Eric, Details on memory/cpu usage if HBase not available? Can you give more information on the HBase usage? --> are you sending everything to HBase --> schema --> queries --> latency ... etc Also, Demux as a M/R is very important for a lot of people as HBase is just for near real time analytics and not for hourly/daily reports so why do we want to change the architecture? Like I've said before, I think that chukwa should be an SDK where you can take/use any part as you need to build your own system. That's seems to be what you want and I'm fine with that but we also need to get all big parts to work together. So is it an architecture change (incompatible with the current demux) or a different assembly in order to get the near real time? Thanks, /Jerome.
          Hide
          eyang Eric Yang added a comment -

          Also writerClass in chukwa-collector.xml:

            <property>
              <name>chukwaCollector.writerClass</name>
              <value>org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter</value>
            </property>
          
          Show
          eyang Eric Yang added a comment - Also writerClass in chukwa-collector.xml: <property> <name>chukwaCollector.writerClass</name> <value>org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter</value> </property>
          Hide
          eyang Eric Yang added a comment -

          You will need hbase jar file in chukwa/lib directory and define sink table in chukwa-collector.xml like:

            <property>
              <name>data.sink.table</name>
              <value>chukwa</value>
            </property>
          
          Show
          eyang Eric Yang added a comment - You will need hbase jar file in chukwa/lib directory and define sink table in chukwa-collector.xml like: <property> <name>data.sink.table</name> <value>chukwa</value> </property>
          Hide
          eyang Eric Yang added a comment -

          The current data sink structure is suboptimal for visualizing data in near real time. Hence, I purpose to modify the demux parser to work on data inside collector. When data is arriving in the data sink, Hbase acts as the micro time series data indexer for small incremental updates. Other reporting jobs could run in the background as mapreduce jobs to organize data as described in previous comment. Data inside of hbase is immediately available for REST API like stargate and visualization tool HICC.

          The pipeline would look like this:

          Adaptor -> Agent -> Collector -> HbaseWriter -> DemuxParsers -> Hbase +-> Stargate  --> HICC (For monitoring)
                                                                                +-> Mapreduce --> Hbase --> Stargate --> HICC (For reporting)
          

          Existing structure still works, but will be marked as deprecated:

          Adaptor -> Agent -> Collector -> SeqFileWriter -> HDFS +-> Archives
                                                                 +-> Demux -> Database -> HICC (For reporting)
          

          It will also be possible to combine both combination to get archives to work in parallel as the new structure:

          Adaptor -> Agent -> Collector +-> PipelineWriter
                                        +-> SeqFileWriter -> HDFS -> Archive
                                        +-> HbaseWriter   -> DemuxParsers -> Hbase +-> Stargate  --> HICC (For monitoring)
                                                                                   +-> Mapreduce --> Hbase --> Stargate --> HICC (For reporting)
          
          Show
          eyang Eric Yang added a comment - The current data sink structure is suboptimal for visualizing data in near real time. Hence, I purpose to modify the demux parser to work on data inside collector. When data is arriving in the data sink, Hbase acts as the micro time series data indexer for small incremental updates. Other reporting jobs could run in the background as mapreduce jobs to organize data as described in previous comment. Data inside of hbase is immediately available for REST API like stargate and visualization tool HICC. The pipeline would look like this: Adaptor -> Agent -> Collector -> HbaseWriter -> DemuxParsers -> Hbase +-> Stargate --> HICC (For monitoring) +-> Mapreduce --> Hbase --> Stargate --> HICC (For reporting) Existing structure still works, but will be marked as deprecated: Adaptor -> Agent -> Collector -> SeqFileWriter -> HDFS +-> Archives +-> Demux -> Database -> HICC (For reporting) It will also be possible to combine both combination to get archives to work in parallel as the new structure: Adaptor -> Agent -> Collector +-> PipelineWriter +-> SeqFileWriter -> HDFS -> Archive +-> HbaseWriter -> DemuxParsers -> Hbase +-> Stargate --> HICC (For monitoring) +-> Mapreduce --> Hbase --> Stargate --> HICC (For reporting)
          Hide
          eyang Eric Yang added a comment -

          More refined plan:

          Type 1 Data:

          Having a post demux data loader which wait to receive new ChukwaRecords files, and merge with the existing ChukwaRecords files through a second MR
          job. The second MR job also produces low resolution of the data for report.

          /chukwa/repos/TYPE/DATE <-- Original data goes here.
          /chukwa/report/TYPE/[yearly,monthly,weekly,daily] <-- Summarized JSON data goes here.

          The report JSON will be fixed to 300 data points per series, optimized for graphing.

          Type 2 data for plain text searching:

          After data has been archived, use full body indexer like lucene to build searchable indexes.

          Architecture look like this:

          Adaptor -> Agent -> Collector |-> Archive -> Full Body Index |-> Retention
                                        +-> Demux   -> Aggregation     |-> Retention
                                                                       +-> Hicc
          
          Show
          eyang Eric Yang added a comment - More refined plan: Type 1 Data: Having a post demux data loader which wait to receive new ChukwaRecords files, and merge with the existing ChukwaRecords files through a second MR job. The second MR job also produces low resolution of the data for report. /chukwa/repos/TYPE/DATE <-- Original data goes here. /chukwa/report/TYPE/ [yearly,monthly,weekly,daily] <-- Summarized JSON data goes here. The report JSON will be fixed to 300 data points per series, optimized for graphing. Type 2 data for plain text searching: After data has been archived, use full body indexer like lucene to build searchable indexes. Architecture look like this: Adaptor -> Agent -> Collector |-> Archive -> Full Body Index |-> Retention +-> Demux -> Aggregation |-> Retention +-> Hicc

            People

            • Assignee:
              eyang Eric Yang
              Reporter:
              eyang Eric Yang
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development