Chukwa
  1. Chukwa
  2. CHUKWA-569

Add an option to speed up graph_explorer.jsp in fetching Column Name and Row Name

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.5.0
    • Component/s: User Interface
    • Labels:
      None
    • Environment:

      Java 6, MacOSX 10.6

      Description

      When select a column family, the only way to figure out all columns inside the column family is to do a scan. Since HBase does not have an API to get column names only, we have to provide a temporary workaround. We can add an option to do full table scan, and we scan small amount of data to figure out the column patterns by default. This short cut make assumption that data are in repeated pattern, scanning one row should be sufficient to find all the columns. The same principle applies in scanning for unique row names.

        Activity

        Hide
        Ari Rabkin added a comment -

        I think that assumption is usually a safe one – particularly if the scan only covers, say, the most recent 20 rows.

        What about explicitly keeping a separate metadata table, either in HBase or in Zookeeper?

        Show
        Ari Rabkin added a comment - I think that assumption is usually a safe one – particularly if the scan only covers, say, the most recent 20 rows. What about explicitly keeping a separate metadata table, either in HBase or in Zookeeper?
        Hide
        Eric Yang added a comment -

        A separate metadata table to keep track of the columns is a better implementation than this jira implies. There are two missing pieces for the recommendation to work. We need a workflow scheduler to run periodically, and the mapreduce job to scan for extra columns with specified table name and time range.

        For the first one, workflow scheduler system, we can use Oozie workflow scheduler. For mapreduce which scan for columns, and update the meta data table, I filed CHUKWA-570.

        Show
        Eric Yang added a comment - A separate metadata table to keep track of the columns is a better implementation than this jira implies. There are two missing pieces for the recommendation to work. We need a workflow scheduler to run periodically, and the mapreduce job to scan for extra columns with specified table name and time range. For the first one, workflow scheduler system, we can use Oozie workflow scheduler. For mapreduce which scan for columns, and update the meta data table, I filed CHUKWA-570 .
        Hide
        Eric Yang added a comment -

        This is a temporary fix to speed up graph_explorer until we have an aggregation system to scan and extract all columns and store in metadata table.

        Show
        Eric Yang added a comment - This is a temporary fix to speed up graph_explorer until we have an aggregation system to scan and extract all columns and store in metadata table.
        Hide
        Eric Yang added a comment -

        I just committed this.

        Show
        Eric Yang added a comment - I just committed this.

          People

          • Assignee:
            Eric Yang
            Reporter:
            Eric Yang
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development