Uploaded image for project: 'Chukwa'
  1. Chukwa
  2. CHUKWA-569

Add an option to speed up graph_explorer.jsp in fetching Column Name and Row Name

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.5.0
    • Component/s: User Interface
    • Labels:
      None
    • Environment:

      Java 6, MacOSX 10.6

      Description

      When select a column family, the only way to figure out all columns inside the column family is to do a scan. Since HBase does not have an API to get column names only, we have to provide a temporary workaround. We can add an option to do full table scan, and we scan small amount of data to figure out the column patterns by default. This short cut make assumption that data are in repeated pattern, scanning one row should be sufficient to find all the columns. The same principle applies in scanning for unique row names.

        Activity

        Hide
        asrabkin Ari Rabkin added a comment -

        I think that assumption is usually a safe one – particularly if the scan only covers, say, the most recent 20 rows.

        What about explicitly keeping a separate metadata table, either in HBase or in Zookeeper?

        Show
        asrabkin Ari Rabkin added a comment - I think that assumption is usually a safe one – particularly if the scan only covers, say, the most recent 20 rows. What about explicitly keeping a separate metadata table, either in HBase or in Zookeeper?
        Hide
        eyang Eric Yang added a comment -

        A separate metadata table to keep track of the columns is a better implementation than this jira implies. There are two missing pieces for the recommendation to work. We need a workflow scheduler to run periodically, and the mapreduce job to scan for extra columns with specified table name and time range.

        For the first one, workflow scheduler system, we can use Oozie workflow scheduler. For mapreduce which scan for columns, and update the meta data table, I filed CHUKWA-570.

        Show
        eyang Eric Yang added a comment - A separate metadata table to keep track of the columns is a better implementation than this jira implies. There are two missing pieces for the recommendation to work. We need a workflow scheduler to run periodically, and the mapreduce job to scan for extra columns with specified table name and time range. For the first one, workflow scheduler system, we can use Oozie workflow scheduler. For mapreduce which scan for columns, and update the meta data table, I filed CHUKWA-570 .
        Hide
        eyang Eric Yang added a comment -

        This is a temporary fix to speed up graph_explorer until we have an aggregation system to scan and extract all columns and store in metadata table.

        Show
        eyang Eric Yang added a comment - This is a temporary fix to speed up graph_explorer until we have an aggregation system to scan and extract all columns and store in metadata table.
        Hide
        eyang Eric Yang added a comment -

        I just committed this.

        Show
        eyang Eric Yang added a comment - I just committed this.

          People

          • Assignee:
            eyang Eric Yang
            Reporter:
            eyang Eric Yang
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development