HBase
  1. HBase
  2. HBASE-4368

Expose processlist in shell (per regionserver and perhaps by cluster)

    Details

    • Type: Task Task
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: shell
    • Labels:

      Description

      HBASE-4057 adds processlist and it shows in the RS UI. This issue is about getting the processlist to show in the shell, like it does in mysql.

      Labelling it noob; this is a pretty substantial issue but it shouldn't be too hard – it'd mostly be plumbing from RS into the shell.

      1. HBASE-4368.patch
        11 kB
        Shahin Saneinejad
      2. HBASE-4368v2.patch
        10 kB
        Talat UYARER
      3. HBASE-4368v2-withunittest.patch
        13 kB
        Talat UYARER
      4. HBASE-4368v3.patch
        14 kB
        Talat UYARER
      5. HBASE-4368v4.patch
        14 kB
        Talat UYARER

        Activity

        Hide
        Talat UYARER added a comment -

        Hi stack,

        Now it works for you request. It accept hostname, hostname and port, servername type strings.

        Thanks for reviewing.

        Show
        Talat UYARER added a comment - Hi stack , Now it works for you request. It accept hostname, hostname and port, servername type strings. Thanks for reviewing.
        Hide
        stack added a comment -

        This looks very nice.

        How do I query a single server?

        hbase(main):017:0> processlist 'all', 'c2024.halxg.cloudera.com,16020,1422392840119'

        ERROR: cannot convert instance of class org.jruby.RubyString to class org.apache.hadoop.hbase.ServerName

        The help is not clear. Says 'localhost':

        hbase> processlist 'all','localhost'

        Looks like it takes hostname only but but everywhere else we take servername so should say hostname instead of localhost. Might want to do servername rather than hostname or be able to take both for case where machine has restarted or there are two hosts on one node.

        Regards patch, it looks good. I like how you reuse formatter.

        Nice one Talat UYARER

        Show
        stack added a comment - This looks very nice. How do I query a single server? hbase(main):017:0> processlist 'all', 'c2024.halxg.cloudera.com,16020,1422392840119' ERROR: cannot convert instance of class org.jruby.RubyString to class org.apache.hadoop.hbase.ServerName The help is not clear. Says 'localhost': hbase> processlist 'all','localhost' Looks like it takes hostname only but but everywhere else we take servername so should say hostname instead of localhost. Might want to do servername rather than hostname or be able to take both for case where machine has restarted or there are two hosts on one node. Regards patch, it looks good. I like how you reuse formatter. Nice one Talat UYARER
        Hide
        stack added a comment -

        Let me play w/ it some more.

        Show
        stack added a comment - Let me play w/ it some more.
        Hide
        stack added a comment -

        That works:

        hbase(main):002:0> processlist
        1 tasks as of: 2015-01-26 22:35:35
        +-----------------+---------------------+----------+----------------------------------+--------------------------------------+
        | Host            | Start Time          | State    | Description                      | Status                               |
        +-----------------+---------------------+----------+----------------------------------+--------------------------------------+
        | c2022.halxg.... | 2015-01-26 22:34:09 | COMPLETE | Compacting meta in Integratio... | Compaction complete (since 57 sec... |
        +-----------------+---------------------+----------+----------------------------------+--------------------------------------+
        
        Show
        stack added a comment - That works: hbase(main):002:0> processlist 1 tasks as of: 2015-01-26 22:35:35 +-----------------+---------------------+----------+----------------------------------+--------------------------------------+ | Host | Start Time | State | Description | Status | +-----------------+---------------------+----------+----------------------------------+--------------------------------------+ | c2022.halxg.... | 2015-01-26 22:34:09 | COMPLETE | Compacting meta in Integratio... | Compaction complete (since 57 sec... | +-----------------+---------------------+----------+----------------------------------+--------------------------------------+
        Hide
        Talat UYARER added a comment -

        In shell I addressed to processlist command. Can you use processlist ?

        Show
        Talat UYARER added a comment - In shell I addressed to processlist command. Can you use processlist ?
        Hide
        stack added a comment -

        I installed the patch but I see nothing in the general help listing. Should I?

        hbase(main):010:0* help
        HBase Shell, version 2.0.0-SNAPSHOT, raaeafca9206341761094b1bcd27580f3978356bb, Mon Jan 26 22:27:55 PST 2015
        Type 'help "COMMAND"', (e.g. 'help "get"' -- the quotes are necessary) for help on a specific command.
        Commands are grouped. Type 'help "COMMAND_GROUP"', (e.g. 'help "general"') for help on a command group.
        
        COMMAND GROUPS:
          Group name: general
          Commands: processlist, status, table_help, version, whoami
        
          Group name: ddl
          Commands: alter, alter_async, alter_status, create, describe, disable, disable_all, drop, drop_all, enable, enable_all, exists, get_table, is_disabled, is_enabled, list, show_filters
        
          Group name: namespace
          Commands: alter_namespace, create_namespace, describe_namespace, drop_namespace, list_namespace, list_namespace_tables
        
          Group name: dml
          Commands: append, count, delete, deleteall, get, get_counter, incr, put, scan, truncate, truncate_preserve
        
          Group name: tools
          Commands: assign, balance_switch, balancer, catalogjanitor_enabled, catalogjanitor_run, catalogjanitor_switch, close_region, compact, compact_rs, flush, major_compact, merge_region, move, split, trace, unassign, wal_roll, zk_dump
        
          Group name: replication
          Commands: add_peer, append_peer_tableCFs, disable_peer, enable_peer, list_peers, list_replicated_tables, remove_peer, remove_peer_tableCFs, set_peer_tableCFs, show_peer_tableCFs
        
          Group name: snapshots
          Commands: clone_snapshot, delete_all_snapshot, delete_snapshot, list_snapshots, restore_snapshot, snapshot
        
          Group name: configuration
          Commands: update_all_config, update_config
        
          Group name: quotas
          Commands: list_quotas, set_quota
        
          Group name: security
          Commands: grant, revoke, user_permission
        
          Group name: visibility labels
          Commands: add_labels, clear_auths, get_auths, list_labels, set_auths, set_visibility
        

        I tried tasksOnHost... that don't work either. Am I doing it wrong?

        Show
        stack added a comment - I installed the patch but I see nothing in the general help listing. Should I? hbase(main):010:0* help HBase Shell, version 2.0.0-SNAPSHOT, raaeafca9206341761094b1bcd27580f3978356bb, Mon Jan 26 22:27:55 PST 2015 Type 'help "COMMAND" ', (e.g. 'help "get" ' -- the quotes are necessary) for help on a specific command. Commands are grouped. Type 'help "COMMAND_GROUP" ', (e.g. 'help "general" ') for help on a command group. COMMAND GROUPS: Group name: general Commands: processlist, status, table_help, version, whoami Group name: ddl Commands: alter, alter_async, alter_status, create, describe, disable, disable_all, drop, drop_all, enable, enable_all, exists, get_table, is_disabled, is_enabled, list, show_filters Group name: namespace Commands: alter_namespace, create_namespace, describe_namespace, drop_namespace, list_namespace, list_namespace_tables Group name: dml Commands: append, count, delete, deleteall, get, get_counter, incr, put, scan, truncate, truncate_preserve Group name: tools Commands: assign, balance_switch, balancer, catalogjanitor_enabled, catalogjanitor_run, catalogjanitor_switch, close_region, compact, compact_rs, flush, major_compact, merge_region, move, split, trace, unassign, wal_roll, zk_dump Group name: replication Commands: add_peer, append_peer_tableCFs, disable_peer, enable_peer, list_peers, list_replicated_tables, remove_peer, remove_peer_tableCFs, set_peer_tableCFs, show_peer_tableCFs Group name: snapshots Commands: clone_snapshot, delete_all_snapshot, delete_snapshot, list_snapshots, restore_snapshot, snapshot Group name: configuration Commands: update_all_config, update_config Group name: quotas Commands: list_quotas, set_quota Group name: security Commands: grant, revoke, user_permission Group name: visibility labels Commands: add_labels, clear_auths, get_auths, list_labels, set_auths, set_visibility I tried tasksOnHost... that don't work either. Am I doing it wrong?
        Hide
        Talat UYARER added a comment -

        Hi stack,

        Hope this is final patch for master branch. it is works in my environment. Can you review it ?

        BTW Enis Soztutar IMHO this is useful feature for shell. If everything is OK, can we add Hbase 1.0 ?

        Show
        Talat UYARER added a comment - Hi stack , Hope this is final patch for master branch. it is works in my environment. Can you review it ? BTW Enis Soztutar IMHO this is useful feature for shell. If everything is OK, can we add Hbase 1.0 ?
        Hide
        Talat UYARER added a comment -

        Hi Folks,

        I fixed it some problem and wrote a basic unit test. But I need open Regionserver Web ui of MiniHbaseCluster during unittest. I tried every ways however it does not open. Do you have any idea ?

        BTW Code is working on my computer But test fails because of the problem. I wait your reviews.

        Thanks

        Show
        Talat UYARER added a comment - Hi Folks, I fixed it some problem and wrote a basic unit test. But I need open Regionserver Web ui of MiniHbaseCluster during unittest. I tried every ways however it does not open. Do you have any idea ? BTW Code is working on my computer But test fails because of the problem. I wait your reviews. Thanks
        Hide
        Talat UYARER added a comment -

        i just converted to our current master. When i started to writing test, i realized it had some problem. Today i am planning this issue with fixed coding bug

        Show
        Talat UYARER added a comment - i just converted to our current master. When i started to writing test, i realized it had some problem. Today i am planning this issue with fixed coding bug
        Hide
        stack added a comment -

        I tried it and get this:

        hbase(main):010:0> processlist 'rpc'
        
        ERROR: undefined method `taskmonitor' for #<Hbase::Hbase:0x54ee5a15>
        

        Am I doing something wrong (doing it against master). Thanks Talat.

        Show
        stack added a comment - I tried it and get this: hbase(main):010:0> processlist 'rpc' ERROR: undefined method `taskmonitor' for #<Hbase::Hbase:0x54ee5a15> Am I doing something wrong (doing it against master). Thanks Talat.
        Hide
        stack added a comment -

        oh.... this looks nice. Let me try it out.

        Show
        stack added a comment - oh.... this looks nice. Let me try it out.
        Hide
        Talat UYARER added a comment -

        Sean Busbey, I started to work the issue. Now I am update patch for current master. I uploaded that version. I will write unit test for the path.

        Show
        Talat UYARER added a comment - Sean Busbey , I started to work the issue. Now I am update patch for current master. I uploaded that version. I will write unit test for the path.
        Hide
        Shahin Saneinejad added a comment -

        Sean Busbey: Nope, I'm no longer working with HBase. Thanks for the heads up.

        Show
        Shahin Saneinejad added a comment - Sean Busbey : Nope, I'm no longer working with HBase. Thanks for the heads up.
        Hide
        Sean Busbey added a comment -

        Shahin Saneinejad, are you still interested in working on this issue?

        If so, could you update it to work with current master?

        I'm planning on adding unit tests modelled on admin_test.rb as soon as I figure out how to run individual jruby unit tests (is there a maven option?).

        You can run just the shell tests with mvn -Dtest=TestShell test, but there is no option to only run a subset of them.

        Show
        Sean Busbey added a comment - Shahin Saneinejad , are you still interested in working on this issue? If so, could you update it to work with current master ? I'm planning on adding unit tests modelled on admin_test.rb as soon as I figure out how to run individual jruby unit tests (is there a maven option?). You can run just the shell tests with mvn -Dtest=TestShell test , but there is no option to only run a subset of them.
        Hide
        Shahin Saneinejad added a comment -

        Attached a patch, I'd appreciate any feedback.

        I'm planning on adding unit tests modelled on admin_test.rb as soon as I figure out how to run individual jruby unit tests (is there a maven option?).

        Show
        Shahin Saneinejad added a comment - Attached a patch, I'd appreciate any feedback. I'm planning on adding unit tests modelled on admin_test.rb as soon as I figure out how to run individual jruby unit tests (is there a maven option?).
        Hide
        stack added a comment -

        Is it called processlist in mysql? Should we use that name instead? Otherwise, 'tasks' sounds good to me. For the info port, you can read from Configuration the value of the int hbase.regionserver.info.port: i.e. conf.getInt("hbase.regionserver.info.port", 60030); Its not a guarantee that server out on cluster has something up on this port but its better than a hardcoded 60030.

        Show
        stack added a comment - Is it called processlist in mysql? Should we use that name instead? Otherwise, 'tasks' sounds good to me. For the info port, you can read from Configuration the value of the int hbase.regionserver.info.port: i.e. conf.getInt("hbase.regionserver.info.port", 60030); Its not a guarantee that server out on cluster has something up on this port but its better than a hardcoded 60030.
        Hide
        Shahin Saneinejad added a comment -

        Yeah, GET-ing JSON seems the most straightforward since regionservers already provide it. Let me know if there's a preferable alternative.

        Cluster status works for regionserver names, thanks for the suggestion. I'll assume Jetty is on the default port 60030 for now. I haven't found a good way to get the hbase.regionserver.info.port setting for each regionserver, yet.

        I imagine the command will look something like:

        > tasks
        [lists tasks using the default 'general' filter, on all regionservers]

        > tasks 'all'
        [lists tasks using 'all' filter, on all regionservers]

        > tasks 'all','rs-host'
        [lists tasks using 'all' filter, on the regionserver rs-host only]

        > tasks 'rs-host'
        [lists tasks using the default 'general' filter, on the regionserver rs-host only]

        Show
        Shahin Saneinejad added a comment - Yeah, GET-ing JSON seems the most straightforward since regionservers already provide it. Let me know if there's a preferable alternative. Cluster status works for regionserver names, thanks for the suggestion. I'll assume Jetty is on the default port 60030 for now. I haven't found a good way to get the hbase.regionserver.info.port setting for each regionserver, yet. I imagine the command will look something like: > tasks [lists tasks using the default 'general' filter, on all regionservers] > tasks 'all' [lists tasks using 'all' filter, on all regionservers] > tasks 'all','rs-host' [lists tasks using 'all' filter, on the regionserver rs-host only] > tasks 'rs-host' [lists tasks using the default 'general' filter, on the regionserver rs-host only]
        Hide
        stack added a comment -

        Go for it Shanin.

        You thinking of making an http GET asking for json from each regionserver?

        You may be able to get the list of regionservers by getting cluster status first and then iterate on the servers returned?

        Show
        stack added a comment - Go for it Shanin. You thinking of making an http GET asking for json from each regionserver? You may be able to get the list of regionservers by getting cluster status first and then iterate on the servers returned?
        Hide
        Shahin Saneinejad added a comment -

        Hi. I'd like to give this a shot. I think I qualify as a noob.

        My plan is to request a JSON task list from the endpoint defined by regionserver.info in the shell's HBase::HBase::configuration. A new tasks.rb command would retrieve a list of tasks and print them to the shell in a table, displaying the same information as in the RS HTML dashboard.

        Please let me know if this sounds crazy, or if it precludes any significant use cases. Thanks.

        Show
        Shahin Saneinejad added a comment - Hi. I'd like to give this a shot. I think I qualify as a noob. My plan is to request a JSON task list from the endpoint defined by regionserver.info in the shell's HBase::HBase::configuration. A new tasks.rb command would retrieve a list of tasks and print them to the shell in a table, displaying the same information as in the RS HTML dashboard. Please let me know if this sounds crazy, or if it precludes any significant use cases. Thanks.

          People

          • Assignee:
            Talat UYARER
            Reporter:
            stack
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development