Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: Impala 2.7.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Backend
    • Labels:
      None

      Description

      Having to restart to change the log level makes it difficult to timely troubleshoot issues in production, where restarts must be scheduled and planned at times that will minimize business impact.

      Would be nice to have the option to increase or decrease different log levels (catalog, ss, impalad) while running.

        Activity

        Hide
        scotts_impala_39e4 Scott Sappenfield added a comment -

        This is a great idea, thank you for filing it. This will be exceedingly useful in production environments and in particular multi-tenant envs as well.

        Show
        scotts_impala_39e4 Scott Sappenfield added a comment - This is a great idea, thank you for filing it. This will be exceedingly useful in production environments and in particular multi-tenant envs as well.
        Hide
        henryr Henry Robinson added a comment -

        Thanks for filing this. What would be your preferred interface to admin controls like this? We haven't really built any administrative tools into Impala (yet). A CLI interface that accessed an admin RPC service would be my preference, but I'm interested in what makes sense for your uses cases.

        Show
        henryr Henry Robinson added a comment - Thanks for filing this. What would be your preferred interface to admin controls like this? We haven't really built any administrative tools into Impala (yet). A CLI interface that accessed an admin RPC service would be my preference, but I'm interested in what makes sense for your uses cases.
        Hide
        PeterEbert Peter Ebert added a comment -

        The main thing for me is there will of course need to be a way to restrict access to this change.

        I did not give it deep thought, but presume we could leverage something similar to the way dynamic resource pool changes do not require a restart.

        I could see a CLI working, another alternative would be a simple addition to the webUI for each service, keeping in mind again there would need to be a way to restrict access to make changes.

        Show
        PeterEbert Peter Ebert added a comment - The main thing for me is there will of course need to be a way to restrict access to this change. I did not give it deep thought, but presume we could leverage something similar to the way dynamic resource pool changes do not require a restart. I could see a CLI working, another alternative would be a simple addition to the webUI for each service, keeping in mind again there would need to be a way to restrict access to make changes.
        Hide
        scotts_impala_39e4 Scott Sappenfield added a comment -

        I could also see a CLI working. I hadn't thought about the restricted access, but yeah, smart. That's a good one.

        Show
        scotts_impala_39e4 Scott Sappenfield added a comment - I could also see a CLI working. I hadn't thought about the restricted access, but yeah, smart. That's a good one.
        Hide
        bharathv bharath v added a comment -

        I've recently sent out a CR [1] to do something similar to this, for Frontend and Catalog log4j configurations. I did a small screencast [2] of what it looks like. Are you looking for something similar? With that patch, we can adjust the logging level of a single class/whole package/root level dynamically without a restart. Feedback is welcome.

        [1] https://gerrit.cloudera.org/#/c/5792/
        [2] https://vimeo.com/200946011

        Show
        bharathv bharath v added a comment - I've recently sent out a CR [1] to do something similar to this, for Frontend and Catalog log4j configurations. I did a small screencast [2] of what it looks like. Are you looking for something similar? With that patch, we can adjust the logging level of a single class/whole package/root level dynamically without a restart. Feedback is welcome. [1] https://gerrit.cloudera.org/#/c/5792/ [2] https://vimeo.com/200946011
        Hide
        bharathv bharath v added a comment -

        Fixed as a part of IMPALA-4822.

        IMPALA-4822: Implement dynamic log level changes

        Very often we have to change the logging levels
        of Impalads and Catalog server for debugging purposes.
        Currently, there is no way to do that without a restart
        of the daemons, which is not a viable option in production
        deployments.

        This patch addresses this supportability gap by exposing
        the ability to set dynamic logging levels using a simple
        web endpoint on Impalad/Catalog/Statestore web pages.

        This includes setting VLOG levels (equivalent to --v flag)
        as well as setting log4j levels on the Frontend and the
        Catalog JVMs.

        Change-Id: I588418e9bcb0b66d33138baf96207a5a35bfbd63
        Reviewed-on: http://gerrit.cloudera.org:8080/5792
        Reviewed-by: Bharath Vissapragada <bharathv@cloudera.com>
        Tested-by: Impala Public Jenkins

        Show
        bharathv bharath v added a comment - Fixed as a part of IMPALA-4822 . IMPALA-4822 : Implement dynamic log level changes Very often we have to change the logging levels of Impalads and Catalog server for debugging purposes. Currently, there is no way to do that without a restart of the daemons, which is not a viable option in production deployments. This patch addresses this supportability gap by exposing the ability to set dynamic logging levels using a simple web endpoint on Impalad/Catalog/Statestore web pages. This includes setting VLOG levels (equivalent to --v flag) as well as setting log4j levels on the Frontend and the Catalog JVMs. Change-Id: I588418e9bcb0b66d33138baf96207a5a35bfbd63 Reviewed-on: http://gerrit.cloudera.org:8080/5792 Reviewed-by: Bharath Vissapragada <bharathv@cloudera.com> Tested-by: Impala Public Jenkins

          People

          • Assignee:
            Unassigned
            Reporter:
            PeterEbert Peter Ebert
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development