Cassandra
  1. Cassandra
  2. CASSANDRA-4479

JMX attribute setters not consistent with cassandra.yaml

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Fix Version/s: 1.2.0 beta 1
    • Component/s: None
    • Labels:
      None

      Description

      If a setting is configurable both via cassandra.yaml and JMX, the two should be consistent. If that doesn't hold, then the JMX setter can't be trusted.

      Here I present the example of phi_convict_threshold.

      I'm trying to set phi_convict_threshold via JMX, which sets FailureDetector.phiConvictThreshold_, but this doesn't update Config.phi_convict_threshold, which gets its value from cassandra.yaml when starting up.

      Some places, such as FailureDetector.interpret(InetAddress), use FailureDetector.phiConvictThreshold_; others, such as AntiEntropyService.line 813 in cassandra-1.1.2, use Config.phi_convict_threshold:

                  // We want a higher confidence in the failure detection than usual because failing a repair wrongly has a high cost.
                  if (phi < 2 * DatabaseDescriptor.getPhiConvictThreshold())
                      return;
      

      where DatabaseDescriptor.getPhiConvictThreshold() returns Conf.phi_convict_threshold.

      So, it looks like there are cases where a value is stored in multiple places, and setting the value via JMX doesn't set all of them. I'd say there should only be a single place where a configuration parameter is stored, and that single field:

      • should read in the value from cassandra.yaml, optionally falling back to a sane default
      • should be the field that the JMX attribute reads and sets, and
      • any place that needs the current global setting should get it from that field. However, there could be cases where you read in a global value at the start of a task and keep that value locally until the end of the task.

      Also, anything settable via JMX should be volatile or set via a synchronized setter, or else according to the Java memory model other threads may be stuck with the old setting.

      So, I'm requesting the following:

      • Setting up guidelines for how to expose a configuration parameter both via cassandra.yaml and JMX, based on what I've mentioned above
      • Going through the list of configuration parameters and fixing any that don't match those guidelines

      I'd also recommend logging any changes to configuration parameters.

      1. trunk-4479.txt
        13 kB
        Chris Merrill

        Activity

        Gavin made changes -
        Workflow patch-available, re-open possible [ 12753183 ] reopen-resolved, no closed status, patch-avail, testing [ 12755910 ]
        Gavin made changes -
        Workflow no-reopen-closed, patch-avail [ 12718919 ] patch-available, re-open possible [ 12753183 ]
        Brandon Williams made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Assignee Chris Merrill [ cmerrill ]
        Resolution Fixed [ 1 ]
        Jonathan Ellis made changes -
        Fix Version/s 1.2.0 [ 12319262 ]
        Affects Version/s 1.1.2 [ 12321445 ]
        Priority Major [ 3 ] Trivial [ 5 ]
        Reviewer brandon.williams
        Chris Merrill made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Chris Merrill made changes -
        Attachment trunk-4479.txt [ 12541616 ]
        Eric Dong made changes -
        Field Original Value New Value
        Summary Multiple phi_convict_threshold fields not all settable via JMX JMX attribute setters not consistent with cassandra.yaml
        Description If a setting is configurable both via cassandra.yaml and JMX, the two should be consistent, but that is not the case for phi_convict_threshold.

        I'm trying to set phi_convict_threshold via JMX, which sets FailureDetector.phiConvictThreshold_, but this doesn't update Config.phi_convict_threshold, which gets its value from cassandra.yaml when starting up.

        Some places, such as FailureDetector.interpret(InetAddress), use FailureDetector.phiConvictThreshold_; others, such as AntiEntropyService.line 813 in cassandra-1.1.2, use Config.phi_convict_threshold:
        {code}
                    // We want a higher confidence in the failure detection than usual because failing a repair wrongly has a high cost.
                    if (phi < 2 * DatabaseDescriptor.getPhiConvictThreshold())
                        return;
        {code}

        where DatabaseDescriptor.getPhiConvictThreshold() returns Conf.phi_convict_threshold.

        So, it looks like there are cases where a value is stored in multiple places, and setting the value via JMX doesn't set all of them. I'd say there should only be a single place where a configuration parameter is stored, and that single field:
        * should read in the value from cassandra.yaml, optionally falling back to a sane default
        * should be the field that the JMX attribute reads and sets, and
        * any place that needs the current global setting should get it from that field. However, there could be cases where you read in a global value at the start of a task and keep that value locally until the end of the task.

        Also, anything settable via JMX should be volatile or set via a synchronized setter, or else according to the Java memory model other threads may be stuck with the old setting.
        If a setting is configurable both via cassandra.yaml and JMX, the two should be consistent. If that doesn't hold, then the JMX setter can't be trusted.

        Here I present the example of phi_convict_threshold.

        I'm trying to set phi_convict_threshold via JMX, which sets FailureDetector.phiConvictThreshold_, but this doesn't update Config.phi_convict_threshold, which gets its value from cassandra.yaml when starting up.

        Some places, such as FailureDetector.interpret(InetAddress), use FailureDetector.phiConvictThreshold_; others, such as AntiEntropyService.line 813 in cassandra-1.1.2, use Config.phi_convict_threshold:
        {code}
                    // We want a higher confidence in the failure detection than usual because failing a repair wrongly has a high cost.
                    if (phi < 2 * DatabaseDescriptor.getPhiConvictThreshold())
                        return;
        {code}

        where DatabaseDescriptor.getPhiConvictThreshold() returns Conf.phi_convict_threshold.


        So, it looks like there are cases where a value is stored in multiple places, and setting the value via JMX doesn't set all of them. I'd say there should only be a single place where a configuration parameter is stored, and that single field:
        * should read in the value from cassandra.yaml, optionally falling back to a sane default
        * should be the field that the JMX attribute reads and sets, and
        * any place that needs the current global setting should get it from that field. However, there could be cases where you read in a global value at the start of a task and keep that value locally until the end of the task.

        Also, anything settable via JMX should be volatile or set via a synchronized setter, or else according to the Java memory model other threads may be stuck with the old setting.


        So, I'm requesting the following:
        * Setting up guidelines for how to expose a configuration parameter both via cassandra.yaml and JMX, based on what I've mentioned above
        * Going through the list of configuration parameters and fixing any that don't match those guidelines


        I'd also recommend logging any changes to configuration parameters.
        Eric Dong created issue -

          People

          • Assignee:
            Chris Merrill
            Reporter:
            Eric Dong
            Reviewer:
            Brandon Williams
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development