Cassandra
  1. Cassandra
  2. CASSANDRA-4430

optional pluggable o.a.c.metrics reporters

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Fix Version/s: 2.0.2
    • Component/s: Core
    • Labels:
      None

      Description

      CASSANDRA-4009 expanded the use of the metrics library which has a set of reporter modules http://metrics.codahale.com/manual/core/#reporters You can report to flat files, ganglia, spit everything over http, etc. The next step is a mechanism for using those reporters with o.a.c.metrics. To avoid bundling everything I suggest following the mx4j approach of "enable only if on classpath coupled with a reporter configuration file.

      Strawman file:

      console:
        time: 1
        timeunit: "seconds"
      csv:
       - time: 1
         timeunit: minutes
         file: foo.csv
       - time: 10
         timeunit: seconds
          file: bar.csv
      ganglia:
       - time: 30
         timunit: seconds
         host: server-1
         port: 8649
       - time: 30
         timunit: seconds
         host: server-2
         port: 8649
      
      1. cassandra-ganglia-example.png
        88 kB
        Chris Burroughs
      2. 4430-trunk.txt
        8 kB
        Tyler Hobbs
      3. 4430-2.0.txt
        6 kB
        Tyler Hobbs

        Issue Links

          Activity

          Hide
          Chris Burroughs added a comment -
          Show
          Chris Burroughs added a comment - https://github.com/addthis/metrics-reporter-config <-- library, needs integration
          Hide
          Jonathan Ellis added a comment -

          Not quite sure what to make of that last comment tbh.

          Show
          Jonathan Ellis added a comment - Not quite sure what to make of that last comment tbh.
          Hide
          Chris Burroughs added a comment -

          POC Screenshot working

          Show
          Chris Burroughs added a comment - POC Screenshot working
          Hide
          Chris Burroughs added a comment -

          https://github.com/cburroughs/cassandra/tree/metrics-config with the meat at https://github.com/cburroughs/cassandra/commit/c966fa8da4667768dea1aab1ef26bac811412bdd

          A few notes at this stage:

          • It works!
          • Requires newer snakeyaml ( CASSANDRA-5317 ), I'm not sure if that's something that can be worked around
          • Not sure if the jboss logging transitive dep can be avoided yet. All of the transitives dependencies could be avoided (I think) if we don't care about validating configs (ie getting helpful messages along the lines of "fortnight" is not a valid timeunit, must match "^(DAYS|HOURS|MICROSECONDS|MILLISECONDS|MINUTES|NANOSECONDS|SECONDS)$")
          • In theory the config could be embedded in cassandra.yaml. I went with a new file since jmx/log4j/mx4j etc are all done outside of cassandra.yaml. This is the design choice I was least sure about.

          To use:

          Show
          Chris Burroughs added a comment - https://github.com/cburroughs/cassandra/tree/metrics-config with the meat at https://github.com/cburroughs/cassandra/commit/c966fa8da4667768dea1aab1ef26bac811412bdd A few notes at this stage: It works! Requires newer snakeyaml ( CASSANDRA-5317 ), I'm not sure if that's something that can be worked around Not sure if the jboss logging transitive dep can be avoided yet. All of the transitives dependencies could be avoided (I think) if we don't care about validating configs (ie getting helpful messages along the lines of "fortnight" is not a valid timeunit, must match "^(DAYS|HOURS|MICROSECONDS|MILLISECONDS|MINUTES|NANOSECONDS|SECONDS)$") In theory the config could be embedded in cassandra.yaml. I went with a new file since jmx/log4j/mx4j etc are all done outside of cassandra.yaml. This is the design choice I was least sure about. To use: Configure conf/metrics-reporter-config.yaml as described in https://github.com/addthis/metrics-reporter-config copy the jar for whatever metrics sink you are using into lib/ (csv reporter is built in, the more interesting ganglia/graphite sinks are independent jars http://mvnrepository.com/artifact/com.yammer.metrics/metrics-ganglia )
          Hide
          Chris Burroughs added a comment -

          Also a cluster with the stress keyspaces seems to have on the order of 1.5k metric from o.a.c.metrics (yeah metrics!), but that can be a rather large amount for ganglia/graphite to deal with. I may need to think about how to approach that.

          Show
          Chris Burroughs added a comment - Also a cluster with the stress keyspaces seems to have on the order of 1.5k metric from o.a.c.metrics (yeah metrics!), but that can be a rather large amount for ganglia/graphite to deal with. I may need to think about how to approach that.
          Hide
          Chris Burroughs added a comment -

          Rebased to 1.2.3 without trouble (although I think snakeyaml is going to force this to be a 2.0 feature), and we have been running with this in a per-production environment without incident. Keeping per connection stats around is not tenable for larger clusters, so I think we will need a default config with an example blacklist (what we do with internal services is a convention like ending metric names with JMX_ONLY, but that's probably too ugly and brittle for general use).

          What I need at this point is:

          • feedback on the configuration approach bin/cassandra referenced system property
          • Are the transitive deps okay?
          Show
          Chris Burroughs added a comment - Rebased to 1.2.3 without trouble (although I think snakeyaml is going to force this to be a 2.0 feature), and we have been running with this in a per-production environment without incident. Keeping per connection stats around is not tenable for larger clusters, so I think we will need a default config with an example blacklist (what we do with internal services is a convention like ending metric names with JMX_ONLY, but that's probably too ugly and brittle for general use). What I need at this point is: feedback on the configuration approach bin/cassandra referenced system property Are the transitive deps okay?
          Hide
          Jonathan Ellis added a comment -

          What do you think, Tyler?

          Show
          Jonathan Ellis added a comment - What do you think, Tyler?
          Hide
          Tyler Hobbs added a comment -

          Overall I'm not opposed to the idea of making it easy to enable various reporters, but I do have some questions and comments.

          Keeping per connection stats around is not tenable for larger clusters

          Can you clarify which stats you're referring to? I didn't think we were exposing any metrics on a per-connection basis.

          I'm not sure which component is failing, but if I enable the csv reporter, Cassandra logs this error repeatedly:

          java.io.IOException: Unable to create /tmp/metrics/crazy-debugging/LiveSSTableCount.csv
          	at com.yammer.metrics.reporting.CsvReporter.createStreamForMetric(CsvReporter.java:141)
          	at com.yammer.metrics.reporting.CsvReporter.getPrintStream(CsvReporter.java:257)
          	at com.yammer.metrics.reporting.CsvReporter.access$000(CsvReporter.java:22)
          	at com.yammer.metrics.reporting.CsvReporter$1.getStream(CsvReporter.java:156)
          	at com.yammer.metrics.reporting.CsvReporter.processGauge(CsvReporter.java:229)
          	at com.yammer.metrics.reporting.CsvReporter.processGauge(CsvReporter.java:22)
          	at com.yammer.metrics.core.Gauge.processWith(Gauge.java:28)
          	at com.yammer.metrics.reporting.CsvReporter.run(CsvReporter.java:163)
          	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
          	at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
          	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
          	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
          	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
          	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          	at java.lang.Thread.run(Thread.java:662)
          

          I also seem to only get a subset of the metrics (BloomFilterFalsePositives, CurrentlyBlockedTasks, MeanRowSize, Size, CompressionRatio, LiveSSTableCount, MinRowSize, and TotalDiskSpaceUsed) which appear to only cover one column family. Is the csv reporter just no good, or am I missing something here?

          Regarding whitelisting/blacklisting, is the regex applied to the full metric name (e.g. o.a.c.metrics.ColumnFamily.WriteLatency.keyspace1.cf1)?

          Show
          Tyler Hobbs added a comment - Overall I'm not opposed to the idea of making it easy to enable various reporters, but I do have some questions and comments. Keeping per connection stats around is not tenable for larger clusters Can you clarify which stats you're referring to? I didn't think we were exposing any metrics on a per-connection basis. I'm not sure which component is failing, but if I enable the csv reporter, Cassandra logs this error repeatedly: java.io.IOException: Unable to create /tmp/metrics/crazy-debugging/LiveSSTableCount.csv at com.yammer.metrics.reporting.CsvReporter.createStreamForMetric(CsvReporter.java:141) at com.yammer.metrics.reporting.CsvReporter.getPrintStream(CsvReporter.java:257) at com.yammer.metrics.reporting.CsvReporter.access$000(CsvReporter.java:22) at com.yammer.metrics.reporting.CsvReporter$1.getStream(CsvReporter.java:156) at com.yammer.metrics.reporting.CsvReporter.processGauge(CsvReporter.java:229) at com.yammer.metrics.reporting.CsvReporter.processGauge(CsvReporter.java:22) at com.yammer.metrics.core.Gauge.processWith(Gauge.java:28) at com.yammer.metrics.reporting.CsvReporter.run(CsvReporter.java:163) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) I also seem to only get a subset of the metrics (BloomFilterFalsePositives, CurrentlyBlockedTasks, MeanRowSize, Size, CompressionRatio, LiveSSTableCount, MinRowSize, and TotalDiskSpaceUsed) which appear to only cover one column family. Is the csv reporter just no good, or am I missing something here? Regarding whitelisting/blacklisting, is the regex applied to the full metric name (e.g. o.a.c.metrics.ColumnFamily.WriteLatency.keyspace1.cf1)?
          Hide
          Chris Burroughs added a comment -

          Can you clarify which stats you're referring to? I didn't think we were exposing any metrics on a per-connection basis.

          See the per connection ones in http://wiki.apache.org/cassandra/Metrics ; fox example per IP ones in StreamingMetrics https://github.com/apache/cassandra/blob/cassandra-1.2.3/src/java/org/apache/cassandra/metrics/StreamingMetrics.java. I just mean that as an example of something that generates too much noise to be a good out of the box experience.

          Is the csv reporter just no good, or am I missing something here?

          The csv reporter is probably not used in production much. I'm not exactly sure what is going on when trying to use it here, but I agree it's producing a paucity of metrics.

          Regarding whitelisting/blacklisting, is the regex applied to the full metric name (e.g. o.a.c.metrics.ColumnFamily.WriteLatency.keyspace1.cf1)?

          Currently only the "name" part of MetricName (so "LiveSSTableCount" not "org.apache.cassandra.metrics.ColumnFamily.LiveSSTableCount.system.IndexInfo") is used. However, that's a metric-reporter-config limitation, not a Metrics one, and could be changed. a.o.c.metrics has a rather sophisticated naming scheme and I'm not sure how best to keep it simple.

          Show
          Chris Burroughs added a comment - Can you clarify which stats you're referring to? I didn't think we were exposing any metrics on a per-connection basis. See the per connection ones in http://wiki.apache.org/cassandra/Metrics ; fox example per IP ones in StreamingMetrics https://github.com/apache/cassandra/blob/cassandra-1.2.3/src/java/org/apache/cassandra/metrics/StreamingMetrics.java . I just mean that as an example of something that generates too much noise to be a good out of the box experience. Is the csv reporter just no good, or am I missing something here? The csv reporter is probably not used in production much. I'm not exactly sure what is going on when trying to use it here, but I agree it's producing a paucity of metrics. Regarding whitelisting/blacklisting, is the regex applied to the full metric name (e.g. o.a.c.metrics.ColumnFamily.WriteLatency.keyspace1.cf1)? Currently only the "name" part of MetricName (so "LiveSSTableCount" not "org.apache.cassandra.metrics.ColumnFamily.LiveSSTableCount.system.IndexInfo") is used. However, that's a metric-reporter-config limitation, not a Metrics one, and could be changed. a.o.c.metrics has a rather sophisticated naming scheme and I'm not sure how best to keep it simple.
          Hide
          Tyler Hobbs added a comment -

          Currently only the "name" part of MetricName (so "LiveSSTableCount" not "org.apache.cassandra.metrics.ColumnFamily.LiveSSTableCount.system.IndexInfo") is used. However, that's a metric-reporter-config limitation, not a Metrics one, and could be changed. a.o.c.metrics has a rather sophisticated naming scheme and I'm not sure how best to keep it simple.

          I feel like this is probably the main problem that needs to be solved. Besides per-connection metrics (thanks for pointing those out), there are many users with hundreds or thousands of column families, each of which have quite a few metrics. Ideally, it would be easy to:
          1. Capture all metrics for only a specific column family
          2. Capture only one specific metric (e.g. read latency) across all column families

          It seems like the regex support should be sufficient for that if it's applied to the full metric name (o.a.c.metrics.ColumnFamily.LiveSSTableCount.system.IndexInfo). How hard would it be to make changes to metric-reporter-config to support using the full metric name?

          Show
          Tyler Hobbs added a comment - Currently only the "name" part of MetricName (so "LiveSSTableCount" not "org.apache.cassandra.metrics.ColumnFamily.LiveSSTableCount.system.IndexInfo") is used. However, that's a metric-reporter-config limitation, not a Metrics one, and could be changed. a.o.c.metrics has a rather sophisticated naming scheme and I'm not sure how best to keep it simple. I feel like this is probably the main problem that needs to be solved. Besides per-connection metrics (thanks for pointing those out), there are many users with hundreds or thousands of column families, each of which have quite a few metrics. Ideally, it would be easy to: 1. Capture all metrics for only a specific column family 2. Capture only one specific metric (e.g. read latency) across all column families It seems like the regex support should be sufficient for that if it's applied to the full metric name (o.a.c.metrics.ColumnFamily.LiveSSTableCount.system.IndexInfo). How hard would it be to make changes to metric-reporter-config to support using the full metric name?
          Hide
          Chris Burroughs added a comment -

          Using the full metric name is reasonable and should not be too hard. I'll let you know when I have a release with that.

          (One minor pedantic note is that that "full name" is actually an artifact of the Reporter, and not of the metric itself. In theory someone could write a reporter that did "scope + metricName + reverse(pkg)", or something like that and end up confused. I'm going to ignore that and concat the strings in normal (jmx) order unless there is a super popular 3rd party reporter that does something funky.)

          Show
          Chris Burroughs added a comment - Using the full metric name is reasonable and should not be too hard. I'll let you know when I have a release with that. (One minor pedantic note is that that "full name" is actually an artifact of the Reporter, and not of the metric itself. In theory someone could write a reporter that did "scope + metricName + reverse(pkg)", or something like that and end up confused. I'm going to ignore that and concat the strings in normal (jmx) order unless there is a super popular 3rd party reporter that does something funky.)
          Hide
          Tyler Hobbs added a comment -

          That sounds great, thanks.

          Show
          Tyler Hobbs added a comment - That sounds great, thanks.
          Hide
          Jonathan Ellis added a comment -

          Still working on this, Chris?

          Show
          Jonathan Ellis added a comment - Still working on this, Chris?
          Hide
          Chris Burroughs added a comment -

          Yep, hope to have something to show for it soon.

          Show
          Chris Burroughs added a comment - Yep, hope to have something to show for it soon.
          Hide
          Chris Burroughs added a comment -

          Traveling for the next two weeks or so, but since this would have saved my Friday afternoon it should be near the top of my plate when I get back.

          Show
          Chris Burroughs added a comment - Traveling for the next two weeks or so, but since this would have saved my Friday afternoon it should be near the top of my plate when I get back.
          Hide
          Chris Burroughs added a comment -

          New draft up !
          https://github.com/cburroughs/cassandra/tree/metrics-config-2, uses metrics-report-config SNAPSHOT with "qualified" names (org.apache.cassandra.metrics.ClientRequest...)

          • Rebased against latest 1.2 branch. I'm running 1.2 and hope to have this on my production clusters soon.
          • Added a sample file that sends all global (not good terminology, as in metrics that are not per connection/columFamily) to a file
          • Didn't like hard coding in bin/cassandra so -Dcassandra.metricsReporterConfigFile must be specified for this to do anything

          CSVReporter only works if the non-qualifed names are unique. The same uses the console reporter to dump everything to one big file which is really only useful as a sample

          Show
          Chris Burroughs added a comment - New draft up ! https://github.com/cburroughs/cassandra/tree/metrics-config-2 , uses metrics-report-config SNAPSHOT with "qualified" names (org.apache.cassandra.metrics.ClientRequest...) Rebased against latest 1.2 branch. I'm running 1.2 and hope to have this on my production clusters soon. Added a sample file that sends all global (not good terminology, as in metrics that are not per connection/columFamily) to a file Didn't like hard coding in bin/cassandra so -Dcassandra.metricsReporterConfigFile must be specified for this to do anything CSVReporter only works if the non-qualifed names are unique. The same uses the console reporter to dump everything to one big file which is really only useful as a sample
          Hide
          Chris Burroughs added a comment -

          Nick Bailey, any comments?

          Show
          Chris Burroughs added a comment - Nick Bailey , any comments?
          Hide
          Nick Bailey added a comment -

          sorry, this fell off my radar. Tyler Hobbs is back now though. Can you take over review again Tyler?

          Show
          Nick Bailey added a comment - sorry, this fell off my radar. Tyler Hobbs is back now though. Can you take over review again Tyler?
          Hide
          Tyler Hobbs added a comment -

          Sure.

          Show
          Tyler Hobbs added a comment - Sure.
          Hide
          Tyler Hobbs added a comment -

          I didn't test out everything this could do, but it seems like a solid starting point.

          The attached 4430-trunk.txt rebases Chris's changes against trunk (it looks like we want this to go in 2.1 now). The changes are also in my branch, which includes the new jar files.

          Show
          Tyler Hobbs added a comment - I didn't test out everything this could do, but it seems like a solid starting point. The attached 4430-trunk.txt rebases Chris's changes against trunk (it looks like we want this to go in 2.1 now). The changes are also in my branch , which includes the new jar files.
          Hide
          Tyler Hobbs added a comment -

          4430-2.0.txt (and branch) makes the same changes for the 2.0 branch.

          Show
          Tyler Hobbs added a comment - 4430-2.0.txt (and branch ) makes the same changes for the 2.0 branch.
          Hide
          Jonathan Ellis added a comment -

          Hang on, we need to ship 4 extra jars now? Or are those only necessary if you want to enable different reporters?

          Show
          Jonathan Ellis added a comment - Hang on, we need to ship 4 extra jars now? Or are those only necessary if you want to enable different reporters?
          Hide
          Chris Burroughs added a comment - - edited

          Hang on, we need to ship 4 extra jars now? Or are those only necessary if you want to enable different reporters?

          The jars are for validation of the config, javax.validation.validation-api the actual implementation of said api from the hibernate folks, and the logging jar that the hibernate folks use. That's admittedly a case study in somewhat excessive java modularity. I think if those deps are excluded the only consequence is less useful error messages. On the other hand cassandra.yaml itself might benefit from the javax.validation stuff, but that's a different topic.

          Ex:

              @Pattern(
                  regexp = "^(DAYS|HOURS|MICROSECONDS|MILLISECONDS|MINUTES|NANOSECONDS|SECONDS)$",
                  message = "must be a valid java.util.concurrent.TimeUnit"
              )
              private String timeunit;
          

          I didn't currently include metrics-ganglia/metrics-graphite/any-third party metrics packages (so users would have to toss a jar in like we do for mx4j).

          Show
          Chris Burroughs added a comment - - edited Hang on, we need to ship 4 extra jars now? Or are those only necessary if you want to enable different reporters? The jars are for validation of the config, javax.validation.validation-api the actual implementation of said api from the hibernate folks, and the logging jar that the hibernate folks use. That's admittedly a case study in somewhat excessive java modularity. I think if those deps are excluded the only consequence is less useful error messages. On the other hand cassandra.yaml itself might benefit from the javax.validation stuff, but that's a different topic. Ex: @Pattern( regexp = "^(DAYS|HOURS|MICROSECONDS|MILLISECONDS|MINUTES|NANOSECONDS|SECONDS)$" , message = "must be a valid java.util.concurrent.TimeUnit" ) private String timeunit; I didn't currently include metrics-ganglia/metrics-graphite/any-third party metrics packages (so users would have to toss a jar in like we do for mx4j).
          Hide
          Mikhail Stepura added a comment -

          BTW, there is a line with typo: - "^org.apache.cassandra.metrics.ThradPools.+" in both patches

          Show
          Mikhail Stepura added a comment - BTW, there is a line with typo: - "^org.apache.cassandra.metrics. ThradPools .+" in both patches
          Hide
          Jonathan Ellis added a comment -

          I think if those deps are excluded the only consequence is less useful error messages.

          Let's verify that, because I really don't want to increase our list of hand-maintained jars and licenses (which appear to be missing, btw) by > 10% for a very optional feature. Especially when half of the four new ones are not final releases...

          Show
          Jonathan Ellis added a comment - I think if those deps are excluded the only consequence is less useful error messages. Let's verify that, because I really don't want to increase our list of hand-maintained jars and licenses (which appear to be missing, btw) by > 10% for a very optional feature. Especially when half of the four new ones are not final releases...
          Hide
          Chris Burroughs added a comment -

          v3:

          • Fixed sp
          • Use just released version instead of identical snapshot of reporter-config
          • license file
          • Disabled config validation and removed related jars. Errors messages are slightly less friendly but behavior is otherwise unchanged.

          Pushed a 1.2 and rebased on 2.0 version:

          Show
          Chris Burroughs added a comment - v3: Fixed sp Use just released version instead of identical snapshot of reporter-config license file Disabled config validation and removed related jars. Errors messages are slightly less friendly but behavior is otherwise unchanged. Pushed a 1.2 and rebased on 2.0 version: https://github.com/cburroughs/cassandra/tree/metrics-config-3 https://github.com/cburroughs/cassandra/tree/2.0-metrics-config-3
          Hide
          Chris Burroughs added a comment -

          v4 really really removes all references to the other jars from both places in build.xml

          Show
          Chris Burroughs added a comment - v4 really really removes all references to the other jars from both places in build.xml https://github.com/cburroughs/cassandra/tree/metrics-config-4 https://github.com/cburroughs/cassandra/tree/2.0-metrics-config-4
          Hide
          Tyler Hobbs added a comment -

          Looks good except for a couple of nitpicks:

          • Check that metricsReporterConfigFile isn't null before logging and attempting to load the file in CassandraDaemon
          • There are some blank lines at the end of the example config file
          Show
          Tyler Hobbs added a comment - Looks good except for a couple of nitpicks: Check that metricsReporterConfigFile isn't null before logging and attempting to load the file in CassandraDaemon There are some blank lines at the end of the example config file
          Hide
          Chris Burroughs added a comment -

          v4 branches went on a newline diet and have a null check.

          Show
          Chris Burroughs added a comment - v4 branches went on a newline diet and have a null check.
          Hide
          Tyler Hobbs added a comment -

          +1 (just did a quick test on the 2.0 branch, since I don't think we'll add features this late in the 1.2 branch)

          Show
          Tyler Hobbs added a comment - +1 (just did a quick test on the 2.0 branch, since I don't think we'll add features this late in the 1.2 branch)
          Hide
          Chris Burroughs added a comment -

          Ticket says 2.1 but I think Jonathan Ellis was cool with it in 2.0.x as long as it was ready "soon". 1.2 is because that's what I am running in production but I agree it doesn't make sense to risk merging there.

          Show
          Chris Burroughs added a comment - Ticket says 2.1 but I think Jonathan Ellis was cool with it in 2.0.x as long as it was ready "soon". 1.2 is because that's what I am running in production but I agree it doesn't make sense to risk merging there.
          Hide
          Jonathan Ellis added a comment -

          Committed!

          Show
          Jonathan Ellis added a comment - Committed!

            People

            • Assignee:
              Chris Burroughs
              Reporter:
              Chris Burroughs
              Reviewer:
              Tyler Hobbs
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development