Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2
    • Component/s: None
    • Labels:
      None

      Description

      This proposal is simply to allow the output of _active_tasks to be less rigid. Basically to allow each task to be able to output different JSON fields.

      Somethings like the status text simply go away. Instead application can built it based on more granular fields provided by _active_tasks.
      Some examples:

      1) "progress" (an integer percentage, for all tasks)

      2) "database" (for compactions and indexer tasks)

      3) "design_document" (for indexer and view compaction tasks)

      4) "source" and "target" (for replications)

      5) "docs_read", "docs_written", "doc_write_failures",
      "missing_revs_found", "missing_revs_checked", "source_seq",
      "checkpointed_source_seq" and "continuous" for replications

      1. alternate-api.patch
        4 kB
        Paul Joseph Davis

        Activity

        Hide
        Filipe Manana added a comment -

        Applied to trunk and branch 1.2.x.

        Here's an _active_tasks example output:

        $ curl http://localhost:5984/_active_tasks
        [

        { "pid":"<0.242.0>", "changes_done":31209, "database":"indexer_test_3", "design_document":"_design/test", "progress":5, "started_on":1316228432, "total_changes":551201, "type":"indexer", "updated_on":1316228461 }

        ],

        { "pid":"<0.1156.0>", "database":"indexer_test_3", "design_document":"_design/test", "progress":21, "started_on":1316229336, "type":"view_compaction", "updated_on":1316229352 }

        ,

        { "pid":"<0.1303.0>", "checkpointed_source_seq":17333, "continuous":false, "doc_write_failures":0, "docs_read":17833, "docs_written":17833, "missing_revisions_found":17833, "progress":3, "revisions_checked":17833, "source":"http://fdmanana.couchone.com/indexer_test/", "source_seq":551202, "started_on":1316229471, "target":"indexer_test", "type":"replication", "updated_on":1316230082 }

        ]

        Show
        Filipe Manana added a comment - Applied to trunk and branch 1.2.x. Here's an _active_tasks example output: $ curl http://localhost:5984/_active_tasks [ { "pid":"<0.242.0>", "changes_done":31209, "database":"indexer_test_3", "design_document":"_design/test", "progress":5, "started_on":1316228432, "total_changes":551201, "type":"indexer", "updated_on":1316228461 } ], { "pid":"<0.1156.0>", "database":"indexer_test_3", "design_document":"_design/test", "progress":21, "started_on":1316229336, "type":"view_compaction", "updated_on":1316229352 } , { "pid":"<0.1303.0>", "checkpointed_source_seq":17333, "continuous":false, "doc_write_failures":0, "docs_read":17833, "docs_written":17833, "missing_revisions_found":17833, "progress":3, "revisions_checked":17833, "source":"http://fdmanana.couchone.com/indexer_test/", "source_seq":551202, "started_on":1316229471, "target":"indexer_test", "type":"replication", "updated_on":1316230082 } ]
        Hide
        Paul Joseph Davis added a comment -

        Awesome work, Filipe. +1 on committing this as it is.

        Show
        Paul Joseph Davis added a comment - Awesome work, Filipe. +1 on committing this as it is.
        Hide
        Filipe Manana added a comment -

        I just rebased the patch on current trunk, after Paul's view refactoring:

        https://github.com/fdmanana/couchdb/commit/9c5481157fc47090a20fe098eb18a7cdcc823231

        I'll rename this ticket's title, as the purpose is no longer to add a "stats" field but rather to make _active_tasks more generic, less rigid, and allow different tasks to specify different properties.

        Show
        Filipe Manana added a comment - I just rebased the patch on current trunk, after Paul's view refactoring: https://github.com/fdmanana/couchdb/commit/9c5481157fc47090a20fe098eb18a7cdcc823231 I'll rename this ticket's title, as the purpose is no longer to add a "stats" field but rather to make _active_tasks more generic, less rigid, and allow different tasks to specify different properties.
        Hide
        Filipe Manana added a comment -

        Patch updated,

        https://github.com/fdmanana/couchdb/compare/task_stats.diff

        The status text string in futon is now constructed by JavaScript via the better structured _active_tasks output.

        Show
        Filipe Manana added a comment - Patch updated, https://github.com/fdmanana/couchdb/compare/task_stats.diff The status text string in futon is now constructed by JavaScript via the better structured _active_tasks output.
        Hide
        Filipe Manana added a comment -

        Exactly Paul.
        I'll be changing the "type" field to be easier to handle with, for example for view compaction it is the string "View Group Compaction" - something like "view_compaction" would be easier for applications. Then Futon, using a simple switch/if-then-else logic can display "View Group Compaction" or something else.

        Show
        Filipe Manana added a comment - Exactly Paul. I'll be changing the "type" field to be easier to handle with, for example for view compaction it is the string "View Group Compaction" - something like "view_compaction" would be easier for applications. Then Futon, using a simple switch/if-then-else logic can display "View Group Compaction" or something else.
        Hide
        Paul Joseph Davis added a comment -

        Ah, I see what you mean now. Yeah, I'd probably just improve the task tab so that it has a table for each type of task and then each table can have the columns and formatting it needs.

        Show
        Paul Joseph Davis added a comment - Ah, I see what you mean now. Yeah, I'd probably just improve the task tab so that it has a table for each type of task and then each table can have the columns and formatting it needs.
        Hide
        Filipe Manana added a comment -

        I see what you mean now Paul.

        I still think the progress field should be given by couch and is one of the fields people care most about.

        As for Futon, specially for replication tasks, I see all (or most) of those stats meaningful for the normal user. Other than displaying the json object, I don't see a way to display these fields which are not common for all tasks, I guess someone specialized on UIs would find a much better way.

        Show
        Filipe Manana added a comment - I see what you mean now Paul. I still think the progress field should be given by couch and is one of the fields people care most about. As for Futon, specially for replication tasks, I see all (or most) of those stats meaningful for the normal user. Other than displaying the json object, I don't see a way to display these fields which are not common for all tasks, I guess someone specialized on UIs would find a much better way.
        Hide
        Paul Joseph Davis added a comment -

        After thinking about it for a few more minutes there's no reason to even carry the prop list around, we can just give couch_task_status an API to mutate its internal task representation. This saves even more code and makes the whole thing quite a bit more friendly.

        Show
        Paul Joseph Davis added a comment - After thinking about it for a few more minutes there's no reason to even carry the prop list around, we can just give couch_task_status an API to mutate its internal task representation. This saves even more code and makes the whole thing quite a bit more friendly.
        Hide
        Paul Joseph Davis added a comment -

        Not sure why we'd want to delay changing the status code until 2.0. 1.2 seems like a big enough version that we could improve this part of the code.

        Why would you want to see this stuff in Futon? If Futon is doing its job all of the relevant information should be represented in the UI. I can't imagine that normal people are going to be interested in seeing a JSON blob duplicating all of the status information that's already formatted for display.

        I guess its possible that the definition of progress changes in the future. Not sure I see how it would change enough to be considerably different yet the same enough that it's interpreted the same.

        As to the record comment I'm not sure where that's coming from. The suggestion on using a proplist is so that we can carry the task information around and update it more gracefully. As an example I did a quick sketch of what this might look like for the view compactor. Attached is a diff from trunk showing what I was thinking.

        Show
        Paul Joseph Davis added a comment - Not sure why we'd want to delay changing the status code until 2.0. 1.2 seems like a big enough version that we could improve this part of the code. Why would you want to see this stuff in Futon? If Futon is doing its job all of the relevant information should be represented in the UI. I can't imagine that normal people are going to be interested in seeing a JSON blob duplicating all of the status information that's already formatted for display. I guess its possible that the definition of progress changes in the future. Not sure I see how it would change enough to be considerably different yet the same enough that it's interpreted the same. As to the record comment I'm not sure where that's coming from. The suggestion on using a proplist is so that we can carry the task information around and update it more gracefully. As an example I did a quick sketch of what this might look like for the view compactor. Attached is a diff from trunk showing what I was thinking.
        Hide
        Filipe Manana added a comment -

        Thanks Paul and Adam,

        I agree the current status text field could go away, as it can be built from this stats property. However I would be more confortable doing this change for 2.0 for e.g.

        I like to see this in Futon. I agree it currently doesn't look like very formatted and so on, so it could get polished by someone more confortable with Futon/CSS/HTML.

        About the progress, right now it's trivial to compute (and it's more of an estimation, without a very high degree of accuracy). But i prefer to leave it's computation to the tasks and not the applications. This is because how it's calculated can change in the future as the implementation of the tasks (compaction, indexing, replication) evolve.

        I agree that with setelement and #record.fieldname tricks some code size can be reduced.

        Show
        Filipe Manana added a comment - Thanks Paul and Adam, I agree the current status text field could go away, as it can be built from this stats property. However I would be more confortable doing this change for 2.0 for e.g. I like to see this in Futon. I agree it currently doesn't look like very formatted and so on, so it could get polished by someone more confortable with Futon/CSS/HTML. About the progress, right now it's trivial to compute (and it's more of an estimation, without a very high degree of accuracy). But i prefer to leave it's computation to the tasks and not the applications. This is because how it's calculated can change in the future as the implementation of the tasks (compaction, indexing, replication) evolve. I agree that with setelement and #record.fieldname tricks some code size can be reduced.
        Hide
        Paul Joseph Davis added a comment -

        Ooh. If we added a couple proplist helpers like couch_util:set_value/3 and couch_util:incr_value/3 this could trim out a lot of redundant code.

        Show
        Paul Joseph Davis added a comment - Ooh. If we added a couple proplist helpers like couch_util:set_value/3 and couch_util:incr_value/3 this could trim out a lot of redundant code.
        Hide
        Adam Kocoloski added a comment -

        Totally agree that it's silly for us to be formatting status messages in the internals. I like what you've done here, Filipe, but I would have also supported a patch where a task was specified as a simple property list.

        Show
        Adam Kocoloski added a comment - Totally agree that it's silly for us to be formatting status messages in the internals. I like what you've done here, Filipe, but I would have also supported a patch where a task was specified as a simple property list.
        Hide
        Paul Joseph Davis added a comment -

        In fact, the more I think about it, I don't see why we wouldn't just create a single JSON blob for each task, then all clients can do whatever it is they want to do with the data. I'd also get rid of things like progress and instead focus on allowing clients to do their own calculations and display.

        Show
        Paul Joseph Davis added a comment - In fact, the more I think about it, I don't see why we wouldn't just create a single JSON blob for each task, then all clients can do whatever it is they want to do with the data. I'd also get rid of things like progress and instead focus on allowing clients to do their own calculations and display.
        Hide
        Paul Joseph Davis added a comment -

        Awesome.

        Two comments:

        1. I don't see a reason to expose the JSON in Futon. The JSON is surely for script consumption. I was mulling over something similar to this and was thinking of replacing the current status entry with JSON like you have and then Futon and other tools would just build the status string client side. This would allow scripts to consume the same information without resorting to string processing.

        Also, I always found it quite annoying to be writing user facing text deep in the various internals like view indexing or compaction.

        2. I think it be a good idea for the stats to include the db name and view/design name when appropriate.

        Show
        Paul Joseph Davis added a comment - Awesome. Two comments: 1. I don't see a reason to expose the JSON in Futon. The JSON is surely for script consumption. I was mulling over something similar to this and was thinking of replacing the current status entry with JSON like you have and then Futon and other tools would just build the status string client side. This would allow scripts to consume the same information without resorting to string processing. Also, I always found it quite annoying to be writing user facing text deep in the various internals like view indexing or compaction. 2. I think it be a good idea for the stats to include the db name and view/design name when appropriate.

          People

          • Assignee:
            Filipe Manana
            Reporter:
            Filipe Manana
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development