CouchDB
  1. CouchDB
  2. COUCHDB-1165

Allow CouchDB-managed OS daemons to include information in _active_tasks

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
      None
    • Skill Level:
      Regular Contributors Level (Easy to Medium)

      Description

      The new Externals API coupled with the HTTP Proxying enables powerful background data processing scenarios, similar to the replication and compaction API's in the core CouchDB API. The Replication API returns a JSON structure that includes information about the replication that has completed but progress information is also available through _active_tasks and thus, Futon.

      An OS daemon that handles proxied requests to perform data processing tasks can also reply with information about a completed request but has no way other than ["log", DMESG] to communicate progress and append to the CouchDB log, similar how view checkpointing is included in the CouchDB log.

      The suggested JSON commands supported by the Externals API would allow a daemon to register a long-running tasks, report completion progress and status and signal when the long running task completes.

      While I am not intimately familiar with the inner workings of _active_tasks and

      {couch_httpd_misc_handlers, handle_task_status_req}

      , the following API would work (suggestions welcome!):

      Register a new active task:

      ["task",

      { "id":"import:foo.csv", "type": "File Import", "task":"foo.csv", "status": "Imported 0 of 360,410 bytes (0%)" }

      ]

      Report progress (same structure as registering a new active task):

      ["task",

      { "id":"import:foo.csv", "type": "File Import", "task":"foo.csv", "status": "Imported 150,428 of 360,410 bytes (41%)" }

      ]

      Register task completion:

      ["task",

      {"id":"import:foo.csv", complete: true}

      ]

      Of course, id could be a number id that uniquely identifies the process itself. Currently, active tasks include the "pid" of the process that performs the task.

      It would be lovely to see this functionality in 1.2 and I will have time to work on this in the 1.2 timeframe so if there are no objections and everyone agrees on a meaningful command structure, I can work on the patch.

      1. COUCHDB-1165.patch
        19 kB
        Paul Joseph Davis

        Activity

        Hide
        Paul Joseph Davis added a comment -

        @Gabor,

        That's a pretty good idea. The only hiccup is that it'll require refactoring the way that cuch_task_status works, but scanning through it I already see how to manage that. If you're interested in giving a go at this yourself feel free to ping me directly with questions on how to go about this.

        The API itself I'd probably change up a bit though. Something more like:

        % Create task
        <- ["task_create", Type, Task, Status]
        -> [true, TaskId]

        % Update task
        <- ["task_update", TaskId, NewStatus]
        % no response

        % Delete task
        <- ["task_done", TaskId]

        Where TaskId, Typ, Task, and Status are all strings.

        Show
        Paul Joseph Davis added a comment - @Gabor, That's a pretty good idea. The only hiccup is that it'll require refactoring the way that cuch_task_status works, but scanning through it I already see how to manage that. If you're interested in giving a go at this yourself feel free to ping me directly with questions on how to go about this. The API itself I'd probably change up a bit though. Something more like: % Create task <- ["task_create", Type, Task, Status] -> [true, TaskId] % Update task <- ["task_update", TaskId, NewStatus] % no response % Delete task <- ["task_done", TaskId] Where TaskId, Typ, Task, and Status are all strings.
        Hide
        Gabor Ratky added a comment -

        Great! API suggestions make an awful lot of sense, although the JSON array API over stdin/stdout is exotic by itself . I'd love to take a crack at it, great exercise to ramp up on couchdb-dev.

        Are we over any sort of feature freeze for 1.1? I can put this on the front burner if it's only a question of timing.

        Show
        Gabor Ratky added a comment - Great! API suggestions make an awful lot of sense, although the JSON array API over stdin/stdout is exotic by itself . I'd love to take a crack at it, great exercise to ramp up on couchdb-dev. Are we over any sort of feature freeze for 1.1? I can put this on the front burner if it's only a question of timing.
        Hide
        Paul Joseph Davis added a comment -

        This won't make 1.1 as its already been branched. Though I wouldn't wait too long to take a crack at it because I might get bored and write it.

        Show
        Paul Joseph Davis added a comment - This won't make 1.1 as its already been branched. Though I wouldn't wait too long to take a crack at it because I might get bored and write it.
        Hide
        Gabor Ratky added a comment -

        Do it then. I will look at the patch and pull together the build environment anyways. I'd rather `brew install couchdb --HEAD` and have this functionality included as I would use it immediately.

        Show
        Gabor Ratky added a comment - Do it then. I will look at the patch and pull together the build environment anyways. I'd rather `brew install couchdb --HEAD` and have this functionality included as I would use it immediately.
        Hide
        Paul Joseph Davis added a comment -

        Patch to allow os_daemons to register tasks in _active_tasks. Patch is also available as a comparison on GitHub at [1].

        [1] https://github.com/davisp/couchdb/compare/112abf7...os-tasks

        Show
        Paul Joseph Davis added a comment - Patch to allow os_daemons to register tasks in _active_tasks. Patch is also available as a comparison on GitHub at [1] . [1] https://github.com/davisp/couchdb/compare/112abf7...os-tasks
        Hide
        Alexander Shorin added a comment -

        Bump. What the status?

        Show
        Alexander Shorin added a comment - Bump. What the status?
        Hide
        Paul Joseph Davis added a comment -

        Looks like this code is rather out of date compared to master. It shouldn't be a huge issue to fix but the tests will need to be updated among other things.

        Show
        Paul Joseph Davis added a comment - Looks like this code is rather out of date compared to master. It shouldn't be a huge issue to fix but the tests will need to be updated among other things.

          People

          • Assignee:
            Unassigned
            Reporter:
            Gabor Ratky
          • Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:

              Development