Uploaded image for project: 'Apache Nemo'
  1. Apache Nemo
  2. NEMO-20

RESTful APIs to Access Job State and Metric

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      We should provide RESTful APIs to access various job states and metric (ex. execution times) for NEMO users.

      by Yunseong Lee
      I'm implementing the HTTP server on my branch 528-rest-api.

      The description of APIs can be found at https://goo.gl/4ngKBb; currently job-state and executors are implemented.

      You can test the feature by simply running an example and by sending HTTP requests as follows:

      $ curl "http://localhost:19614/vortex/v1/executors"
      {"running": [Executor-2, Executor-3, Executor-1], "failed": []}
      $ curl "http://localhost:19614/vortex/v1/job-state"
      {"dag": {"vertices": [{"id": "Stage-0", "properties": {"scheduleGroupIndex": 0, "taskGroupList": [{"taskGroupId": "TaskGroup-1", "taskGroupIdx": 0, "taskDAG": {"vertices": [{"id": "Task-1", "properties": {"runtimeVertexId": "vertex1", "index": 0}}, {"id": "Task-2", "properties": {"runtimeVertexId": "vertex5", "index": 0}}],
       "edges": [{"src": "Task-1", "dst": "Task-2", "properties": {"runtimeEdgeId": "edge4", "edgeProperties": {"DataCommunicationPattern": "class edu.snu.vortex.runtime.executor.datatransfer.data_communication_pattern.OneToOne", "DataFlowModel": "Pull", "DataStore": "class edu.snu.vortex.runtime.executor.data.MemoryStore", "Partitioning": "class edu.snu.vortex.runtime.executor.datatransfer.partitioning.Hash"}, "coder": "StringUtf8Coder"}}]}, "containerType": "None"}]}},
      ...
      

      NOTE: For now, you need to find the HTTP server's port number from the `driver.stderr` log, but we can set `TcpPortRangeBegin` with which Driver starts scanning available ports.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            skystar Jae Hyeon Park
            sanha Sanha Lee
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment