Uploaded image for project: 'Aurora'
  1. Aurora
  2. AURORA-1320

when instance is running in docker container, thermos observer reports 0 resources

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Docker, Thermos
    • Labels:
      None

      Description

      To see the problem start a job inside a docker container and view the task/instance page. You'll cpu/ram/disk all at zero regardless of their actual usage.

      I see errors like this in the thermos observer log:

          W0513 18:41:39.415406 3564 process_collector_psutil.py:42] Error during process sampling [pid=112]: process no longer exists (pid=112)
          W0513 18:41:39.415612 3564 process_collector_psutil.py:76] Error during process sampling: process no longer exists (pid=112)
          W0513 18:41:39.513972 3564 process_collector_psutil.py:76] Error during process sampling: no process found with pid 122
      

      This is likely because observer is running in a different pid namespace than the process. One solution would be for the runner to write out the pid namespace it is running in to the checkpoint and then have observer enter that namespace while sampling.

      Or we can just get rid of the observer?

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                jaybuff Jay Buffington
              • Votes:
                1 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: