[MESOS-5021] Memory leak in subprocess when 'environment' argument is provided.

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.23.0, 0.23.1, 0.24.0, 0.24.1, 0.25.0, 0.26.0, 0.27.0, 0.27.1, 0.27.2, 0.28.0
• Fix Version/s: 0.24.2, 0.25.1, 0.26.1, 0.27.3, 0.28.1
• Component/s: agent, libprocess
• Assignee: Benjamin Mahler
• Reporter: Benjamin Mahler
• Created: 24/Mar/16 03:18
• Updated: 28/Mar/16 19:54
• Resolved: 24/Mar/16 03:59

Description

A memory leak in process::subprocess was introduced here:
https://github.com/apache/mesos/commit/14b49f31840ff1523b31007c21b12c604700323f

This was found when jieyu and I examined a memory leak in the health check program (see MESOS-4869).

The leak is here:
https://github.com/apache/mesos/blob/0.28.0/3rdparty/libprocess/src/subprocess.cpp#L451-L456

  // Like above, we need to construct the environment that we'll pass
  // to 'os::execvpe' as it might not be async-safe to perform the
  // memory allocations.
  char** envp = os::raw::environment();

  if (environment.isSome()) {
    // NOTE: We add 1 to the size for a NULL terminator.
    envp = new char*[environment.get().size() + 1];

    size_t index = 0;
    foreachpair (const string& key, const string& value, environment.get()) {
      string entry = key + "=" + value;
      envp[index] = new char[entry.size() + 1];
      strncpy(envp[index], entry.c_str(), entry.size() + 1);
      ++index;
    }

    envp[index] = NULL;
  }

  ...

  // Need to delete 'envp' if we had environment variables passed to
  // us and we needed to allocate the space.
  if (environment.isSome()) {
    CHECK_NE(os::raw::environment(), envp);
    delete[] envp; // XXX Does not delete the sub arrays.
  }

Auditing the code, it appears to affect a number of locations:

• docker::run
• health check binary
• liblogrotate
• Docker containerizer: here and here
• External containerizer
• Posix launcher and Linux launcher
• Fetcher

Issue Links

• is related to: MESOS-3487 Running libprocess tests in a loop leads to unbounded memory growth.
• relates to: MESOS-4869 /usr/libexec/mesos/mesos-health-check using/leaking a lot of memory