Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
Reviewed
-
Ups the nproc (processes) limit from 12500 to 30000 in yetus (so build container can have new limit).
Description
We (Apache Phoenix team) have recently switched our precommit tests to Dockerized Yetus (mostly adopted from the solution in Hbase)
We see
java.lang.OutOfMemoryError: unable to create new native thread
errors , while Yetus shows
Max. process+thread count | 6833 (vs. ulimit of 12500) |
While I couldn't determine what was the job that we shared the Agent with at the time, statistically it was very likely HBase, and an HBase job probably failed with a similar error.
Some research has thrown up the official Docker docs:
https://docs.docker.com/engine/reference/commandline/run/#set-ulimits-in-container-ulimit
According to which it is not possible to set container level nprocs ulimit with Docker.
All settings apply to the docker Daemon user instead, and the limit is shared between all containers.
Based on this, I think that it makes no sense to set a container (really docker user) nprocs ulimit any lower than the current hard limit of 30000.
I have already set PROC_LIMIT=30000 in the Phoenix Yetus personality, but it is only a half solution until some Docker users set lower values, as the later setting will apply as soon as the container is started.
Attachments
Attachments
Issue Links
- is related to
-
HBASE-24126 Up the container nproc uplimit from 10000 to 12500
- Resolved
- relates to
-
PHOENIX-6111 Jenkins jobs are unable to create new native thread
- Resolved
- links to