Python multiprocessing crashes docker container

  • A+
Category:Languages

There is simple python multiprocessing code that works like a charm, when I run it in console:

# mp.py import multiprocessing as mp   def do_smth():     print('something')   if __name__ == '__main__':     ctx = mp.get_context("spawn")     p = ctx.Process(target=do_smth, args=tuple())     p.start()     p.join() 

Result:

> $ python3 mp.py something 

Then I've created a simple Docker container with Dockerfile:

FROM python:3.6  ADD . /app WORKDIR /app 

And docker-compose.yml:

version: '3.6'  services:   bug:     build:       context: .     environment:       - PYTHONUNBUFFERED=1     command: su -c "python3.6 forever.py" 

Where forever.py is:

from time import sleep  if __name__ == '__main__':     i = 0     while True:         sleep(1.0)         i += 1         print(f'hello {i:3}') 

Now I run forever.py with docker compose:

> $ docker-compose build && docker-compose up  ... some output ... Attaching to mpbug_bug_1 bug_1  | hello   1 bug_1  | hello   2 bug_1  | hello   3 bug_1  | hello   4 

Up to this moment everything is good and understandable. But when I'm trying to run mp.py in the docker container it crashes without any message:

> $ docker exec -it mpbug_bug_1 /bin/bash root@09779ec47f9d:/app# python mp.py  something root@09779ec47f9d:/app# %  

Gist with the code can be found here: https://gist.github.com/ilalex/83649bf21ef50cb74a2df5db01686f18

Can you explain why docker container is crashed and how to do it without crashing?

Thank you in advance!

 


for a quick fix, do not use spawn start method, and/or do not use su -c ..., both are unnecessary IMO. change to:

p = mp.Process(target=do_smth, args=tuple()) 

or you could start container with --init option.

with spawn start method, Python will also start a semaphore tracker process to prevent semaphore leaking, you could see this process by pausing mp.py in the middle, it looks like:

472   463 /usr/local/bin/python3 -c from multiprocessing.semaphore_tracker import main;main(3) 

this process is started by mp.py but exited after mp.py, thus it will not be reaped by mp.py, but is supposed to be reaped by init by design.

the problem is there is no init in this container(namespace), instead of init, PID 1 is su -c, therefore the dead semaphore tracker process is adopted by su.

it appears that su think the dead child process is the command process(forever.py) mistakenly, without checking the relationship, so su exited blindly, as PID 1 exit, kernel kills all other processes in the container, including forever.py.

this behavior could be observed with strace:

docker run --security-opt seccomp:unconfined --rm -it ex_bug strace -e trace=process -f su -c 'python3 forever.py' 

will output error message like:

strace: Exit of unknown pid 14 ignored 

ref: Docker and the PID 1 zombie reaping problem (phusion.nl)

Comment

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen: