Hm ok apparently it is required to use smbpasswd as backend.

On the host I created my used with
pdbedit -b smbpasswd -a <username>

I also added
passdb backend = smbpasswd to my hosts smb.conf for future use.

Then I bound it to the docker container
--volume /etc/samba/private/smbpasswd:/etc/samba/private/smbpasswd

and used the -i option:
-i "/etc/samba/private/smbpasswd"

I'm just not sure how well that will be in sync with my users. But that's another problem.