Skip to content

chown error when restarting containers #939

Closed
@tsmith-rv

Description

@tsmith-rv

Similar to #852, when using images mysql:5.7 and mysql:8.0 I receive chown errors when trying to restart the container. The error specifically is: chown: changing ownership of '/var/lib/mysql/mysql.sock': No such file or directory.

Here is my docker-compose file:

version: '3.3'

services:
  db:
    platform: linux/x86_64
    image: mysql:8.0
    restart: always
    environment:
      MYSQL_DATABASE: 'database1'
      MYSQL_ROOT_PASSWORD: 'pass'
    ports:
      - '3306:3306'
    expose:
      - '3306'
    command: --sql-mode=''
    volumes:
      - ./_mysql-docker:/var/lib/mysql

Whether I stop the containers with Ctrl+C and run docker-compose up, or run docker stop then docker start I see the same issue. The first time I start the container it works fine, then subsequent runs produce the error in the output below.

My container logs when trying to start the container again:

db_1  | 2023-01-18 04:38:06+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.32-1.el8 started.
db_1  | chown: changing ownership of '/var/lib/mysql/mysql.sock': No such file or directory
db_1  | chown: changing ownership of '/var/lib/mysql/mysql.sock': No such file or directory

My host directory _mysql_docker contains the mysql.sock symbolic link and as a sanity test I checked inside the container and that symbolic link is exactly where it should be at /var/lib/mysql/mysql.sock.

My host machine is ARM running macOS Monterey.

Activity

dbagwell

dbagwell commented on Jan 18, 2023

@dbagwell

I am getting the same error. If I remove the mysql.sock symlink from my host directory before starting the container, the container starts up just fine (recreating the symlink in the host directory).

tsmith-rv

tsmith-rv commented on Jan 18, 2023

@tsmith-rv
Author

I am getting the same error. If I remove the mysql.sock symlink from my host directory before starting the container, the container starts up just fine (recreating the symlink in the host directory).

Same @dbagwell! Any solution that involves deleting mysql.sock before starting the container allows the container to start with no problem. Seems like one solution might involve cleaning up at the symbolic link at some point in the process, but really I just want to know why chown says it can't find the file. That part just has me stumped.

tianon

tianon commented on Jan 18, 2023

@tianon
Member

That's weird -- even if the file is a dangling symlink, this should work fine because we use --no-dereference (so it should change the ownership of the symlink, which should work fine):

find "${!dirs[@]}" \! -user mysql -exec chown --no-dereference mysql '{}' +

tsmith-rv

tsmith-rv commented on Jan 18, 2023

@tsmith-rv
Author

That's weird -- even if the file is a dangling symlink, this should work fine because we use --no-dereference (so it should change the ownership of the symlink, which should work fine):

find "${!dirs[@]}" \! -user mysql -exec chown --no-dereference mysql '{}' +

Oh agreed for sure. Seems like some subtle nuance of how chown works, maybe? My guess is that "No such file or directory" is actually just a bit of a red herring since the symlink is obviously there. Possibly something negating the --no-dereference switch?

tianon

tianon commented on Jan 18, 2023

@tianon
Member

Do you still have a way to reliably reproduce? Could you maybe try something smaller like docker run --rm --mount ... mysql:image-you-use chown --no-dereference mysql /var/lib/mysql/mysql.sock directly?

(If you can reproduce reliably, steps to help others reproduce would be useful 😅)

tsmith-rv

tsmith-rv commented on Jan 19, 2023

@tsmith-rv
Author

@tianon sure! Here is an explicit list of steps I'm following:

  1. From your terminal, run docker-compose -f ./docker-compose.local.yml up (docker-compose.local.yml contains the contents shown above in my original comment)
  2. In that same terminal, once you receive the "all good" from your container (something like db_1 | Version: '5.7.41' socket: '/var/run/mysqld/mysqld.sock' port: 3306 MySQL Community Server (GPL)) , press Ctrl+C at which point you'll see something like Gracefully stopping... (press Ctrl+C again to force). Wait until you get Stopping service_db_1 ... done.
  3. Continuing in that same terminal, run docker-compose -f ./docker-compose.local.yml up again. You should then see:
db_1  | 2023-01-19 00:28:27+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.41-1.el7 started.
db_1  | chown: changing ownership of '/var/lib/mysql/mysql.sock': No such file or directory
db_1  | chown: changing ownership of '/var/lib/mysql/mysql.sock': No such file or directory

then the container stops.

Note: I have also tried this same methodology, but instead replacing step 2 and step 3 with docker stop and docker start commands respectively and achieved the same result.

This is reliable as hell for me. Never fails (or, you know, succeeds?)

tianon

tianon commented on Jan 19, 2023

@tianon
Member

Doh, I guess this is probably something about either macOS not supporting chown on a symlink or the Docker Desktop file sharing not supporting chown on a symlink? Not sure, but I think the interaction between those is probably the culprit (to be honest, I'm surprised you're not seeing more issues with MySQL itself not liking its database files existing on that shared filesystem too).

tianon

tianon commented on Jan 19, 2023

@tianon
Member

To put that another way, if you switch from a bind-mount of /var/lib/mysql to your "host" system (outside the Docker Desktop VM) to a named volume (which then lives inside the VM), this problem should go away.

tsmith-rv

tsmith-rv commented on Jan 19, 2023

@tsmith-rv
Author

So, the good news...

@tianon heeeeeey switching to the named volume did work.

Funny enough, on my host machine, I tried running chown on symbolic links and that didn't return an error (even forced a "No such file or directory" error by trying it on a non-existent file to make sure it produces an error on macOS).

Here is what my file looks like now, which is working (and has actually seriously reduced database startup times as well as reduced latency on database inserts):

version: '3.3'

services:
  db:
    platform: linux/x86_64
    image: mysql:8.0
    restart: always
    environment:
      MYSQL_DATABASE: 'database1'
      MYSQL_ROOT_PASSWORD: 'pass'
    ports:
      - '3306:3306'
    expose:
      - '3306'
    command: --sql-mode=''
    volumes:
      - db:/var/lib/mysql

volumes:
  db:

Also:

I'm surprised you're not seeing more issues with MySQL itself not liking its database files existing on that shared filesystem

Funny you should mention it: we have. It is has been a huge headache, actually. We actually started seeing issues all of a sudden because of case sensitivity issues in the database files, but we resolved the issue using the top answer here: https://stackoverflow.com/questions/64146845/mysql-not-starting-in-a-docker-container-on-macos-after-docker-update

Of course what we didn't realize is that we were fixing a symptom rather than the cause. After switching to the named volume, I tested running the containers and our tests with the "grpc fuse for file sharing" option enabled again (and restarting docker, using new images, containers, volumes, etc.) and everything is still working, which means we no longer need this "workaround".

But still...

While I am now going to use the named volume instead (thank you very much @tianon), doesn't the problem I initially reported still seem like a bug? It seems like regardless of whether someone is running the oracle images on mac or not (arm or not? still not sure what the actual problem with chown is here) they should reasonably expect container restarts to be successful.

tsmith-rv

tsmith-rv commented on Jan 19, 2023

@tsmith-rv
Author

Heeeeeyyyyyyy making some progress here (maybe).

If I add a group to the chown invocation, it works. So, I updated the entrypoint.sh script with find "${!dirs[@]}" \! -user mysql -exec chown --no-dereference mysql:root '{}' + and no longer receive the error.

rabbitlhf

rabbitlhf commented on Jan 29, 2023

@rabbitlhf

here is my way:

  1. find the user and group id on host machine with command id in macos
  2. add to docker-compose file with 'user' label :user: 501:20
amushate

amushate commented on Apr 12, 2023

@amushate

this worked for me by adding user and rw as below. Note user has to be mysql not your root user

version: '3.3'

services:
  db:
    platform: linux/x86_64
    image: mysql:8.0
    restart: always
    environment:
      MYSQL_DATABASE: 'database1'
      MYSQL_ROOT_PASSWORD: 'pass'
    ports:
      - '3306:3306'
    expose:
      - '3306'
    command: --sql-mode=''
    volumes:
      - db:/var/lib/mysql:rw
    user: mysql

volumes:
  db:
JoseSecmotic

JoseSecmotic commented on Aug 17, 2023

@JoseSecmotic

In my case, I removed the mysql.sock file and it did work again

tianon

tianon commented on Dec 19, 2023

@tianon
Member

Running this image with the database shared over Docker Desktop's (or other) file sharing which is effectively remote file sharing is not likely to be a good time, and not likely to be well supported by MySQL itself. It's unfortunate that it doesn't work out-of-the-box, but I don't think it's going to be super fruitful for us to be chasing workarounds due to esoteric filesystems that MySQL upstream doesn't officially support either. 🙇

ahzhol84

ahzhol84 commented on Oct 27, 2024

@ahzhol84
tianon

tianon commented on Oct 28, 2024

@tianon
fakerdog1

fakerdog1 commented on Feb 13, 2025

@fakerdog1

A bit late to the party, but I have an sh script which you can put into your docker-compose file to handle this

 volumes:
    ....
    - ./mysql-entrypoint-wrapper.sh:/docker-mysql-entrypoint-wrapper.sh:ro
 entrypoint: [ "/docker-mysql-entrypoint-wrapper.sh" ]
#!/bin/bash
set -eo pipefail
rm -f /var/lib/mysql/mysql.sock

if [ "$#" -eq 0 ]; then
    set -- mysqld
fi

exec /usr/local/bin/docker-entrypoint.sh "$@"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @tianon@rabbitlhf@dbagwell@amushate@tsmith-rv

        Issue actions

          chown error when restarting containers · Issue #939 · docker-library/mysql