Skip to content

Docker.qcow2 never shrinks - disk space usage leak in docker for mac #371

New issue
New issue
Closed
Closed
Docker.qcow2 never shrinks - disk space usage leak in docker for mac#371
Labels
area/storagekind/bugkind/enhancementlifecycle/frozenlifecycle/stalestatus/2-in-progress
@yakticus

Description

@yakticus
yakticus
opened on Aug 19, 2016 · edited by yakticus

Expected behavior

Amount of disk space used under ~/Library/Containers/com.docker.docker/ should shrink when deleting images and/or containers

Actual behavior

Amount of disk space used under ~/Library/Containers/com.docker.docker/ doesn't shrink after deleting ~50 containers and ~141 images (out of a total of 194). My Docker.qcow2 was 41GB both before and after doing these deletions.

On the Diagnose & Feedback screen I see an error that looks relevant:

Docker for Mac: version: mac-v1.12.0.1
OS X: version 10.11.6 (build: 15G31)
logs: /tmp/5F7A165B-83AC-4514-AFC4-DB53430D2E9D/20160819-141130.tar.gz
[OK]     docker-cli
[OK]     app
[OK]     moby-syslog
[ERROR]  disk
         disk check failed with: Failure("exec: /Applications/Docker.app/Contents/Resources/bin/../../../Contents/MacOS/qemu-img check /Users/julie/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2 > /tmp/5F7A165B-83AC-4514-AFC4-DB53430D2E9D/20160819-141130/qemu-img-check.stdout 2> /tmp/5F7A165B-83AC-4514-AFC4-DB53430D2E9D/20160819-141130/qemu-img-check.stderr: exit 2")
[OK]     virtualization
[OK]     system
[OK]     menubar
[OK]     osxfs
[OK]     db
[OK]     slirp
[OK]     moby-console
[OK]     logs
[OK]     vmnetd
[OK]     env
[OK]     moby
[OK]     driver.amd64-linux

Information

For some history, see moby/moby#23437

  • Diagnostic ID from "Diagnose & Feedback" in the menu.
    I might not have this anymore since I had to do a factory reset in order to reclaim the disk space. After the factory reset, it is: 5F7A165B-83AC-4514-AFC4-DB53430D2E9D
  • a reproducible case if this is a bug, Dockerfiles FTW

Easy to reproduce by pulling some images, observing the size of Docker.qcow2, deleting the images and seeing that the size of Docker.qcow2 does not change. Let me know if more info is needed

  • page URL if this is a docs issue or the name of a man page
  • host distribution and version (OSX 10.10.x, OSX 10.11.x)

OSX 10.11.6

Steps to reproduce the behavior

  1. pull several images
  2. run a couple of containers
  3. observe size of ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
  4. stop and delete all containers
  5. delete all images
  6. repeat step 3. Size has not shrunk despite there being no images or containers

Transcript:

$ docker pull frolvlad/alpine-scala
Using default tag: latest
latest: Pulling from frolvlad/alpine-scala
e110a4a17941: Pull complete
052edde983dc: Pull complete
44b3390a7725: Pull complete
0f6b9d2879c1: Pull complete
Digest: sha256:1d9f6fd716526764a4ba6e389b635cbef8c5d413216bf5d00ca5175fc0d9bac8
Status: Downloaded newer image for frolvlad/alpine-scala:latest
$ du -sm ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
1290    /Users/julie/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
$ docker pull ubuntu
Using default tag: latest
latest: Pulling from library/ubuntu
2f0243478e1f: Pull complete
d8909ae88469: Pull complete
820f09abed29: Pull complete
01193a8f3d88: Pull complete
Digest: sha256:8e2324f2288c26e1393b63e680ee7844202391414dbd48497e9a4fd997cd3cbf
Status: Downloaded newer image for ubuntu:latest
$ du -sm ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
1456    /Users/julie/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
$ docker pull python
Using default tag: latest
latest: Pulling from library/python
357ea8c3d80b: Pull complete
52befadefd24: Pull complete
3c0732d5313c: Pull complete
ceb711c7e301: Pull complete
4211bb537697: Pull complete
71f9074c0739: Pull complete
3e5349707036: Pull complete
Digest: sha256:a755ad5a30b2146f9a132bf74107aa5be9a26b7cc77956b379af678e0f311b3c
Status: Downloaded newer image for python:latest
$ docker run ubuntu
$ docker run -it ubuntu bash
root@a2ae24adfd12:/# exit
$ docker run -itd ubuntu bash
95f5bed47a790d2c8d60544cbc1fa870e1ef4b8c04653508bed34fb25e650536
$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                      PORTS               NAMES
95f5bed47a79        ubuntu              "bash"              9 seconds ago       Up 9 seconds                                    hungry_lalande
a2ae24adfd12        ubuntu              "bash"              28 seconds ago      Exited (0) 21 seconds ago                       sharp_borg
a567a8a7aed3        ubuntu              "/bin/bash"         53 seconds ago      Exited (0) 52 seconds ago                       ecstatic_poitras
$ du -sm ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
2371    /Users/julie/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
$ docker stop hungry_lalande
hungry_lalande
$ docker rm -v $(docker ps -a -q -f status=exited)
95f5bed47a79
a2ae24adfd12
a567a8a7aed3
$ docker rmi $(docker images -q)
Untagged: ubuntu:latest
Untagged: ubuntu@sha256:8e2324f2288c26e1393b63e680ee7844202391414dbd48497e9a4fd997cd3cbf
Deleted: sha256:f8d79ba03c00bbcd8079cf05b7526ac8f4f422744aad8c3747a29a38ed8c4a41
Deleted: sha256:b4119cffabb62bd6374f5b06adcb820344490252b621cb8f6a8fad7f6f61b3ae
Deleted: sha256:83a17331cb4259e2690395586b641aabadf30256f25a032019b816a53a06ae6e
Deleted: sha256:30ae5404a524e7a713f86ebe832b1e198d1270d62b27c860721055b890a3ae9c
Deleted: sha256:d8d865b23727fa63dc7d4e9f02efc3e2433c8f220972689c6b6ba09e8136c162
Untagged: python:latest
Untagged: python@sha256:a755ad5a30b2146f9a132bf74107aa5be9a26b7cc77956b379af678e0f311b3c
Deleted: sha256:a08871375facdc1619fc6d19608a4555db479219b2853e2e11de05eddf97e4cb
Deleted: sha256:0878d93537f4951aa5fa1ca6fd7ef085aa993e520b5af01148d84704ef49abcf
Deleted: sha256:2d5707b912a332c89665a078a2dabb088cbb4e6982b0d3db36217c89640074da
Deleted: sha256:5f6d1bd4cff3a349d2b5989bf04295987b6d679c411357612523341d4b6f2264
Deleted: sha256:0e54104f61d3ae37dc10fbc275796123a40b54859776fb6e937bb5c39655c3e6
Deleted: sha256:d3c72419ca593d81f207884b70670fd69c4fa3bdecf926ecbbbd9d3bbdd6c7bf
Deleted: sha256:016fc440b1e6b49d889b0f325d2439c82c92f554d3f015db67ffde823bd2ff3e
Deleted: sha256:2f71b45e4e254ddceb187b1467f5471f0e14d7124ac2dd7fdd7ddbc76e13f0e5
Untagged: frolvlad/alpine-scala:latest
Untagged: frolvlad/alpine-scala@sha256:1d9f6fd716526764a4ba6e389b635cbef8c5d413216bf5d00ca5175fc0d9bac8
Deleted: sha256:ac1474efe52e017303f24209fff310c734eed81cdd947604545f160d03f2caad
Deleted: sha256:a08f99075408eb40ce74942b01d08003c8d923b8df0b0e593b73b13540850171
Deleted: sha256:5c86a5cfb5b67143c7ec277ac533176abb9906072eac82de512f8e822b06e769
Deleted: sha256:2615ff2c2f8c27913239080269f5be78d541b2564592856548b435f5daa42514
Deleted: sha256:4fe15f8d0ae69e169824f25f1d4da3015a48feeeeebb265cd2e328e15c6a869f
$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.12.0
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 0
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.4.15-moby
Operating System: Alpine Linux v3.4
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 1.954 GiB
Name: moby
ID: KQ7I:O3AX:CY77:DKU4:6LZR:Y6RF:ZM2P:U2YA:TZUV:3YDV:DQPU:OTJV
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 18
 Goroutines: 29
 System Time: 2016-08-19T21:24:33.294921776Z
 EventsListeners: 1
No Proxy: *.local, 169.254/16
Registry: https://index.docker.io/v1/
Insecure Registries:
 127.0.0.0/8
$ du -sm ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2
2371    /Users/julie/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2

Activity

kommen

kommen commented on Aug 19, 2016

@kommen
kommen
on Aug 19, 2016

Thanks for filing it over here @yakticus.

I also still have the problem and in the forum others have reported the same: https://forums.docker.com/t/docker-doesnt-release-disk-space-used-by-library-containers-com-docker-docker/15194

lxhunter

lxhunter commented on Aug 20, 2016

@lxhunter
lxhunter
on Aug 20, 2016

+1

noslouch
mentioned this on Aug 20, 2016
simond

simond commented on Aug 23, 2016

@simond
simond
on Aug 23, 2016

Yep, same issue here.

yankcrime

yankcrime commented on Aug 24, 2016

@yankcrime
yankcrime
on Aug 24, 2016

I've found the quickest way (as mentioned in other issues / forum threads) to fix this is to reset the client which forces the recreation of Docker.qcow2. If you don't want to do that however, there is another fix, once you've cleaned up any dangling images or exited containers.

NB: You'll need to install qemu via Homebrew as this process requires qemu-img to recompress the qcow2 disk image.

Connect to the VM with screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty and then login as root

Fill up the disk with zeroes:

dd if=/dev/zero of=/var/tempfile

This'll take at least couple of minutes on SSD. I wouldn't bother if you're on spinning rust...

Then you need to rm /var/tempfile, logout of the VM, and quit the Docker client entirely. Now we can recompress the disk:

$ pwd
/Users/nick/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux
$ mv Docker.qcow2 Docker.qcow2.original
$ du -hs Docker.qcow2.original
12G     Docker.qcow2.original
$ qemu-img convert -O qcow2 Docker.qcow2.original Docker.qcow2
$ rm Docker.qcow2.original
$ du -hs Docker.qcow2
772M    Docker.qcow2
geerlingguy

geerlingguy commented on Sep 2, 2016

@geerlingguy
geerlingguy
on Sep 2, 2016

@yankcrime's workaround works... but it's a lot of steps to make Docker not continuously gobble up all the remaining space on my poor little MacBook's SSD. Between this and osxfs perf issues (#77) requiring similarly involved file sync workarounds, it's really hard to recommend Docker for Mac to anyone who's not an infra hacker at heart... I don't want to have to spend half my day helping other devs figure out why all their disk space is gone!

kevinsimper

kevinsimper commented on Sep 5, 2016

@kevinsimper
kevinsimper
on Sep 5, 2016

Just deleting it seems to be the easiest solution, but it would be nice if the client could reclaim space itself!

gumho

gumho commented on Sep 6, 2016

@gumho
gumho
on Sep 6, 2016

I've been running into this issue as well on the latest Stable Docker for Mac. @yankcrime's workaround does work. Would be nice to be able to provide an option to grow the disk size as well.

creynders

creynders commented on Sep 8, 2016

@creynders
creynders
on Sep 8, 2016

👍 💯

chrisclark

chrisclark commented on Sep 9, 2016

@chrisclark
chrisclark
on Sep 9, 2016

To add to this, I just deleted my qcow2 file and restart docker and disconnect completely from the internet, and docker ps shows no running containers, and docker images -a lists two containers, both of which are < 300mb...Docker will recreate the qcow2 file and I can just sit there, watching it grow in size, while Docker, and I, do absolutely nothing. There are no running containers, no nothing. What on earth is it even doing? I'm literally sitting here staring at the file in Finder (after re-enabling my network...) watching it grow from 200kb to...

...1.14GB
...1.15GB
...1.16GB

This is nuts. It'll happily just keep growing until it eats all of my available hard disk space.

mochaslave

mochaslave commented on Sep 12, 2016

@mochaslave
mochaslave
on Sep 12, 2016

@chrisclark I have the same problem. There is no image, no container, qcow2 just keeping grow up, but until to 1.16GB. Seem 1.16GB is the VM basic requested working size.

marvin-w

marvin-w commented on Sep 13, 2016

@marvin-w
marvin-w
on Sep 13, 2016

Same issue for me

AzatKhalilov

AzatKhalilov commented on Sep 13, 2016

@AzatKhalilov
AzatKhalilov
on Sep 13, 2016

Yep! The same issue for me. If often rebuild my images it's trouble,headache! After 5-6 rebuild image has 1G size but disk space decrease on 20-30G!!!!!

dmill-bz

dmill-bz commented on Sep 14, 2016

@dmill-bz
dmill-bz
on Sep 14, 2016 · edited by dmill-bz

Can we have an update on this? It's a really big issue that needs solving.
I can appreciate that it might not be an easy fix but some kind of feedback would be awesome.

alex-ross

alex-ross commented on Sep 14, 2016

@alex-ross
alex-ross
on Sep 14, 2016

Using Docker to manage multiple development environments and this image often ends up to take 60GB for me. It's a lot on my 256GB SSD. All dev environments is using Docker Compose so I can reset docker once in a while but it's really annoying that I have to do so.

347 remaining items

CamJN

CamJN commented on Oct 16, 2019

@CamJN
CamJN
on Oct 16, 2019

Quoting from https://docs.docker.com/docker-for-mac/space/

It might take a few minutes to reclaim space on the host depending on the format of the disk image file:

If the file is named Docker.raw: space on the host should be reclaimed within a few seconds.
If the file is named Docker.qcow2: space will be freed by a background process after a few minutes.

Space is only freed when images are deleted. Space is not freed automatically when files are deleted inside running containers. To trigger a space reclamation at any point, run the command:

$ docker run --privileged --pid=host docker/desktop-reclaim-space

Note that many tools report the maximum file size, not the actual file size. To query the actual size of the file on the host from a terminal, run:

$ cd ~/Library/Containers/com.docker.docker/Data
$ cd vms/0/data
$ ls -klsh Docker.raw
2333548 -rw-r--r--@ 1 username  staff    64G Dec 13 17:42 Docker.raw

In this example, the actual size of the disk is 2333548 KB, whereas the maximum size of the disk is 64 GB.

cjancsar
mentioned this on Jan 22, 2020
charliereese

charliereese commented on Feb 3, 2020

@charliereese
charliereese
on Feb 3, 2020

This Stack Overflow thread fairly succinctly explains the high level cause and a quick solution (not perfect, but I was happy with it): https://stackoverflow.com/questions/49887747/what-is-docker-qcow2

Although I thought this behaviour was odd at first, after reading the SO thread, this doesn't really seem like a bug.

Maybe Docker for Mac should default to a smaller disk image size (16GB vs 64GB)?

wanliqun

wanliqun commented on Feb 14, 2020

@wanliqun
wanliqun
on Feb 14, 2020

I've tried this to trigger a manual TRIM from this blog:
Docker for Mac: reducing disk space （https://djs55.github.io/jekyll/update/2017/11/27/docker-for-mac-disk-space.html）

docker run --rm -it --privileged --pid=host walkerlee/nsenter -t 1 -m -u -i -n fstrim /var/lib/docker

spacelatte

spacelatte commented on Feb 15, 2020

@spacelatte
spacelatte
on Feb 15, 2020

@wanliqun link has extra parenthesis at the end :)

Should be:

https://djs55.github.io/jekyll/update/2017/11/27/docker-for-mac-disk-space.html

Btw since el-capitan got EOL (1) and Sierra (10.12) has APFS support (2), .qcow2 does not matter anymore.

dmasterstempus

dmasterstempus commented on Apr 13, 2021

@dmasterstempus
dmasterstempus
on Apr 13, 2021 · edited by dmasterstempus

@charliereese

This Stack Overflow thread fairly succinctly explains the high level cause and a quick solution (not perfect, but I was happy with it): https://stackoverflow.com/questions/49887747/what-is-docker-qcow2

Although I thought this behaviour was odd at first, after reading the SO thread, this doesn't really seem like a bug.

Maybe Docker for Mac should default to a smaller disk image size (16GB vs 64GB)?

From the linked stackoverflow answer (emphasis mine):

You can stop Docker and delete this file, however deleting it will also remove all your containers and images. And Docker will recreate this file on start.

Do you consider that a solution?!

ds-clearago

ds-clearago commented on May 17, 2021

@ds-clearago
ds-clearago
on May 17, 2021

See https://github.com/wanliqun/macos_docker_toolkit – that seems to do the trick. On my end docker system prune helped a lot.

joaocc

joaocc commented on Sep 18, 2021

@joaocc
joaocc
on Sep 18, 2021

Hi
When I try to run the reclaim disk space scripts on macos bigsur (11.5.2) and docker 4.0.1 (68347), in x64 mode, I get the following errors. it seems it is unable to set namespace.
Any ideas pls? Thx

$ docker run --rm --privileged --pid=host docker/desktop-reclaim-space
setns:mnt: Invalid argument

(base) bash-5.1$ docker run --rm -it --privileged --pid=host walkerlee/nsenter -t 1 -m -u -i -n fstrim /var/lib/docker
nsenter: reassociate to namespace 'ns/mnt' failed: Invalid argument
aaronkurz

aaronkurz commented on Feb 13, 2022

@aaronkurz
aaronkurz
on Feb 13, 2022

I just got the error mkdir: cannot create directory '/bitnami/postgresql/data': No space left on device

For me, docker system prune worked! 👍

niccolomineo

niccolomineo commented on May 15, 2023

@niccolomineo
niccolomineo
on May 15, 2023 · edited by niccolomineo

I have been struggling with this since forever.

Out of 58 gbs, pruning everything would only free a total of 11.

Only way to gain the whole space again is to manually delete ~/Library/Containers/com.docker.docker/Data/vms/0/data/Docker.raw

lorenrh

lorenrh commented on May 30, 2023

@lorenrh
lorenrh
on May 30, 2023
Member

Closing this issue because a fix has been released in Docker Desktop 4.20. See the release notes for more details.

lorenrh
closed this as completedon May 30, 2023
chamini2

chamini2 commented on May 30, 2023

@chamini2
chamini2
on May 30, 2023

Hey @lorenrh, the release notes say

You can now reclaim disk space more quickly when files are deleted in containers. Related to docker/for-mac#371.

And that links here.

Can you please explain how can this be done?

larssb

larssb commented on Jul 17, 2023

@larssb
larssb
on Jul 17, 2023 · edited by larssb

@lorenrh is this

Screenshot 2023-07-17 at 21 32 03

what you refer to?

Edit:

Hmm seems like it:

Screenshot 2023-07-17 at 21 36 31

From > https://docs.docker.com/desktop/faqs/macfaqs/

tyanyaw

tyanyaw commented on Sep 2, 2023

@tyanyaw
tyanyaw
on Sep 2, 2023

Come 2023, already done all docker system prune & clean, but the Docker.raw size still big.
after searching, I found with method to :

  1. SAVE needed image to exclude
  2. DELETE Docker.raw
  3. LOAD chosen image back

Here the script to automate it :
clean-docker-for-mac.sh

EXAMPLE :
Chosen image : docker/getting-started alpine
Command : ./clean-docker-for-mac.sh docker/getting-started alpine

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/storagekind/bugkind/enhancementlifecycle/frozenlifecycle/stalestatus/2-in-progress

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @kommen@mcmire@tcurdt@ak47@dbourguignon

        Issue actions
          Docker.qcow2 never shrinks - disk space usage leak in docker for mac · Issue #371 · docker/for-mac