+1 I think it would make sense to make this change.

yes memory is not right

@spf13
This is a little harder then it seems, as it requires modifying libcontainer code here: https://github.com/docker/libcontainer/blob/master/cgroups/fs/memory.go
Do we use memory.stat.rss, memory.stat.usage_in_bytes - memory.stat.cache, or some other combination of stats...?

I switched to the proficient label

On Sunday, March 22, 2015, Matthew Gallagher notifications@github.com
wrote:

@spf13 https://github.com/spf13
This is a little harder then it seems, as it requires modifying
libcontainer code here:
https://github.com/docker/libcontainer/blob/master/cgroups/fs/memory.go
Do we use memory.stat.rss, memory.stat.usage_in_bytes - memory.stat.cache,
or some other combination of stats...?

—
Reply to this email directly or view it on GitHub
#10824 (comment).

Try set the param of docker run --memory，then check your
/sys/fs/cgroup/memory/docker/<container_id>/memory.usage_in_bytes
It should be right.

#dibs ping @wonderflow, I think this is what you are trying to fix. Just send a PR.

Hi @resouer @wonderflow,
Please remember to include the keywords closes or fixes once you've submitted your PR to help make tracking of fixes easier. Thanks!

"Also recommend using standard unix postfixes here, so it 1.311G not 1.311 GiB"
This is related using function HumanSize or BytesSize in size.go.

I'm not sure if I should change. @crosbymichael
cc @SamSaffron

@wonderflow thanks for the fixes !

I can’t find your comment on github.

Can you tell me more details about it? Thanks.

On Jul 21, 2015, at 12:14 AM, Etienne Bruines notifications@github.com wrote:

Since this PR (#12172 #12172) was merged April 14th, and I'm running 1.6.2 (released May 13th), I believe I'm running a version that should have this fix.

However, docker stats still returns the memory usage including the buffers.

containername 0.00% 218.4 MiB/512 MiB 42.66% 17.22 KiB/130.1 KiB

$ free -m
total used free shared buffers
Mem: 971 889 82 26 46
-/+ buffers: 842 129
Swap: 2047 0 2047
I like the fact that it's not saying the usage is 889M, but 889 - 842 != 218.

Any way this can be fixed?

(Posting in this issue because it feels like the same issue, that's not 100% fixed, instead of a new issue)

—
Reply to this email directly or view it on GitHub #10824 (comment).

Just want to point out that this has had me chasing phantom memory leaks for more hours than I care to admit ;)

top says "18.4" MB, docker stats says "192.9" MB.

Update:

Here's the part that worries me, if I set a memory limit on the container of 32MB, it looks like my container will be killed even though resident memory doesn't go above 18.4 MB.

Is this resolved? I have the same problem as @pdericson with my apps running in docker containers (running on my local machine). top shows me 15MB in RES and the docker stats show a slowly linearly increasing memory usage. Which climbs to around 250MB over 12 hours.
I am using Docker Version: 1.12.5

@theonlydoo If I understand correctly, the memory usage in docker stats is exactly read from containers's memory cgroup, you can see the value is the same with 490270720 which you read from cat /sys/fs/cgroup/memory/docker/665e99f8b760c0300f10d3d9b35b1a5e5fdcf1b7e4a0e27c1b6ff100981d9a69/memory.usage_in_bytes , and the limit is also the memory cgroup limit which is set by -m when you create container. The statistics of RES and memory cgroup are different, the RES does not take caches into account, but the memory cgroup does, that's why MEM USAGE in docker stats is much more than RES in top

Hope this helps :)

@coolljt0725 so caches are increasing as the app is writing logs on the host's filesystem, and it triggers
oom reaping even if the app isn't using that memory and it's only caches.

OOM reaping is handled by the kernel though

@Sergeant007, your suggested container memory usage formula does not match the sum of process memory usage for me. When I compare memory used from cgroups (minus cache) to the sum of process rss, I get a not so small difference:

# RSS_USED=$(ps -eo rss --sort -rss --no-headers|awk '{RSS+=$1}END{print RSS*1024}')
# MEM_USED=$((`cat /sys/fs/cgroup/memory/memory.usage_in_bytes` - `awk '/^cache/{print $2}' /sys/fs/cgroup/memory/memory.stat`))
# echo $RSS_USED - $MEM_USED = $(($RSS_USED-$MEM_USED))
27811590144 - 27780145152 = 31444992

Does ps not reflect actual memory usage of processes in the container like some other tools?

^{This is in a CentOS 7.3 container running on Amazon Linux 2016.09.}

@axelabs, I'm not sure if ps work 100% accurate, but even my method described above does not pretend to be fully correct. It just fixes the original behavior that was really-really misleading.

So in your particular measurement the difference was 32MB out of 27GB of consumed memory. This is 0.1%. I'm not sure if there is an easier to achieve something better. And I'm not even sure which one (RSS_USED or MEM_USED) was correct.

If anyone interested why it could be misleading :
https://medium.com/@bobzsj87/demist-the-memory-ghost-d6b7cf45dd2a

I got headaches trying to understand those misleading stats until reading this article.

@Sergeant007

Step 6. Conclusion
If you need more reliable statistics for your monitoring, just subtract "cache" from "usage", what in our case would give 896794624-896299008=495616 (400KB instead of reported 900MB).

Shouldn't taking the value of "rss": 495616 be enough ?

Hi @omasseau,

It's been so long time ago (6 years), that I'm not even sure if "just taking rss" was correct or not. As far as I remember, subtracting filesystem caches was the only reliable way. If you're working on this issue, then just play with the "steps to reproduce" I posted on Apr 9, 2017 to reconfirm the numbers. I'm not actively working with Docker at my current project, so can't investigate...