@jpetazzo can you take a look at this one?

This is a kind of bug in AUFS. When a directory has a given permission mask in a lower layer, the upper layers cannot have a broader mask. Well, they can, but the more restrictive permission mask will be enforced anyways.

The rationale is the following:

• suppose that you have directory /secret with permissions 0700, containing file /secret/key.pem
• in an upper layer, you give /secret permissions 0755
• now /secret/key.pem could become reachable

Multiple behaviors could be considered "acceptable" in this scenario:

• give access to the file anyway (but this option was vetoed because it was deemed insecure)
• prevent access
• place a kind of "tombstone" or "opaque whiteout" for the whole directory so that the directory below becomes "opaqued" or "whited out", and the new one takes precedence

My understanding is that the last solution should be used, but for some reason, AUFS doesn't behave correctly. It might be because the directory exists in a lower layer, then doesn't exist anymore (because of the rm), then exists again (because of the ADD).

I'm willing to take a guess: the logic that decides whether or not to do a whiteout is not exactly the same as the one looking up permissions; so the first one stops when /etc/puppet is marked as non-existent in the middle layer, while the latter goes bottom up.

Anyway!

As a workaround, you can rm /etc/puppet/* instead of rm /etc/puppet, and that will do the trick.

Labelling as aufs-related.

Since this is documented aufs behavior, we can 1) close this as wontfix, 2) close + document the behavior in the docker docs, or 3) other?

@jpetazzo what do you think?

Hmmm... What about a "KNOWN BUGS AND ISSUES" section in the documentation? /cc @metalivedev

Ok, some discussion here: https://botbot.me/freenode/docker-dev/msg/6889335/
I'll look at how to provide the known issues in context, which implies that the docs need some place to describe the file system in use.

This bit me.

@xaviershay Could you give me the steps to reproduce the problem? I haven't been able to see an error with @astraw 's steps.

vagrant@precise64:~/src/783$ docker version
Client version: 0.6.4
Go version (client): go1.1.2
Git commit (client): 2f74b1c
Server version: 0.6.4
Git commit (server): 2f74b1c
Go version (server): go1.1.2
Last stable version: 0.6.4

vagrant@precise64:~/src/783$ docker build -t dockbug .
Uploading context 10240 bytes
Step 1 : FROM ubuntu:12.10
Pulling repository ubuntu
...
Step 6 : RUN chmod 755 /etc/puppet
 ---> Running in cf062fd724fb
 ---> 17620a2e9ea7
Successfully built 17620a2e9ea7

vagrant@precise64:~/src/783$ echo "note the directory is owned by puppet with full read/write/execute privs"
note the directory is owned by puppet with full read/write/execute privs

vagrant@precise64:~/src/783$ docker run dockbug ls -al /etc/puppet
total 12
drwxr-xr-x  2 puppet puppet 4096 Oct 19 00:18 .
drwxr-xr-x 78 root   root   4096 Oct 19 00:19 ..
-rw-rw-r--  1 puppet puppet    3 Oct 19 00:18 hello.txt

vagrant@precise64:~/src/783$ echo "but we get a permission error here"
but we get a permission error here

vagrant@precise64:~/src/783$ docker run dockbug sudo -u puppet ls -al /etc/puppet
total 12
drwxr-xr-x  2 puppet puppet 4096 Oct 19 00:18 .
drwxr-xr-x 78 root   root   4096 Oct 19 00:19 ..
-rw-rw-r--  1 puppet puppet    3 Oct 19 00:18 hello.txt

# HMM, no permission error

Having trouble replicating again.

The shape of my setup was building an image on top of itself (which is potentially a terrible idea):

# Dockerfile.bootstrap
FROM ubuntu:12.10

# Dockerfile
FROM thisimage

RUN rm -Rf /app
RUN mkdir /app

docker build -t thisimage - < Dockerfile.boostrap
docker build -t thisimage - < Dockerfile
docker build -t thisimage - < Dockerfile # Do this a couple of times.

That looks to me like you'd run into the AUFS 42 layer limit pretty quickly.

Yeah that was the next bug I hit :)

Have since redone my process to always build off a single base image.

rm /etc/puppet/* didn't work for me, instead I found that deleting individual files on the same layer works perfectly well:

find /etc/pappet -type f | xargs -L1 rm -f

Here is link to my Gist

Thanks, it works for me!