Skip to content

Dockerfile COPY with file globs will copy files from subdirectories to the destination directory #15858

Closed
@jfchevrette

Description

@jfchevrette

Description of problem:
When using COPY in a Dockerfile and using globs to copy files & folders, docker will (sometimes?) also copy files from subfolders to the destination folder.

$ docker version
Client:
 Version:      1.8.1
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   d12ea79
 Built:        Thu Aug 13 19:47:52 UTC 2015
 OS/Arch:      darwin/amd64

Server:
 Version:      1.8.0
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   0d03096
 Built:        Tue Aug 11 17:17:40 UTC 2015
 OS/Arch:      linux/amd64

$ docker info
Containers: 26
Images: 152
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 204
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.0.9-boot2docker
Operating System: Boot2Docker 1.8.0 (TCL 6.3); master : 7f12e95 - Tue Aug 11 17:55:16 UTC 2015
CPUs: 4
Total Memory: 3.858 GiB
Name: dev
ID: 7EON:IEHP:Z5QW:KG4Z:PG5J:DV4W:77S4:MJPX:2C5P:Z5UY:O22A:SYNK
Debug mode (server): true
File Descriptors: 42
Goroutines: 95
System Time: 2015-08-26T17:17:34.772268259Z
EventsListeners: 1
Init SHA1:
Init Path: /usr/local/bin/docker
Docker Root Dir: /mnt/sda1/var/lib/docker
Username: jfchevrette
Registry: https://index.docker.io/v1/
Labels:
 provider=vmwarefusion

$ uname -a
Darwin cerberus.local 14.5.0 Darwin Kernel Version 14.5.0: Wed Jul 29 02:26:53 PDT 2015; root:xnu-2782.40.9~1/RELEASE_X86_64 x86_64

Environment details:
Local setup on OSX /w boot2docker built with docker-machine

How to Reproduce:

Context

$ tree
.
├── Dockerfile
└── files
    ├── dir
    │   ├── dirfile1
    │   ├── dirfile2
    │   └── dirfile3
    ├── file1
    ├── file2
    └── file3

Dockerfile

FROM busybox

RUN mkdir /test
COPY files/* /test/

Actual Results

$ docker run -it copybug ls -1 /test/
dirfile1
dirfile2
dirfile3
file1
file2
file3

Expected Results
The resulting image should have the same directory structure from the context

Activity

jfchevrette

jfchevrette commented on Aug 26, 2015

@jfchevrette
Author

Updated original message with output from docker info and uname -a and reformatted it to be according to the issue reporting template.

jrabbit

jrabbit commented on Sep 1, 2015

@jrabbit
Contributor

I've had this on 1.6.2 and 1.8
https://gist.github.com/jrabbit/e4f864ca1664ec0dd288 second level directories are treated as first level ones should be for some reason?

for those googling: if you're having issues with COPY * /src try COPY / /src

duglin

duglin commented on Sep 1, 2015

@duglin
Contributor

@jfchevrette I think I know why this is happening.
You have COPY files/* /test/ which expands to COPY files/dir files/file1 files/file2 files/file /test/. If you split this up into individual COPY commands (e.g. COPY files/dir /test/) you'll see that (for better or worse) COPY will copy the contents of each arg dir into the destination dir. Not the arg dir itself, but the contents. If you added a 3rd level of dirs I bet those will stick around.

I'm not thrilled with that fact that COPY doesn't preserve the top-level dir but its been that way for a while now.

You can try to make this less painful by copying one level higher in the src tree, if possible.

jfchevrette

jfchevrette commented on Sep 1, 2015

@jfchevrette
Author

I'm pretty confident that @duglin in right and it could be very risky to change that behavior. many dockerfiles may break or simply copy inuntended stuff.

However I'd argue that for the long run it would be better if COPY was following the way tools such as cp or rsync handle globs & trailing slashes on folders. It's definitely not expected for COPY to copy files from a subfolder matching dir/* into the destination IMO

duglin

duglin commented on Sep 1, 2015

@duglin
Contributor

@jfchevrette yep - first chance we get we should "fix" this.
Closing it for now...

tugberkugurlu

tugberkugurlu commented on Feb 27, 2016

@tugberkugurlu

@duglin so, closing means it will not get fixed?

duglin

duglin commented on Feb 27, 2016

@duglin
Contributor

@tugberkugurlu yup, at least for now. There's work underway to redo the entire build infrastructure and when we do that is when we can make COPY (or its new equivalent) act the way it should.

tugberkugurlu

tugberkugurlu commented on Feb 27, 2016

@tugberkugurlu

@duglin thanks. Is it possible to keep this issue open and update the status here? Or is there any other issue for this that I can subscribe to?

duglin

duglin commented on Feb 27, 2016

@duglin
Contributor

@tugberkugurlu I thought we had an issue for "client-side builder support" but I can't seem to find it. So all we may have is what the ROADMAP ( https://github.com/docker/docker/blob/master/ROADMAP.md#22-dockerfile-syntax ) says.

As for keeping the issue open, I don't think we can do that. The general rule that Docker has been following is to close any issue that isn't actionable right away. Issues for future work are typically closed and then reopened once the state of things change such that some action (PR) can be taken for the issue.

deric

deric commented on Nov 5, 2016

@deric

@duglin This is very serious issue, you shouldn't just close it because the problem was introduced in 0.1 release. It would be more appropriate to target this for 2.0 release (milestones are on github too).

I guess most people use:

COPY . /app

and blacklist all other folders in .gitignore or have single level directory structure and use COPY which actually has mv semantics:

COPY src /myapp

It's quite hard for me to imagine that someone would actually use COPY for flattening directory structure. The other workaround for this is using tar -cf .. & ADD tarfile.tar.gz. Changing at least this would be really helpful. The other thing is respecting slashes in directory names COPY src /src vs COPY src/ /src (which are currently completely ignored).

tjwebb

tjwebb commented on Jan 14, 2017

@tjwebb

duglin closed this on Sep 1, 2015

@duglin This is a ridiculous and infuriating issue and should not be closed. The COPY command behaves specifically in disagreement with the documented usage and examples.

thaJeztah

thaJeztah commented on Jan 14, 2017

@thaJeztah
Member

@tjwebb there's still an open issue #29211. This can only be looked into if there's a way to fix this that's fully backward compatible. We're open to suggestions if you have a proposal how this could be implemented (if you do, feel free to write this up, and open a proposal, linking to this issue). Note that there's already a difference between (for example), OS X, and Linux in the way cp is handled;

mkdir -p repro-15858 \
  && cd repro-15858 \
  && mkdir -p source/dir1 source/dir2 \
  && touch source/file1 source/dir1/dir1-file1 \
  && mkdir -p target1 target2 target3 target4 target5 target6

cp -r source target1 \
&& cp -r source/ target2 \
&& cp -r source/ target3/ \
&& cp -r source/* target4/ \
&& cp -r source/dir* target5/ \
&& cp -r source/dir*/ target6/ \
&& tree

OS X:

.
├── source
│   ├── dir1
│   │   └── dir1-file1
│   ├── dir2
│   └── file1
├── target1
│   └── source
│       ├── dir1
│       │   └── dir1-file1
│       ├── dir2
│       └── file1
├── target2
│   ├── dir1
│   │   └── dir1-file1
│   ├── dir2
│   └── file1
├── target3
│   ├── dir1
│   │   └── dir1-file1
│   ├── dir2
│   └── file1
├── target4
│   ├── dir1
│   │   └── dir1-file1
│   ├── dir2
│   └── file1
├── target5
│   ├── dir1
│   │   └── dir1-file1
│   └── dir2
└── target6
    └── dir1-file1

20 directories, 12 files

On Ubuntu (/bin/sh)

.
|-- source
|   |-- dir1
|   |   `-- dir1-file1
|   |-- dir2
|   `-- file1
|-- target1
|   `-- source
|       |-- dir1
|       |   `-- dir1-file1
|       |-- dir2
|       `-- file1
|-- target2
|   `-- source
|       |-- dir1
|       |   `-- dir1-file1
|       |-- dir2
|       `-- file1
|-- target3
|   `-- source
|       |-- dir1
|       |   `-- dir1-file1
|       |-- dir2
|       `-- file1
|-- target4
|   |-- dir1
|   |   `-- dir1-file1
|   |-- dir2
|   `-- file1
|-- target5
|   |-- dir1
|   |   `-- dir1-file1
|   `-- dir2
`-- target6
    |-- dir1
    |   `-- dir1-file1
    `-- dir2

24 directories, 12 files
diff --git a/macos.txt b/ubuntu.txt
index 188d2c3..d776f19 100644
--- a/macos.txt
+++ b/ubuntu.txt
@@ -11,15 +11,17 @@
 │       ├── dir2
 │       └── file1
 ├── target2
-│   ├── dir1
-│   │   └── dir1-file1
-│   ├── dir2
-│   └── file1
+│   └── source
+│       ├── dir1
+│       │   └── dir1-file1
+│       ├── dir2
+│       └── file1
 ├── target3
-│   ├── dir1
-│   │   └── dir1-file1
-│   ├── dir2
-│   └── file1
+│   └── source
+│       ├── dir1
+│       │   └── dir1-file1
+│       ├── dir2
+│       └── file1
 ├── target4
 │   ├── dir1
 │   │   └── dir1-file1
@@ -30,6 +32,8 @@
 │   │   └── dir1-file1
 │   └── dir2
 └── target6
-    └── dir1-file1
+    ├── dir1
+    │   └── dir1-file1
+    └── dir2
 
-20 directories, 12 files
+24 directories, 12 files

86 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @jfchevrette@fnagel@deric@benmccallum@jrabbit

        Issue actions

          Dockerfile COPY with file globs will copy files from subdirectories to the destination directory · Issue #15858 · moby/moby