proposal: cmd/go: support embedding static assets (files) in binaries #35950
Description
There are many tools to embed static asset files into binaries:
- https://godoc.org/perkeep.org/pkg/fileembed / perkeep.org/pkg/fileembed/genfileembed
- https://godoc.org/github.com/gobuffalo/packr
- https://godoc.org/github.com/knadh/stuffbin
- https://github.com/rakyll/statik
- Bazel go_embed_data
Actually, https://tech.townsourced.com/post/embedding-static-files-in-go/ lists more:
- vfsgen - https://github.com/shurcooL/vfsgen
- go.rice - https://github.com/GeertJohan/go.rice
- statik - https://github.com/rakyll/statik
- esc - https://github.com/mjibson/esc
- go-embed - https://github.com/pyros2097/go-embed
- go-resources - https://github.com/omeid/go-resources
- statics - https://github.com/go-playground/statics
- templify - https://github.com/wlbr/templify
- gnoso/go-bindata - https://github.com/gnoso/go-bindata
- shuLhan/go-bindata - https://github.com/shuLhan/go-bindata
- fileb0x - https://github.com/UnnoTed/fileb0x
- gobundle - https://github.com/alecthomas/gobundle
- parcello - https://github.com/phogolabs/parcello
Proposal
I think it's time to do this well once & reduce duplication, adding official support for embedding file resources into the cmd/go tool.
Problems with the current situation:
- There are too many tools
- Using a go:generate-based solution bloats the git history with a second (and slightly larger) copy of each file.
- Not using go:generate means not being
go install-able or making people write their own Makefiles, etc.
Goals:
- don't check in generated files
- don't generate *.go files at all (at least not in user's workspace)
- make
go install/go builddo the embedding automatically - let user choose per file/glob which type of access is needed (e.g. []byte,
func() io.Reader,io.ReaderAt, etc) - Maybe store assets compressed in the binary where appropriate (e.g. if user only needs an
io.Reader)? (edit: but probably not; see comments below) - No code execution at compilation time; that is a long-standing Go policy.
go buildorgo installcan not run arbitrary code, just likego:generatedoesn't run automatically at install time.
The two main implementation approaches are //go:embed Logo logo.jpg or a well-known package (var Logo = embed.File("logo.jpg")).
go:embed approach
For a go:embed approach, one might say that any go/build-selected *.go file can contain something like:
//go:embed Logo logo.jpg
Which, say, compiles to:
func Logo() *io.SectionReader(adding a dependency to the io package)
Or:
//go:embedglob Assets assets/*.css assets/*.js
compiling to, say:
var Assets interface{
Files() []string
Open func(name string) *io.SectionReader
} = runtime.EmbedAsset(123)
Obviously this isn't fully fleshed out. There'd need to be something for compressed files too that yield only an io.Reader.
embed package approach
The other high-level approach is to not have a magic //go:embed syntax and instead just let users write Go code in some new "embed" or "golang.org/x/foo/embed" package:
var Static = embed.Dir("static")
var Logo = embed.File("images/logo.jpg")
var Words = embed.CompressedReader("dict/words")
Then have cmd/go recognize the calls to embed.Foo("foo/*.js") etc and glob do the work in cmd/go, rather than at runtime. Or maybe certain build tags or flags could make it fall back to doing things at runtime instead. Perkeep (linked above) has such a mode, which is nice to speed up incremental development where you don't care about linking one big binary.
Concerns
- Pick a style (//go:embed* vs a magic package).
- Block certain files?
- Probably block embedding
../../../../../../../../../../etc/shadow - Maybe block reaching into
.gittoo
- Probably block embedding
Metadata
Metadata
Assignees
Type
Projects
Milestone
Relationships
Development
Participants
Activity
ianlancetaylor commentedon Dec 4, 2019
It's worth considering whether
embedglobshould support a complete file tree, perhaps using the**syntax supported by some Unix shells.ghost commentedon Dec 4, 2019
Some people would need the ability to serve the embedded assets with HTTP using the
http.FileServer.I personally use either mjibson/esc (which does that) or in some cases my own file embedding implementation which renames files to create unique paths and adds a map from the original paths to the new ones, e.g.
"/js/bootstrap.min.js": "/js/bootstrap.min.827ccb0eea8a706c4c34a16891f84e7b.js". Then you can use this map in the templates like this:href="{{ static_path "/css/bootstrap.min.css" }}".cespare commentedon Dec 4, 2019
I think a consequence of this would be that it would be nontrivial to figure out what files are necessary to build a program.
The
//go:embedapproach introduces another level of complexity too. You'd have to parse the magic comments in order to even typecheck the code. The "embed package" approach seems friendlier to static analysis.(Just musing out loud here.)
bradfitz commentedon Dec 4, 2019
@opennota,
Yes, the first link above is a package I wrote (in 2011, before Go 1) and still use, and it supports using http.FileServer: https://godoc.org/perkeep.org/pkg/fileembed#Files.Open
bradfitz commentedon Dec 4, 2019
@cespare,
Yes, good point. That's a very strong argument for using a package. It also makes it more readable & documentable, since we can document it all with regular godoc, rather than deep in cmd/go's docs.
agnivade commentedon Dec 4, 2019
@bradfitz - Do you want to close this #3035 ?
bradfitz commentedon Dec 4, 2019
@agnivade, thanks for finding that! I thought I remembered that but couldn't find it. Let's leave it open for now and see what others think.
balasanjay commentedon Dec 4, 2019
If we go with the magic package, we could use the unexported type trick to ensure that callers pass compile-time constants as arguments: https://play.golang.org/p/RtHlKjhXcda.
(This is the strategy referenced here: https://groups.google.com/forum/#!topic/golang-nuts/RDA9Hag8RZw/discussion)
AlexRouSg commentedon Dec 4, 2019
One concern I have is how would it hanle invividual or all assets being too big to fit into memory and whether there would be maybe a build tag or per file access option to choose between pritorizing access time vs memory footprint or some middle ground implementation.
urandom commentedon Dec 4, 2019
the way i've solved that problem (because of course i also have my own implementation :) ) is to provide an http.FileSystem implementation that serves all embedded assets. That way, you don't to rely on magic comments in order to appease the typechecker, the assets can easily be served by http, a fallback implementation can be provided for development purposes (http.Dir) without changing the code, and the final implementation is quite versatile, as http.FileSystem covers quite a bit, not only in reading files, but listing directories as well.
One can still use magic comments or whatever to specify what needs to be embedded, though its probably easier to specify all the globs via a plain text file.
ianlancetaylor commentedon Dec 4, 2019
@AlexRouSg This proposal would only be for files which are appropriate to include directly in the final executable. It would not be appropriate to use this for files that are too big to fit in memory. There's no reason to complicate this tool to handle that case; for that case, just don't use this tool.
bradfitz commentedon Dec 4, 2019
@ianlancetaylor, I think the distinction @AlexRouSg was making was between having the files provided as global
[]bytes (unpageable, potentially writable memory) vs providing a read-only, on-demand view of an ELF section that can normally live on disk (in the executable), like via anOpencall that returns an*io.SectionReader. (I don't want to bake inhttp.Fileorhttp.FileSysteminto cmd/go or runtime... net/http can provide an adapter.)urandom commentedon Dec 4, 2019
@bradfitz both http.File itself is an interface with no technical dependencies to the
httppackage. It might be a good idea for anyOpenmethod to provide an implementation that conforms to that interface, because both theStatandReaddirmethods are quite useful for such assets188 remaining items
rsc commentedon Sep 2, 2020
I wasn't watching the comments on this issue.
@atomsymbol welcome back! It's great to see you commenting here again.
I agree in principle that if we had sandboxing many things would be easier.
On the other hand many things might be harder - builds might never finish.
In any event, we definitely don't have that kind of sandboxing today. :-)
@kokes I am not sure about the details,
but we'll make sure serving an embed.Files over HTTP gets ETags right by default.
rsc commentedon Sep 2, 2020
I have filed #41191 for accepting the design draft posted back in July.
I am going to close this issue as superseded by that one.
Thanks for the great preliminary discussion here.