@xiang90, Maintainers: Any opinion/updates about this issue?

etcd/raft does snapshot internally to reclaim disk from WAL files. We do not support incremental snapshot from external. But that is something doable and wont be super hard. Do you have any interest working on it?

Yeah sure. I'll take it forward.

I think this is important for sane backup/restore strategy +1

@xiang90, Could you please point me to documentation on snapshots or explain in detail about how snapshot functionality is used in etcd. Some questions: 1) is there snapshot at raft log (wal) level and another snapshot at bbolt db level. after adding 900k keys to a 3 node cluster, i have restarted all three nodes. two of the nodes after restart were busy with exchanging snapshot as shown in the log below. Is one node trying to send entire db to another node or is it trying to send snapshot of log from last committed index 900168? Thank you
sender node:
2018-08-23 15:34:24.489264 I | etcdserver: wrote database snapshot out [total bytes: 9223372036854775807]
2018-08-23 15:34:24.489504 I | rafthttp: start to send database snapshot [index: 900167, to f078dbaf5ed2490b]...
2018-08-23 15:34:24.491349 W | rafthttp: database snapshot [index: 900167, to: f078dbaf5ed2490b] failed to be sent out (ioutil: short read)
...
2018-08-23 15:34:25.090355 I | etcdserver: wrote database snapshot out [total bytes: 9223372036854775807]
2018-08-23 15:34:25.090482 I | rafthttp: start to send database snapshot [index: 900168, to f078dbaf5ed2490b]...
2018-08-23 15:34:25.092223 W | rafthttp: database snapshot [index: 900168, to: f078dbaf5ed2490b] failed to be sent out (ioutil: short read)
receiver node:
2018-08-23 15:34:24.392607 I | rafthttp: receiving database snapshot [index:900167, from ec5c70c3e0fa567c] ...
2018-08-23 15:34:24.392890 E | rafthttp: failed to save KV snapshot (unexpected EOF)
2018-08-23 15:34:24.489257 E | etcdserver: publish error: etcdserver: request timed out
2018-08-23 15:34:24.690707 I | rafthttp: receiving database snapshot [index:900168, from ec5c70c3e0fa567c] ...
2018-08-23 15:34:24.691197 E | rafthttp: failed to save KV snapshot (unexpected EOF)
.....

cc @jingyih backup/restore
cc @jpbetz nonvoting members

Part of the non-voting member design describes non-disruptive snapshot transmission (#9161). Once that's in place, we could just enable it for etcdctl snapshot save?

etcdctl snapshot save

etcdctl snapshot save --forever could potentially starts a non-voting member and keep on receiving data and making snapshots.

etcdctl snapshot save

etcdctl snapshot save --forever could potentially starts a non-voting member and keep on receiving data and making snapshots.

find no --forever flag in etcdctl snapshot save command

[root@e5d68e04-f6f6-11e9-b1f2-00163e069dac ~]# etcdctl version
etcdctl version: 3.3.11
API version: 3.3
[root@e5d68e04-f6f6-11e9-b1f2-00163e069dac ~]# 
[root@e5d68e04-f6f6-11e9-b1f2-00163e069dac ~]# etcdctl snapshot save -h
NAME:
        snapshot save - Stores an etcd node backend snapshot to a given file

USAGE:
        etcdctl snapshot save <filename>

GLOBAL OPTIONS:
      --cacert=""                               verify certificates of TLS-enabled secure servers using this CA bundle
      --cert=""                                 identify secure client using this TLS certificate file
      --command-timeout=5s                      timeout for short running command (excluding dial timeout)
      --debug[=false]                           enable client-side debug logging
      --dial-timeout=2s                         dial timeout for client connections
  -d, --discovery-srv=""                        domain name to query for SRV records describing cluster endpoints
      --endpoints=[127.0.0.1:2379]              gRPC endpoints
      --hex[=false]                             print byte strings as hex encoded strings
      --insecure-discovery[=true]               accept insecure SRV records describing cluster endpoints
      --insecure-skip-tls-verify[=false]        skip server certificate verification
      --insecure-transport[=true]               disable transport security for client connections
      --keepalive-time=2s                       keepalive time for client connections
      --keepalive-timeout=6s                    keepalive timeout for client connections
      --key=""                                  identify secure client using this TLS key file
      --user=""                                 username[:password] for authentication (prompt if password is not supplied)
  -w, --write-out="simple"                      set the output format (fields, json, protobuf, simple, table)

[root@e5d68e04-f6f6-11e9-b1f2-00163e069dac ~]# 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.