Skip to content

Feature request: option to raise the KV size limitation #3358

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
greg-szabo opened this issue Aug 3, 2017 · 9 comments
Closed

Feature request: option to raise the KV size limitation #3358

greg-szabo opened this issue Aug 3, 2017 · 9 comments

Comments

@greg-szabo
Copy link

The last question on the FAQ (https://www.consul.io/docs/faq.html#q-what-is-the-per-key-value-size-limitation-for-consul-39-s-key-value-store-) states that the consul KV size limitation is 512kB. This is easy to reach even in official use-cases:

I use terraform to create AWS EC2 instances. When I want to create 114 t2.large instances with new SSH key added and DNS name registered, I get the following error:

Failed to save state: Failed request: Value for key "terraform/perf/terraform.tfstate-env:vnet1" is too large (552661 > 524288 bytes)

Consul version 0.9.0
Terraform version 0.9.11

The community gitter couldn't really tell me if it's possible to raise the KV size limitation and the documentation doesn't mention an option. I'm assuming it's hard-wired.

consul info

agent:
	check_monitors = 0
	check_ttls = 0
	checks = 0
	services = 0
build:
	prerelease =
	revision = b79d951
	version = 0.9.0
consul:
	bootstrap = true
	known_datacenters = 1
	leader = true
	leader_addr = 127.0.0.1:8300
	server = true
raft:
	applied_index = 142135
	commit_index = 142135
	fsm_pending = 0
	last_contact = 0
	last_log_index = 142135
	last_log_term = 2
	last_snapshot_index = 137082
	last_snapshot_term = 2
	latest_configuration = [{Suffrage:Voter ID:127.0.0.1:8300 Address:127.0.0.1:8300}]
	latest_configuration_index = 1
	num_peers = 0
	protocol_version = 2
	protocol_version_max = 3
	protocol_version_min = 0
	snapshot_version_max = 1
	snapshot_version_min = 0
	state = Leader
	term = 2
runtime:
	arch = amd64
	cpu_count = 2
	goroutines = 68
	max_procs = 2
	os = linux
	version = go1.8.3
serf_lan:
	coordinate_resets = 0
	encrypted = false
	event_queue = 1
	event_time = 2
	failed = 0
	health_score = 0
	intent_queue = 0
	left = 0
	member_time = 1
	members = 1
	query_queue = 0
	query_time = 1
serf_wan:
	coordinate_resets = 0
	encrypted = false
	event_queue = 0
	event_time = 1
	failed = 0
	health_score = 0
	intent_queue = 0
	left = 0
	member_time = 1
	members = 1
	query_queue = 0
	query_time = 1

Operating system: CentOS 7

Error:

Failed to save state: Failed request: Value for key "terraform/perf/terraform.tfstate-env:vnet1" is too large (552661 > 524288 bytes)

Failed to persist state to backend.

The error shown above has prevented Terraform from writing the updated state
to the configured backend. To allow for recovery, the state has been written
to the file "errored.tfstate" in the current working directory.

Running "terraform apply" again at this point will create a forked state,
making it harder to recover.

To retry writing this state, use the following command:
    terraform state push errored.tfstate

When I edited errored.tfstate and reduced it to less than 512kB, the state push worked. (Although I had to manage some resources manually after that.)
@slackpad
Copy link
Contributor

slackpad commented Aug 3, 2017

Hi @greg-szabo this limit is currently hard coded. We've generally tried to avoid making this configurable to discourage abusive use cases of Consul holding huge amounts of data, and there are some practical limits to how large it can go and still have the servers meet the Raft timing requirements if they are shuttling large chunks of data around.

Terraform could probably chunk its updates to work around this, though I think there's a way to configure it to gzip the state which is probably a practical workaround for most deployments since the data is so compressible - hashicorp/terraform#8748. Will that work for your use case?

@greg-szabo
Copy link
Author

Hi @slackpad,
Thanks for getting back to me and sorry for the silence. (Vacation and whatnot.)

GZip would be an interesting workaround, I'm sure you could get a few times bigger configuration out of it. Thanks for suggesting it, I'll check it out next time I run across this problem.

I had to come up with some quick workarounds and eventually (since locking was a requirement) I implemented the Amazon S3 + DynamoDB solution to store my Terraform configuration. It works fine (especially since a few bugfixes in Terraform were implemented) so I got rid of the consul infrastructure for now.

To be honest, an arbitrary hardcoded value is kind of a red flag for me in any case. I understand that it's a quick way to safeguard any implementation but in any DevOps solution, it can put the wrench in the gears without any apparent reason. Since Terraform's output is fed directly to Consul, I don't even have a way to safeguard against it. The way it's dependent on the size of the file and not the content makes it vulnerable in an unexpected way. If Terraform fails I have to find out if it failed because the network was not consistent, I ran out of credits on the cloud or an arbitrary value within the tool was reached.

I still think that both Terraform and Consul are amazing tools but in my specific use-case, I have to apply workarounds or other solutions. I hope that in the future Consul can get rid of this limitation or maybe Terraform can zip its configuration by default when used in conjunction with Consul.

@slackpad
Copy link
Contributor

@greg-szabo thanks for the follow up explanation - we will keep this in mind!

@OlivierCuyp
Copy link

@slackpad I'm using Consul as a config storage for traefik. Traefik store all Let's Encrypt certificates under the same key. So with the limit 512Kb I can only store about 100 certificates.
I would feel way more confortable with a limit of 1000 certificates so a limit of 5M.

Here is the traefik details: https://docs.traefik.io/configuration/acme/#__code_14

@camilb
Copy link

camilb commented Sep 28, 2018

@OlivierCuyp I had to fork Consul and build it myself in order to increase this limit to 100MB (just to be safe) and it works good for now. Currently storing ~1400 certificates and didn't see any performance issue in Consul. Still not sure how traefik can handle this amount of certificates

@OlivierCuyp
Copy link

@camilb thanks for your feedback but I rather have to change a config than rebuild Consul.
I was hoping to get some reaction from the Consul team on this.

@camilb
Copy link

camilb commented Sep 29, 2018

@OlivierCuyp Just a quick note: even if Consul will support this feature, Traefik becomes quite unstable at ~400 certificates. It consumes a lot of CPU and sometimes it just hangs and you have to restart it to be able to request new certificates. Consul only shows a warning [WARN] consul: Attempting to apply large raft entry (4659230 bytes) but works pretty good. Traefik has this problem with other backends too.

@Shazzberry Shazzberry mentioned this issue Apr 22, 2019
Closed
@mantzas
Copy link

mantzas commented May 20, 2019

We actually have a configuration that is bigger as 512kb. Any way to get the option implemented?

@hprasad068
Copy link

Even the gzip on terraform doesn't suffice as some time we do reach the limit with zip as well. I currently do not have s3 and that sort of pushes me away to look for other option. Any chance of a config change to customize the size in the future or have we decided to keep it this way..?

@aneroid13 aneroid13 mentioned this issue May 27, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@mantzas @slackpad @OlivierCuyp @camilb @greg-szabo @hprasad068