Skip to content

Too many translog files open, and reach ulimit setting(65535) #49970

Closed
@seeIT-52

Description

@seeIT-52

Describe the feature:
Hi,
I found 'too many files open' problem in my ES enviroment. And I use the commad ls -al /proc/ES_PID/fd , found that ES opened particularly large number of translog files.
After some search on this problem , I tried to restart a runing normally ES (6.6.1). I found that ,every time ES is restarted, there will be one more translog file on some shards. And all the translogs have no operation data, like below( The server time is incorrect,but I don't think it's concern ):

image

Elasticsearch version (bin/elasticsearch --version):
Version: 6.6.1, Build: default/tar/1fd8f69/2019-02-13T17:10:04.160291Z, JVM: 1.8.0_60
Plugins installed: []
No Plugins
JVM version (java -version):
java version "1.8.0_60"
Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
OS version (uname -a if on a Unix-like system):
Centos 7.6
Linux 3.10.0-957.5.1.el7.x86_64 #1 SMP Fri Feb 1 14:54:57 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Description of the problem including expected versus actual behavior:
Empty translog files(no operation data) keep increasing.
Old translog exist beyond 12 hours(index.translog.retention.age = 12h).
I think empty translog files should be deleted or don't creat new translog files.
Steps to reproduce:
restart ES node
Please include a minimal but complete recreation of the problem, including
(e.g.) index creation, mappings, settings, query etc. The easier you make for
us to reproduce it, the more likely that somebody will take the time to look at it.

  1. defautl index setting as follow:
    "settings":{ "index.number_of_replicas":"0", "index.refresh_interval":"10s", "index.number_of_shards":"1", "index.routing.allocation.require.tag":"hot", "index.indexing.slowlog.level": "info", "index.indexing.slowlog.threshold.index.trace": "500ms", "index.indexing.slowlog.threshold.index.debug": "2s", "index.indexing.slowlog.threshold.index.info": "5s", "index.indexing.slowlog.threshold.index.warn": "10s", "index.search.slowlog.level": "info", "index.search.slowlog.threshold.fetch.trace": "200ms", "index.search.slowlog.threshold.fetch.debug": "500ms", "index.search.slowlog.threshold.fetch.info": "800ms", "index.search.slowlog.threshold.fetch.warn": "1s", "index.search.slowlog.threshold.query.trace": "500ms", "index.search.slowlog.threshold.query.debug": "2s", "index.search.slowlog.threshold.query.info": "5s", "index.search.slowlog.threshold.query.warn": "10s", "index.translog.durability": "async", "index.translog.flush_threshold_size": "5000mb", "index.translog.sync_interval": "120s", "index.mapping.ignore_malformed": true }
  2. Auto create index use the dynamic mapping, and index name is created by date, only the index created today is active. Indices created before today have no data to index.
  3. Restart ES, the translog of indices created before today (no data to index) keep increasing

If I missed some configuration, please let me know.
I'd be happy to provide additional details, whatever is needed.

Thanks!

Activity

elasticmachine

elasticmachine commented on Dec 9, 2019

@elasticmachine
Collaborator

Pinging @elastic/es-distributed (:Distributed/Engine)

DaveCTurner

DaveCTurner commented on Dec 9, 2019

@DaveCTurner
Contributor

I figured this would have been addressed by #47414, and maybe also no longer an issue in 7.x thanks to #45473, but in fact that's not true. After restarting a one-node 7.5.0 cluster repeatedly it had over 100 translog generations, despite the limit imposed in #47414. We do a kind of partial flush in InternalEngine#recoverFromTranslogInternal, calling commitIndexWriter, but only if there were ops to recover and we don't trim the translog there either way. Maybe we should?

In the meantime a POST /index/_flush?force will, I think, clean things up.

seeIT-52

seeIT-52 commented on Dec 10, 2019

@seeIT-52
Author

Hi @DaveCTurner ,
Thanks a lot for your answer.
And is there any command like elasticsearch-traslog trim for translog merge? Because before merging all the translog ,my ES node recovey very slowly even can't start (too many open files.)

ywelsch

ywelsch commented on Dec 10, 2019

@ywelsch
Contributor

No, you'll have to increase the number of file descriptors before starting up the node next time.

dnhatn

dnhatn commented on Dec 12, 2019

@dnhatn
Member

We create a new translog generation whenever we open a new Translog instance. We need to do that to make sure each generation has at most one primary term. The actual problem is that the translog deletion policy uses the translog generation tag from the safe commit as the baseline. Hence, as David said, we won't be able to clean up translog unless we (force) flush. We need to handle with sync_id carefully if we force flush a recovering shard.

I am prototyping an option where the translog deletion policy uses the local checkpoint from the safe commit instead. It would allow us to clean up the extra translog without having a new commit. However, it's a quite substantial change as we've relied on the translog generation tag in many places.

added a commit that references this issue on Feb 10, 2020
ebc4681
added a commit that references this issue on Feb 26, 2020
a14d4bc
added a commit that references this issue on Feb 26, 2020
db6b9c2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

:Distributed Indexing/EngineAnything around managing Lucene and the Translog in an open shard.>bug

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Development

    Participants

    @ywelsch@DaveCTurner@dnhatn@elasticmachine@seeIT-52

    Issue actions

      Too many translog files open, and reach ulimit setting(65535) · Issue #49970 · elastic/elasticsearch