Skip to content

the ignore_older of tail config is of no effect #716

New issue
New issue
Closed
Closed
the ignore_older of tail config is of no effect#716
Assignees
edsiper
Labels
fixedquestion
@yubobo

Description

@yubobo
yubobo
opened on Aug 18, 2018

image
but，in fact，add all files to scan the queue whenever the file is generated.

Activity

yubobo

yubobo commented on Aug 18, 2018

@yubobo
yubobo
on Aug 18, 2018
Author

image

yubobo

yubobo commented on Aug 18, 2018

@yubobo
yubobo
on Aug 18, 2018
Author

image

yubobo

yubobo commented on Aug 18, 2018

@yubobo
yubobo
on Aug 18, 2018
Author

I change the value of ignore_older to 1m ,the problem is the same.

edsiper

edsiper commented on Aug 20, 2018

@edsiper
edsiper
on Aug 20, 2018 · edited by edsiper
Member

The ignore_older option is only available when a Parser is specificied and that parser matched a line, so when the parser lookup the time of that specific line/record it checks if is older than ignore_older or not. It does not apply to the file age, it's record based.

edsiper
self-assigned this
on Aug 20, 2018
edsiper
added
question
fixed
on Aug 20, 2018
edsiper
closed this as completedon Aug 21, 2018
kilya11
added a commit that references this issue on Dec 10, 2019

Ignore_Older is record based

28a8097
kilya11
mentioned this on Dec 10, 2019
kilya11
added a commit that references this issue on Dec 10, 2019

Ignore_Older is record based

5f38808
edsiper
added 2 commits that reference this issue on Dec 12, 2019

input: tail: Ignore_Older is record based (#244)

70339be

input: tail: Ignore_Older is record based (#244)

19138e5
acarsercan

acarsercan commented on Dec 27, 2019

@acarsercan
acarsercan
on Dec 27, 2019

Someone be kind enough to paste an example config with ignore_older & parser

Thanks

xPB12
mentioned this on Mar 9, 2020
avdhoot

avdhoot commented on Jul 19, 2020

@avdhoot
avdhoot
on Jul 19, 2020

@acarsercan Might late. Set parser=myparser in input plug-in tail. tail plugin will use Time_Key & Time_Format find out correct time to compare.

[PARSER]                                                                                                                                                                
    Name         myparser                                                                                                                                                 
    Format       json                                                                                                                                                   
    Time_Key     time                                                                                                                                                   
    Time_Format  %Y-%m-%dT%H:%M:%S.%L                                                                                                                                   
    Time_Keep    On
[INPUT]
    Name             tail
    Path             /var/log/containers/*.log
    Tag              containers.*
    Refresh_Interval 1
    Rotate_Wait      60
    Mem_Buf_Limit    5MB
    Skip_Long_Lines  On
    DB               /tail-db/tail-containers-state-sumo.db
    DB.Sync          Normal
    Ignore_Older 1m
    Parser  myparser
avdhoot
mentioned this on Jul 19, 2020
vsinghal13
mentioned this in 2 issues on Aug 12, 2020
rawahars
added a commit that references this issue on Oct 24, 2022

output: docs for default workers setting (fluent#716)

a22cdf5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

fixedquestion

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Development

    No branches or pull requests

      Participants

      @edsiper@avdhoot@yubobo@acarsercan

      Issue actions
        the ignore_older of tail config is of no effect · Issue #716 · fluent/fluent-bit