Skip to content

Elasticsearch rejects data #493

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
dmnfortytwo opened this issue Oct 25, 2018 · 10 comments
Closed

Elasticsearch rejects data #493

dmnfortytwo opened this issue Oct 25, 2018 · 10 comments

Comments

@dmnfortytwo
Copy link

I've got some logs which ES refuses to accept:

2018-10-24 16:22:21 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch" location=nil tag="syslog.user.warn" time=2018-10-24 16:22:13.000000000 +0000 record={"host"=>"webserver-aws-ie-2", "ident"=>"webserver", "message"=>"WARNING: write_log(): tried to log ctrl symbol at field 27 into 'requests.log', log message: https://some.domain.com(us/appYy\x9B\x827D6\xD2\xCE\xE9\xCC-Q\x9E?\xF0nia/id733948432?mt=9", "severity"=>"warn", "syslog_facility"=>"user", "fqdn"=>"webserver-aws-ie-2.banana.domain.tld", "hostname"=>"webserver-aws-ie-2.banana.domain.tld", "project"=>"banana", "cluster"=>"default", "original_tag"=>"syslog.user.warn", "forwarder_ip"=>"10.138.0.210", "forwarder_hostname"=>"haproxy-gce-be-3.c.logs.internal", "pushed_by"=>"harvester-gce-be-8.logs.domain.tld:2", "target_index"=>"banana--2018.10.24"}

Output config:

<match **>
  @type copy
  <store>
    @type elasticsearch
    host localhost
    port 9200
    target_type_key @target_key
    type_name fluentd
    target_index_key target_index
    logstash_format true
    logstash_prefix invalid--
    time_key @timestamp
    include_timestamp true
    reconnect_on_error true
    reload_on_failure true
    reload_connections false
    request_timeout 120s
    <buffer>
      @type file
      flush_interval 10s
      retry_type periodic
      retry_forever true
      retry_wait 10s
      chunk_limit_size 16Mb
      queue_limit_length 4096
      total_limit_size 60Gb
      path /var/lib/td-agent/buffers/output_elasticsearch-2
    </buffer>
  </store>
</match>

Fluentd components versions:

2018-10-25 12:07:24 +0000 [info]: starting fluentd-1.2.6 pid=4141 ruby="2.4.4"
2018-10-25 12:07:25 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '2.11.11'
2018-10-25 12:07:25 +0000 [info]: gem 'fluent-plugin-flowcounter' version '1.3.0'
2018-10-25 12:07:25 +0000 [info]: gem 'fluent-plugin-kafka' version '0.7.9'
2018-10-25 12:07:25 +0000 [info]: gem 'fluent-plugin-multiprocess' version '0.2.2'
2018-10-25 12:07:25 +0000 [info]: gem 'fluent-plugin-prometheus' version '1.0.1'
2018-10-25 12:07:25 +0000 [info]: gem 'fluent-plugin-record-modifier' version '1.1.0'
2018-10-25 12:07:25 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.1.0'
2018-10-25 12:07:25 +0000 [info]: gem 'fluent-plugin-s3' version '1.1.6'
2018-10-25 12:07:25 +0000 [info]: gem 'fluent-plugin-td' version '1.0.0'
2018-10-25 12:07:25 +0000 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.4'
2018-10-25 12:07:25 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.2.3'
2018-10-25 12:07:25 +0000 [info]: gem 'fluentd' version '1.2.6'

Ubuntu 18.04
ES 6.2.2
@cosmo0920
Copy link
Collaborator

cosmo0920 commented Oct 26, 2018 •
edited
Loading

Could you try to use @log_level debug?
It will be shown failure reason in Fluentd log.

@dmnfortytwo
Copy link
Author

Error disappeared after td-agent restart :(

@cosmo0920
Copy link
Collaborator

:(

@cosmo0920
Copy link
Collaborator

I'm closing this for now.
If you hit an another reject, please add a comment here.

@godefroi
Copy link

godefroi commented Jan 7, 2019

I had this issue when I had a leftover logstash template (from 5.x) in my ES 6.x cluster. Removing the logstash template solved the problem.

@cosmo0920
Copy link
Collaborator

Thank you for your comment.
Does this issue still occurr with logstash template 6.x?
It looks differrent from 5.x:
https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/master/lib/logstash/outputs/elasticsearch/elasticsearch-template-es6x.json

@godefroi
Copy link

godefroi commented Jan 8, 2019

I didn't try the 6.x template. The 5.x template fails in ES 6.x because it uses a feature that is no longer available in 6.x.

@trotyoung
Copy link

The first thing is to open the debug log, in your fluentdconfiguration file add :

<system>
  log_level debug
  # log_level error
</system>

and then watch your fluentd system log such as /var/log/td-agent/td-agent.log .
For me , I found that Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1001]/[1000] maximum shards open , so I add the cluster.max_shards_per_node and solved my issue.
Hope can help 🤷‍♂️.

@abhinav0927
Copy link

Where did you added cluster.max_shards_per_node in your fluentd config file or in elasticsearch cluster?

@trotyoung
Copy link

Where did you added cluster.max_shards_per_node in your fluentd config file or in elasticsearch cluster?

I have added cluster.max_shards_per_node: 2000 in the /etc/elasticsearch/elasticsearch.yml used by the elasticsearch service, and then restart elasticsearch service.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@cosmo0920 @godefroi @trotyoung @dmnfortytwo @abhinav0927