Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Ingest Pipeline][Set Processor] Add argument for null value check #54783

Closed
P1llus opened this issue Apr 6, 2020 · 2 comments
Closed

[Ingest Pipeline][Set Processor] Add argument for null value check #54783

P1llus opened this issue Apr 6, 2020 · 2 comments
Labels
:Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP Team:Data Management Meta label for data/management team

Comments

@P1llus
Copy link
Member

P1llus commented Apr 6, 2020

Describe the feature:
With an increasing focus on Observability and Security, most of the processing of data is being moved to Ingest pipelines through beats.
Since ingestion rates are always a top priority on these cases, there is one argument in the set processor that always has to be set currently with a painless condition that would be great having as a boolean argument in Elasticsearch instead, and that is checking for Null values in the "value" field.

In most other processors we are have "ignore_missing" argument, in which it will ignore fields that are either missing or is null, but currently if we are trying to use the set processor on a value that is null, it will add it as a empty string, this is something we can currently prevent with something like this:

    {
      "set" : {
        "field": "source.packets",
        "value": "{{ctx.checkpoint?.client_outbound_packets}}",
        "if": "ctx.checkpoint?.client_outbound_packets != null"
      }
    },

Due to logs more often than not is coming in with many different structures, it is impossible to utilise the set processor without either painless or having issues with plenty of fields that are empty strings (instead of null).

Would it be possible to add a "ignore_missing" or similar argument to this field as well? So that it is possible to set a field if the value is not null?

@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features (:Core/Features/Features)

@danhermann
Copy link
Contributor

This feature was added in #57030.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP Team:Data Management Meta label for data/management team
Projects
None yet
Development

No branches or pull requests

4 participants