Configure log4j.properties for Kafka appender, error when parsing property bootstrap.servers

Question

I want to add a Kafka appender to the audit-hdfs log in a Cloudera cluster.

I have successfully configured a log4j2.xml file with a Kafka appender, I need to convert this log4j2.xml into a log4j2.properties file in order to be able to merge it with the HDFS log log4j2.properties file. I am unable to do this because when I launch my dummy process with log4j2.properties instead of XML, I get an error.

I have tried writing the properties file in several different ways, always resulting in problems with the bootstrap.servers property

This is my properties file

filters = threshold

filter.threshold.type = ThresholdFilter
filter.threshold.level = ALL

appenders = console,kafka

appender.console.type = Console
appender.console.name = console
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %m%n

appender.kafka.type = Kafka
appender.kafka.name = kafka
appender.kafka.layout.type = PatternLayout
appender.kafka.layout.pattern =%m%n
appender.kafka.property.type = Property
appender.kafka.property.bootstrap.servers = ip:port
appender.kafka.topic = cdh-audit-hdfs

Here the problem is in this line: appender.kafka.property.bootstrap.servers = ip:port

I have tried the following, to no avail:

appender.kafka.property.bootstrap.servers = ip:port

appender.kafka.property.bootstrap\.servers = ip:port

appender.kafka.property.name = "bootstrap.servers"
appender.kafka.property.bootstrap.servers = ip:port

appender.kafka.property.key = "bootstrap.servers"
appender.kafka.property.value = ip:port

etc...

This is my dummy process:

package blabla

import org.apache.logging.log4j.LogManager

object dummy extends App{
  val logger = LogManager.getLogger
  val record = "...c"
  while(true){
    logger.info(record)
    Thread.sleep(5000)
  }
}

How do I need to configure my log4j2.properties in order to be able to define this property?

Im expecting this process to write the record in my Kafka topic but instead I get errors like this:

Exception in thread "main" org.apache.logging.log4j.core.config.ConfigurationException: No type attribute provided for component bootstrap

Exception in thread "main" org.apache.kafka.common.config.ConfigException: Missing required configuration "bootstrap.servers" which has no default value.

Answer 1

try this solution, it works for me :

appender.kafka.property.type=Property
appender.kafka.property.name=bootstrap.servers
appender.kafka.property.value=host:port

Answer 2

appender.kafka.property1.type=property
appender.kafka.property1.name=bootstrap.servers
appender.kafka.property1.value=ip:port
appender.kafka.property2.type=property
appender.kafka.property2.name=security.protocol
appender.kafka.property2.value=SASL_PLAINTEXT
appender.kafka.property3.type=property
appender.kafka.property3.name=sasl.mechanism
appender.kafka.property3.value=PLAIN
appender.kafka.property4.type=property
appender.kafka.property4.name=sasl.jaas.configappender.kafka.property4.value=`org.apache.kafka.common.security.plain.PlainLoginModule required username=\"xxxx\" password=\"xxxx\";`

Answer 3

From my personal experience of reading through different stuff, I found that the log4j2.properties file is very flexible.

The keys that we use are very arbitrary, and things like the "type" dictate what that key is treated as.

appender.kafka.property.type = Property
appender.kafka.property.bootstrap.servers = ip:port

This signifies bootstrap.servers is a known key, which it isn't. Let me give an example of mapping a xml config to a properties config.

XML:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
   <Appenders>
       <Console name="stdout" target="SYSTEM_OUT">
           <PatternLayout pattern="%d - %m%n"/>
       </Console>

       <Kafka name="kafkastream" topic="my-topic-1">
           <PatternLayout pattern="%d %m"/>
           <Property name="bootstrap.servers">localhost:9092</Property>
       </Kafka>

       <Async name="Async">
           <AppenderRef ref="Kafka"/>
       </Async>
   </Appenders>
   
   <Loggers>
       <Root level="INFO">
           <AppenderRef ref="stdout"/>
           <AppenderRef ref="kafkastream"/>
       </Root>
   </Loggers>
</Configuration>

Each tag inside the tag needs to be given an arbitrary name. For this tag:

<Console name="stdout" target="SYSTEM_OUT">
    <PatternLayout pattern="%d - %m%n"/>
</Console>

We should write the following in properties file:

appenders.someArbitraryKey.type=Console
appenders.someArbitraryKey.name=stdout
appenders.someArbitraryKey.target=SYSTEM_OUT
appenders.someArbitraryKey.anotherCustomKey.type=PatternLayout
appenders.someArbitraryKey.anotherCustomKeytype.pattern=%d - %m%n

We need arbitrary keys to uniquely identify things. We are using someArbitraryKey to signify the <Console> tag and anotherCustomKey to signify its nested <PatternLayout> tag. We must mention:

• type -> The kind of XML tag, which is here Console
• any other properties of the tag itself

For a more complete reference, the complete XML should translate to the following properties file:

rootLogger = INFO, stdout, kafkastream

appender.terminal.type = Console
appender.terminal.name = stdout
appender.terminal.target = SYSTEM_OUT
appender.terminal.layout.type = PatternLayout
appender.terminal.layout.pattern = %d - %m%n

appender.kafka.type = Kafka
appender.kafka.name = kafkastream
appender.kafka.topic = my-topic-1
appender.kafka.layout.type = PatternLayout
appender.kafka.layout.pattern = %d %m
appender.kafka.bootstrap.type = Property
appender.kafka.bootstrap.name = bootstrap.servers
appender.kafka.bootstrap.value = localhost:29092

appender.async.type = Async
appender.async.name = Async
appender.async.appref.type = AppenderRef
appender.async.appref.ref = Kafka

Hope this helps.