There are two options:

• extract the consumer queue with (queue_get_consumer()) and then use consume_batch_queue()
• make your own batch reader by wrapping consumer_poll() until you get the desired number of messages.

Actually I am using the node wrapper https://github.com/Blizzard/node-rdkafka, and so can't directly use these options.

Wouldn't it be useful to support this config so that all the wrappers get the functionality without modifying the code.

This would require a new API since the current consumer_poll() API returns a single message, so it wouldn't automatically trickle down into the bindings.

Correct me if I am wrong.. but you don't fetch single messages from kafka right? Becuase thats not possible to control without using max.poll.records. So it means you are fetching multiple messages, storing them, and only providing single messages in the API.

So API change is not required, as this config only controls the rate of fetching from kafka, you can provide messages one at a time to the library user as before.

Not sure I follow what you are asking for, do you want:

• a consume API that returns a set of messages according to max.poll.records?
• being able to control the number of messages fetched from the broker?

librdkafka pre-fetches messages from the broker into an internal queue which is then served by the application when it calls consumer_poll() (et.al.).

I only need to control the rate of fetch from kafka. I was saying it might not require any changes in the API, since that is independent of the fetch rate.

Can you explain your use-case in more detail?
There are existing configuration options to modify the fetching behaviour:
queued.min.messages, queued.max.messages.kbytes, fetch.wait.max.ms, etc.

https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md

Here's an example of how to implement a batch consume interface:
https://github.com/edenhill/librdkafka/blob/master/examples/rdkafka_consume_batch.cpp#L97

Can't consumer->consume(remaining_timeout) give you more messages than batch_size ?

queued.min.messages, queued.max.messages.kbytes, fetch.wait.max.ms allow us to limit the rate to a certain extent only, and are not a foolproof method to limit the fetch rate to say a 100 messages/sec especially if you have great variation in message sizes.

Instead, a single conf max.poll.records can be used to guarantee that we don't fetch more messages from kafka than configured. I think this is a very valuable feature, lack of throttling capability is a major hinderance to kafka adoption in my org. Official java client already has this feature, but we are restricted to nodejs.

Actually, there is no way to ask the broker for a maximum number of messages, only a maximum total size of messages, this means the Java consumer will also fetch more than max.poll.records but only return a subset of the fetched messages per poll() call.

Even with librdkafka's pre-fetching the fetch rate of the consumer will over time correspond to the consume rate of the application: as the internal fetchq fills up from pre-fetched messages the fetcher will stop fetching until the application has consumed enough messages to make the fetchq drop below the configured thresholds (be it queued.min.messages or queued.max.message.kbytes).

@edenhill Is there any plan to provide support for Java's 'max.poll.records' property in c/c++ ?

@choudhary001 You mean for the consume_batch() API?