Is there any way to check if all the brokers are up and running in Kafka?

Question

I am using Confluent.kafka for my c# publisher and consumer application. Before I publish or consume any message, I would like to check if all the brokers (Endpoints) are up and running.

I have found some articles around it but most of them are not conclusive, or they are for Java, I am specifically looking for .net.

Thanks

Check out this thread: https://stackoverflow.com/questions/54032314/how-to-ensure-a-kafka-cluster-is-fully-up/54032716#54032716 It can be automated using bash scripts if that's fine with you. — Bitswazsky, Jan 04 '19 at 13:34

OneCricketeer · Answer 1 · 2019-01-18T05:32:11.243

If you know how many brokers should be in the cluster, you can use the AdminClient to list out all the brokers that the Kafka Controller knows about

using (var adminClient = new AdminClient(new AdminClientConfig { BootstrapServers = brokerList }))
{
    var meta = adminClient.GetMetadata(TimeSpan.FromSeconds(20));
    Console.WriteLine($"{meta.OriginatingBrokerId} {meta.OriginatingBrokerName}");
    meta.Brokers.ForEach(broker =>
        Console.WriteLine($"Broker: {broker.BrokerId} {broker.Host}:{broker.Port}"));

Source - https://github.com/confluentinc/confluent-kafka-dotnet/blob/master/examples/AdminClient/Program.cs

However, you really only would need to check the subset of brokers where a topic replicas exist (i.e. describe the topic instead and check the replica count), and that there is a Controller available at all (which there would be if the AdminClient returned data)... The Producer and Consumer already provide exception details for when the connection isn't available to be made as well.

Not sure what you mean. If a broker is down, it will not be returned... Like I said, though, not **all** brokers need to be running for you to produce/consume to a single topic (assuming the partitions*replicas are less than your total broker amount) — OneCricketeer, Jan 08 '19 at 18:35

Is there any way to check if all the brokers are up and running in Kafka?

1 Answers1