what is the impact on Kafka topic partitions log when consumed using multiple Consumer groups?

60
June 17, 2022, at 2:20 PM

My question is more related to performance rather than how they consume data in consumer groups. We know that kafka create single PARTITION LOG on filesystem, which is accessed by all consumer group's consumer on that partition.

What will be the impact on accessing that PARTITION LOG file by multiple consumers from different consumer's group?

  1. Increasing the consumer groups for a specific partition log will slow down accessing that partition or not?
  2. How Kafka handles such concurrent file read I/Os on a single partition log from multiple consumer groups to same parition?
Answer 1
  1. Kafka uses memory-mapped files so reads typically should be served from the memory rather than from the disk.
  2. From Kafka 2.4, Kafka consumers can fetch data from closest follower replicas as well which means that if your partition is replicated across different brokers, then different consumers belonging to different consumer groups consuming the same partition can be served by different brokers thus reducing the load on your leader broker.
  3. In previous versions of Kafka, if multiple consumer groups start consuming messages from the same partition and since that partition leader is responsible for serving those read requests, you can observe some slowness and this is due to several factors including your hardware, OS etc.
  4. Also note that partitions are divided into segments in Kafka where each segment is a file on the disk in the partition directory. Different consumers may be consuming different segments of the same partition.
Rent Charter Buses Company
READ ALSO
how can I use the principle in memory cache?

how can I use the principle in memory cache?

1)I currently have to issue a kinit jumreg@JEMRUGCOM and enter a password to get a ticket

66
How to call manually Response.Listener in Android?

How to call manually Response.Listener in Android?

I have the following method that runs some web services commands in Android

53
JavaScript Error Element is null: while interacting ShadownDom element

JavaScript Error Element is null: while interacting ShadownDom element

I am writing code in Javascript to automate the website https://wwwfashionette

84
ViewModel datas don't update

ViewModel datas don't update

I have an UpdateFragment which display cards group by tags using a RecyclerView and ViewModelIn UpdateFragment I can also modify the name of the tags but I don't know why the change isn't detected by UpdateFragment until I recreate the fragment so UI don't update

74