{"id":444765,"date":"2025-01-12T09:00:41","date_gmt":"2025-01-12T09:00:41","guid":{"rendered":"http:\/\/savepearlharbor.com\/?p=444765"},"modified":"-0001-11-30T00:00:00","modified_gmt":"-0001-11-29T21:00:00","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=444765","title":{"rendered":"<span>Apache Kafka\u2026 Basics to drive<\/span>"},"content":{"rendered":"<div><!--[--><!--]--><\/div>\n<div id=\"post-content-body\">\n<div>\n<div class=\"article-formatted-body article-formatted-body article-formatted-body_version-2\">\n<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<h4>What is Apache Kafka?<\/h4>\n<p>Apache Kafka is a distributed event-streaming platform designed to handle real-time data feeds. It allows applications to publish, process, and subscribe to streams of data in a highly scalable, fault-tolerant manner.<\/p>\n<h4>Core notions in Kafka<\/h4>\n<ol>\n<li>\n<p><strong>Topics<\/strong>:<\/p>\n<ul>\n<li>\n<p>Kafka organizes data into <strong>topics<\/strong>, which are similar to tables in a database.<\/p>\n<\/li>\n<li>\n<p>A topic is divided into <strong>partitions<\/strong>, which enable parallel processing.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Producers<\/strong>:<\/p>\n<ul>\n<li>\n<p>Producers publish data (events\/messages) to Kafka topics.<\/p>\n<\/li>\n<li>\n<p>Data is written to partitions based on a partitioning key.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Consumers<\/strong>:<\/p>\n<ul>\n<li>\n<p>Consumers read data from topics.<\/p>\n<\/li>\n<li>\n<p>Consumers are part of <strong>consumer groups<\/strong>, ensuring load balancing.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Brokers<\/strong>:<\/p>\n<ul>\n<li>\n<p>Brokers are the servers that store and serve Kafka data.<\/p>\n<\/li>\n<li>\n<p>Kafka clusters usually have multiple brokers for scalability and fault tolerance.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Zookeeper\/KRaft<\/strong>:<\/p>\n<ul>\n<li>\n<p>Zookeeper (or the newer KRaft mode) manages metadata and coordinates the cluster.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/849\/644\/8e5\/8496448e5391fd4f2fc9e4a9eba69517.png\" width=\"791\" height=\"253\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/849\/644\/8e5\/8496448e5391fd4f2fc9e4a9eba69517.png\"\/><\/figure>\n<h4>How Kafka Works<\/h4>\n<ol>\n<li>\n<p><strong>Producers send messages<\/strong> to Kafka topics. Messages are stored in partitions within the topics.<\/p>\n<\/li>\n<li>\n<p><strong>Consumers fetch messages<\/strong> from these partitions. Kafka retains messages for a configurable retention period.<\/p>\n<\/li>\n<li>\n<p>Data is stored in a log-based structure, ensuring high write and read throughput.<\/p>\n<\/li>\n<\/ol>\n<p><strong>Stream processing<\/strong> involves handling and analyzing data in real time, as it flows through systems. This is different from batch processing, where data is processed in chunks at intervals.<\/p>\n<p><strong>Example usage HDFS &#8212; Kafka<\/strong><\/p>\n<p>Steps to Load HDFS Data into Kafka:<\/p>\n<ul>\n<li>\n<p><strong>Prepare Data<\/strong>: Split large HDFS files into smaller chunks if necessary, as Kafka messages have a size limit (default: 1 MB).<\/p>\n<\/li>\n<li>\n<p><strong>Producer Tool<\/strong>: Use tools like Kafka Connect (a pluggable data integration framework) or write a custom producer script in Python, Java, etc.<\/p>\n<\/li>\n<li>\n<p><strong>Configuration<\/strong>:<\/p>\n<ul>\n<li>\n<p>Define Kafka topic(s) for the data.<\/p>\n<\/li>\n<li>\n<p>Set partitioning logic (e.g., based on a key or round-robin).<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Test and Monitor<\/strong>: Check data flow using tools like Kafka CLI or monitoring dashboards (e.g., Kafka Manager).<\/p>\n<\/li>\n<\/ul>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/98f\/599\/dd9\/98f599dd90d575eb5189dd6511466c6c.png\" width=\"1297\" height=\"806\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/98f\/599\/dd9\/98f599dd90d575eb5189dd6511466c6c.png\"\/><\/figure>\n<p><strong>Brockers<\/strong><\/p>\n<ul>\n<li>\n<p>A <strong>Broker<\/strong> is a Kafka server that stores data and serves client requests (from Producers and Consumers).<\/p>\n<ul>\n<li>\n<p>Kafka is a distributed system, so it typically consists of multiple brokers, forming a <strong>Kafka cluster<\/strong>.<\/p>\n<\/li>\n<li>\n<p><strong>Key Points<\/strong>:<\/p>\n<ul>\n<li>\n<p>Brokers manage topics, partitions, and message storage.<\/p>\n<\/li>\n<li>\n<p>Each partition of a topic resides on one or more brokers, based on the replication factor.<\/p>\n<\/li>\n<li>\n<p>One broker in the cluster is elected as the <strong>Controller<\/strong> to manage metadata and broker coordination.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h4>Producers<\/h4>\n<ul>\n<li>\n<p>Producers are clients that <strong>publish messages<\/strong> to Kafka topics.<\/p>\n<\/li>\n<li>\n<p><strong>How Producers Interact with Brokers<\/strong>:<\/p>\n<ul>\n<li>\n<p>Producers connect to <strong>one or more brokers<\/strong> (usually via a load balancer or bootstrap server addresses).<\/p>\n<\/li>\n<li>\n<p>The producer sends messages to a specific <strong>topic<\/strong>.<\/p>\n<\/li>\n<li>\n<p><strong>Partition Assignment<\/strong>:<\/p>\n<ul>\n<li>\n<p>A producer decides which <strong>partition<\/strong> a message goes to within the topic.<\/p>\n<\/li>\n<li>\n<p><strong>Default Behavior<\/strong>:<\/p>\n<ul>\n<li>\n<p>If a <strong>key<\/strong> is provided with the message, Kafka uses it to determine the partition using a hashing algorithm.<\/p>\n<\/li>\n<li>\n<p>If no key is provided, Kafka assigns partitions in a round-robin manner.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p>Example:<\/p>\n<ul>\n<li>\n<p>Topic <code>events<\/code> has 3 partitions: <code>events-0<\/code>, <code>events-1<\/code>, <code>events-2<\/code>.<\/p>\n<\/li>\n<li>\n<p>A producer sends a message with key <code>user123<\/code>. Kafka hashes the key to assign the message to a specific partition (e.g., <code>events-1<\/code>).<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h4>Consumers<\/h4>\n<ul>\n<li>\n<p>Consumers are clients that <strong>read messages<\/strong> from Kafka topics.<\/p>\n<\/li>\n<li>\n<p><strong>How Consumers Interact with Brokers<\/strong>:<\/p>\n<ul>\n<li>\n<p>Consumers subscribe to one or more topics and consume messages from partitions.<\/p>\n<\/li>\n<li>\n<p>Kafka ensures that messages are delivered <strong>in order<\/strong> within a partition.<\/p>\n<\/li>\n<li>\n<p>Consumers in a <strong>consumer group<\/strong> share the workload:<\/p>\n<ul>\n<li>\n<p>Each consumer in the group is assigned one or more partitions.<\/p>\n<\/li>\n<li>\n<p><strong>Example<\/strong>:<\/p>\n<ul>\n<li>\n<p>Topic <code>logs<\/code> has 3 partitions.<\/p>\n<\/li>\n<li>\n<p>Consumer Group <code>group1<\/code> has 2 consumers: Consumer A and Consumer B.<\/p>\n<\/li>\n<li>\n<p>Consumer A reads from <code>logs-0<\/code> and <code>logs-1<\/code>, while Consumer B reads from <code>logs-2<\/code>.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h4>Broker Coordination<\/h4>\n<ul>\n<li>\n<p>Kafka uses a <strong>leader-follower<\/strong> model for partition replication:<\/p>\n<ul>\n<li>\n<p>For each partition, one broker acts as the <strong>leader<\/strong> and handles all read and write requests.<\/p>\n<\/li>\n<li>\n<p>Other brokers store replica data and act as <strong>followers<\/strong>.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p>Producers and consumers always interact with the <strong>leader broker<\/strong> for a partition.<\/p>\n<\/li>\n<li>\n<p><strong>Failover<\/strong>:<\/p>\n<ul>\n<li>\n<p>If a broker fails, Kafka elects a new leader for the affected partitions from the available replicas.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<hr\/>\n<h4>Interconnections in Action<\/h4>\n<ol>\n<li>\n<p><strong>Producers to Brokers<\/strong>:<\/p>\n<ul>\n<li>\n<p>A producer queries the cluster metadata to determine which broker is the leader for a partition.<\/p>\n<\/li>\n<li>\n<p>The producer sends messages directly to the leader broker for that partition.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Brokers to Consumers<\/strong>:<\/p>\n<ul>\n<li>\n<p>Consumers fetch metadata from the cluster to discover which partitions belong to their subscribed topic(s).<\/p>\n<\/li>\n<li>\n<p>Consumers connect directly to the leader broker(s) for those partitions to fetch data.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Brokers to Each Other<\/strong>:<\/p>\n<ul>\n<li>\n<p>Brokers exchange metadata (e.g., partition assignments and replicas) to keep the cluster consistent.<\/p>\n<\/li>\n<li>\n<p>Replication occurs between brokers to ensure fault tolerance.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<h4>Data Flow Example<\/h4>\n<ol>\n<li>\n<p><strong>Producer sends a message<\/strong>:<\/p>\n<ul>\n<li>\n<p><em>Producer connects to the cluster and queries metadata.<\/em><\/p>\n<\/li>\n<li>\n<p><em>Metadata response identifies the leader broker for the target partition.<\/em><\/p>\n<\/li>\n<li>\n<p><em>Producer sends the message to the leader br<\/em>oker.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Broker stores the message<\/strong>:<\/p>\n<ul>\n<li>\n<p><em>The leader broker appends the message to the partition\u2019s log.<\/em><\/p>\n<\/li>\n<li>\n<p><em>Replication: The leader propagates the message to follower brokers.<\/em><\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Consumer reads the message<\/strong>:<\/p>\n<ul>\n<li>\n<p><em>Consumer fetches metadata to identify the leader broker for the assigned partition.<\/em><\/p>\n<\/li>\n<li>\n<p><em>Consumer connects to the leader broker and fetches messages.<\/em><\/p>\n<\/li>\n<li>\n<p><em>Kafka tracks the <\/em><strong><em>offset<\/em><\/strong><em> of each message to ensure no data is skipped or duplicated.<\/em><\/p>\n<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<hr\/>\n<h4>Key Advantages of These Interconnections<\/h4>\n<ol>\n<li>\n<p><strong>Scalability<\/strong>:<\/p>\n<ul>\n<li>\n<p>Adding brokers increases the cluster\u2019s capacity.<\/p>\n<\/li>\n<li>\n<p>Producers and consumers automatically adjust to the new cluster size.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Fault Tolerance<\/strong>:<\/p>\n<ul>\n<li>\n<p>Replication ensures data availability even if a broker fails.<\/p>\n<\/li>\n<li>\n<p>Consumers can continue reading from replicas if a leader broker goes down.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>High Throughput<\/strong>:<\/p>\n<ul>\n<li>\n<p>Partitioning enables parallel processing.<\/p>\n<\/li>\n<li>\n<p>Producers and consumers can write and read from multiple brokers simultaneously.<\/p>\n<pre><code>Producers --&gt; [Broker1 (Leader)] &lt;-- Consumers               [Broker2 (Follower)]               [Broker3 (Follower)]<\/code><\/pre>\n<p>In this example:<\/p>\n<ul>\n<li>\n<p>Producer sends data to <code>Broker1<\/code>, the leader of a partition.<\/p>\n<\/li>\n<li>\n<p><code>Broker2<\/code> and <code>Broker3<\/code> replicate the data from <code>Broker1<\/code>.<\/p>\n<\/li>\n<li>\n<p>Consumers fetch data from <code>Broker1<\/code> (or one of the followers if configured).<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/3c1\/b14\/960\/3c1b14960468fb5173f672518cc5168f.png\" width=\"1353\" height=\"1305\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/3c1\/b14\/960\/3c1b14960468fb5173f672518cc5168f.png\"\/><\/figure>\n<h3>Kafka&#8217;s Storage Model<\/h3>\n<ol>\n<li>\n<p><strong>Log-Based Storage<\/strong>:<\/p>\n<ul>\n<li>\n<p>Kafka organizes data into <strong>topics<\/strong>, and each topic is divided into <strong>partitions<\/strong>.<\/p>\n<\/li>\n<li>\n<p>Each partition is an <strong>append-only log<\/strong> stored on disk.<\/p>\n<\/li>\n<li>\n<p>Messages are written sequentially to the end of the log, making writes extremely fast due to minimal disk seek overhead.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Retention-Based Storage<\/strong>:<\/p>\n<ul>\n<li>\n<p>Kafka retains messages for a configurable <strong>retention period<\/strong> (e.g., 7 days by default) or until the log reaches a certain size.<\/p>\n<\/li>\n<li>\n<p>Old messages are automatically deleted after they exceed the retention policy.<\/p>\n<\/li>\n<li>\n<p>Kafka is not designed for <strong>long-term data storage<\/strong> like HDFS; it is optimized for real-time streaming and temporary storage.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>File Organization<\/strong>:<\/p>\n<ul>\n<li>\n<p>Each partition is stored as a set of files on the broker&#8217;s disk.<\/p>\n<\/li>\n<li>\n<p>These files are segmented for efficient access (e.g., log-0, log-1).<\/p>\n<\/li>\n<li>\n<p>Kafka indexes these log files to enable fast lookups by message offset.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<h4>Kafka vs. HDFS Storage<\/h4>\n<div>\n<div class=\"table\">\n<table>\n<tbody>\n<tr>\n<th>\n<p>Feature<\/p>\n<\/th>\n<th>\n<p>Kafka<\/p>\n<\/th>\n<th>\n<p>HDFS<\/p>\n<\/th>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\"><strong>Purpose<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">Real-time streaming, message queue<\/p>\n<\/td>\n<td>\n<p align=\"left\">Long-term distributed file storage<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\"><strong>Data Retention<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">Configurable (e.g., time\/size-based)<\/p>\n<\/td>\n<td>\n<p align=\"left\">Permanent until explicitly deleted<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\"><strong>Read\/Write Pattern<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">Append-only log, sequential writes<\/p>\n<\/td>\n<td>\n<p align=\"left\">Random access, distributed writes<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\"><strong>Scalability<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">Horizontal scaling via brokers<\/p>\n<\/td>\n<td>\n<p align=\"left\">Horizontal scaling via DataNodes<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\"><strong>Fault Tolerance<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">Replication at partition level<\/p>\n<\/td>\n<td>\n<p align=\"left\">Replication across DataNodes<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\"><strong>Primary Use Case<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">Real-time data pipelines<\/p>\n<\/td>\n<td>\n<p align=\"left\">Storing large datasets for analysis<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n<figure class=\"full-width\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/habrastorage.org\/r\/w1560\/getpro\/habr\/upload_files\/186\/069\/9b6\/1860699b6c176947c2a1e0052052d5df.png\" width=\"2840\" height=\"1624\" data-src=\"https:\/\/habrastorage.org\/getpro\/habr\/upload_files\/186\/069\/9b6\/1860699b6c176947c2a1e0052052d5df.png\"\/><\/figure>\n<h4>How Kafka Handles Storage<\/h4>\n<ol>\n<li>\n<p><strong>Persistence<\/strong>:<\/p>\n<ul>\n<li>\n<p>Kafka stores all data on disk, even though it&#8217;s a messaging system.<\/p>\n<\/li>\n<li>\n<p>This ensures durability and allows consumers to replay messages if needed.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Replication<\/strong>:<\/p>\n<ul>\n<li>\n<p>Messages in Kafka are replicated across brokers for fault tolerance.<\/p>\n<\/li>\n<li>\n<p>For example, a topic with a replication factor of 3 will have its data stored on three brokers.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Segmented Storage<\/strong>:<\/p>\n<ul>\n<li>\n<p>Kafka splits each partition&#8217;s log into <strong>segments<\/strong>.<\/p>\n<\/li>\n<li>\n<p>When a segment reaches a configured size, Kafka creates a new segment.<\/p>\n<\/li>\n<li>\n<p>Old segments are deleted according to the retention policy.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Compaction<\/strong> (Optional):<\/p>\n<ul>\n<li>\n<p>Kafka offers a <strong>log compaction<\/strong> feature, which retains only the latest value for each key.<\/p>\n<\/li>\n<li>\n<p>This is useful for scenarios like updating state or maintaining a compact view of data.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<p>Kafka does have its own storage, but it is designed for <strong>short-term storage<\/strong> and high-throughput message delivery. For long-term storage or big data analysis, Kafka is usually paired with systems like HDFS, S3, or other data lakes.<\/p>\n<\/p>\n<\/div>\n<\/div>\n<\/div>\n<p><!----><!----><\/div>\n<p><!----><!----><br \/> \u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438 <a href=\"https:\/\/habr.com\/ru\/articles\/872976\/\"> https:\/\/habr.com\/ru\/articles\/872976\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<div><!--[--><!--]--><\/div>\n<div id=\"post-content-body\">\n<div>\n<div class=\"article-formatted-body article-formatted-body article-formatted-body_version-2\">\n<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<h4>What is Apache Kafka?<\/h4>\n<p>Apache Kafka is a distributed event-streaming platform designed to handle real-time data feeds. It allows applications to publish, process, and subscribe to streams of data in a highly scalable, fault-tolerant manner.<\/p>\n<h4>Core notions in Kafka<\/h4>\n<ol>\n<li>\n<p><strong>Topics<\/strong>:<\/p>\n<ul>\n<li>\n<p>Kafka organizes data into <strong>topics<\/strong>, which are similar to tables in a database.<\/p>\n<\/li>\n<li>\n<p>A topic is divided into <strong>partitions<\/strong>, which enable parallel processing.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Producers<\/strong>:<\/p>\n<ul>\n<li>\n<p>Producers publish data (events\/messages) to Kafka topics.<\/p>\n<\/li>\n<li>\n<p>Data is written to partitions based on a partitioning key.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Consumers<\/strong>:<\/p>\n<ul>\n<li>\n<p>Consumers read data from topics.<\/p>\n<\/li>\n<li>\n<p>Consumers are part of <strong>consumer groups<\/strong>, ensuring load balancing.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Brokers<\/strong>:<\/p>\n<ul>\n<li>\n<p>Brokers are the servers that store and serve Kafka data.<\/p>\n<\/li>\n<li>\n<p>Kafka clusters usually have multiple brokers for scalability and fault tolerance.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Zookeeper\/KRaft<\/strong>:<\/p>\n<ul>\n<li>\n<p>Zookeeper (or the newer KRaft mode) manages metadata and coordinates the cluster.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<figure class=\"full-width\"><\/figure>\n<h4>How Kafka Works<\/h4>\n<ol>\n<li>\n<p><strong>Producers send messages<\/strong> to Kafka topics. Messages are stored in partitions within the topics.<\/p>\n<\/li>\n<li>\n<p><strong>Consumers fetch messages<\/strong> from these partitions. Kafka retains messages for a configurable retention period.<\/p>\n<\/li>\n<li>\n<p>Data is stored in a log-based structure, ensuring high write and read throughput.<\/p>\n<\/li>\n<\/ol>\n<p><strong>Stream processing<\/strong> involves handling and analyzing data in real time, as it flows through systems. This is different from batch processing, where data is processed in chunks at intervals.<\/p>\n<p><strong>Example usage HDFS &#8212; Kafka<\/strong><\/p>\n<p>Steps to Load HDFS Data into Kafka:<\/p>\n<ul>\n<li>\n<p><strong>Prepare Data<\/strong>: Split large HDFS files into smaller chunks if necessary, as Kafka messages have a size limit (default: 1 MB).<\/p>\n<\/li>\n<li>\n<p><strong>Producer Tool<\/strong>: Use tools like Kafka Connect (a pluggable data integration framework) or write a custom producer script in Python, Java, etc.<\/p>\n<\/li>\n<li>\n<p><strong>Configuration<\/strong>:<\/p>\n<ul>\n<li>\n<p>Define Kafka topic(s) for the data.<\/p>\n<\/li>\n<li>\n<p>Set partitioning logic (e.g., based on a key or round-robin).<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Test and Monitor<\/strong>: Check data flow using tools like Kafka CLI or monitoring dashboards (e.g., Kafka Manager).<\/p>\n<\/li>\n<\/ul>\n<figure class=\"full-width\"><\/figure>\n<p><strong>Brockers<\/strong><\/p>\n<ul>\n<li>\n<p>A <strong>Broker<\/strong> is a Kafka server that stores data and serves client requests (from Producers and Consumers).<\/p>\n<ul>\n<li>\n<p>Kafka is a distributed system, so it typically consists of multiple brokers, forming a <strong>Kafka cluster<\/strong>.<\/p>\n<\/li>\n<li>\n<p><strong>Key Points<\/strong>:<\/p>\n<ul>\n<li>\n<p>Brokers manage topics, partitions, and message storage.<\/p>\n<\/li>\n<li>\n<p>Each partition of a topic resides on one or more brokers, based on the replication factor.<\/p>\n<\/li>\n<li>\n<p>One broker in the cluster is elected as the <strong>Controller<\/strong> to manage metadata and broker coordination.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h4>Producers<\/h4>\n<ul>\n<li>\n<p>Producers are clients that <strong>publish messages<\/strong> to Kafka topics.<\/p>\n<\/li>\n<li>\n<p><strong>How Producers Interact with Brokers<\/strong>:<\/p>\n<ul>\n<li>\n<p>Producers connect to <strong>one or more brokers<\/strong> (usually via a load balancer or bootstrap server addresses).<\/p>\n<\/li>\n<li>\n<p>The producer sends messages to a specific <strong>topic<\/strong>.<\/p>\n<\/li>\n<li>\n<p><strong>Partition Assignment<\/strong>:<\/p>\n<ul>\n<li>\n<p>A producer decides which <strong>partition<\/strong> a message goes to within the topic.<\/p>\n<\/li>\n<li>\n<p><strong>Default Behavior<\/strong>:<\/p>\n<ul>\n<li>\n<p>If a <strong>key<\/strong> is provided with the message, Kafka uses it to determine the partition using a hashing algorithm.<\/p>\n<\/li>\n<li>\n<p>If no key is provided, Kafka assigns partitions in a round-robin manner.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p>Example:<\/p>\n<ul>\n<li>\n<p>Topic <code>events<\/code> has 3 partitions: <code>events-0<\/code>, <code>events-1<\/code>, <code>events-2<\/code>.<\/p>\n<\/li>\n<li>\n<p>A producer sends a message with key <code>user123<\/code>. Kafka hashes the key to assign the message to a specific partition (e.g., <code>events-1<\/code>).<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h4>Consumers<\/h4>\n<ul>\n<li>\n<p>Consumers are clients that <strong>read messages<\/strong> from Kafka topics.<\/p>\n<\/li>\n<li>\n<p><strong>How Consumers Interact with Brokers<\/strong>:<\/p>\n<ul>\n<li>\n<p>Consumers subscribe to one or more topics and consume messages from partitions.<\/p>\n<\/li>\n<li>\n<p>Kafka ensures that messages are delivered <strong>in order<\/strong> within a partition.<\/p>\n<\/li>\n<li>\n<p>Consumers in a <strong>consumer group<\/strong> share the workload:<\/p>\n<ul>\n<li>\n<p>Each consumer in the group is assigned one or more partitions.<\/p>\n<\/li>\n<li>\n<p><strong>Example<\/strong>:<\/p>\n<ul>\n<li>\n<p>Topic <code>logs<\/code> has 3 partitions.<\/p>\n<\/li>\n<li>\n<p>Consumer Group <code>group1<\/code> has 2 consumers: Consumer A and Consumer B.<\/p>\n<\/li>\n<li>\n<p>Consumer A reads from <code>logs-0<\/code> and <code>logs-1<\/code>, while Consumer B reads from <code>logs-2<\/code>.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h4>Broker Coordination<\/h4>\n<ul>\n<li>\n<p>Kafka uses a <strong>leader-follower<\/strong> model for partition replication:<\/p>\n<ul>\n<li>\n<p>For each partition, one broker acts as the <strong>leader<\/strong> and handles all read and write requests.<\/p>\n<\/li>\n<li>\n<p>Other brokers store replica data and act as <strong>followers<\/strong>.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p>Producers and consumers always interact with the <strong>leader broker<\/strong> for a partition.<\/p>\n<\/li>\n<li>\n<p><strong>Failover<\/strong>:<\/p>\n<ul>\n<li>\n<p>If a broker fails, Kafka elects a new leader for the affected partitions from the available replicas.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<hr\/>\n<h4>Interconnections in Action<\/h4>\n<ol>\n<li>\n<p><strong>Producers to Brokers<\/strong>:<\/p>\n<ul>\n<li>\n<p>A producer queries the cluster metadata to determine which broker is the leader for a partition.<\/p>\n<\/li>\n<li>\n<p>The producer sends messages directly to the leader broker for that partition.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Brokers to Consumers<\/strong>:<\/p>\n<ul>\n<li>\n<p>Consumers fetch metadata from the cluster to discover which partitions belong to their subscribed topic(s).<\/p>\n<\/li>\n<li>\n<p>Consumers connect directly to the leader broker(s) for those partitions to fetch data.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Brokers to Each Other<\/strong>:<\/p>\n<ul>\n<li>\n<p>Brokers exchange metadata (e.g., partition assignments and replicas) to keep the cluster consistent.<\/p>\n<\/li>\n<li>\n<p>Replication occurs between brokers to ensure fault tolerance.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<h4>Data Flow Example<\/h4>\n<ol>\n<li>\n<p><strong>Producer sends a message<\/strong>:<\/p>\n<ul>\n<li>\n<p><em>Producer connects to the cluster and queries metadata.<\/em><\/p>\n<\/li>\n<li>\n<p><em>Metadata response identifies the leader broker for the target partition.<\/em><\/p>\n<\/li>\n<li>\n<p><em>Producer sends the message to the leader br<\/em>oker.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Broker stores the message<\/strong>:<\/p>\n<ul>\n<li>\n<p><em>The leader broker appends the message to the partition\u2019s log.<\/em><\/p>\n<\/li>\n<li>\n<p><em>Replication: The leader propagates the message to follower brokers.<\/em><\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Consumer reads the message<\/strong>:<\/p>\n<ul>\n<li>\n<p><em>Consumer fetches metadata to identify the leader broker for the assigned partition.<\/em><\/p>\n<\/li>\n<li>\n<p><em>Consumer connects to the leader broker and fetches messages.<\/em><\/p>\n<\/li>\n<li>\n<p><em>Kafka tracks the <\/em><strong><em>offset<\/em><\/strong><em> of each message to ensure no data is skipped or duplicated.<\/em><\/p>\n<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<hr\/>\n<h4>Key Advantages of These Interconnections<\/h4>\n<ol>\n<li>\n<p><strong>Scalability<\/strong>:<\/p>\n<ul>\n<li>\n<p>Adding brokers increases the cluster\u2019s capacity.<\/p>\n<\/li>\n<li>\n<p>Producers and consumers automatically adjust to the new cluster size.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Fault Tolerance<\/strong>:<\/p>\n<ul>\n<li>\n<p>Replication ensures data availability even if a broker fails.<\/p>\n<\/li>\n<li>\n<p>Consumers can continue reading from replicas if a leader broker goes down.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>High Throughput<\/strong>:<\/p>\n<ul>\n<li>\n<p>Partitioning enables parallel processing.<\/p>\n<\/li>\n<li>\n<p>Producers and consumers can write and read from multiple brokers simultaneously.<\/p>\n<pre><code>Producers --&gt; [Broker1 (Leader)] &lt;-- Consumers               [Broker2 (Follower)]               [Broker3 (Follower)]<\/code><\/pre>\n<p>In this example:<\/p>\n<ul>\n<li>\n<p>Producer sends data to <code>Broker1<\/code>, the leader of a partition.<\/p>\n<\/li>\n<li>\n<p><code>Broker2<\/code> and <code>Broker3<\/code> replicate the data from <code>Broker1<\/code>.<\/p>\n<\/li>\n<li>\n<p>Consumers fetch data from <code>Broker1<\/code> (or one of the followers if configured).<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<figure class=\"full-width\"><\/figure>\n<h3>Kafka&#8217;s Storage Model<\/h3>\n<ol>\n<li>\n<p><strong>Log-Based Storage<\/strong>:<\/p>\n<ul>\n<li>\n<p>Kafka organizes data into <strong>topics<\/strong>, and each topic is divided into <strong>partitions<\/strong>.<\/p>\n<\/li>\n<li>\n<p>Each partition is an <strong>append-only log<\/strong> stored on disk.<\/p>\n<\/li>\n<li>\n<p>Messages are written sequentially to the end of the log, making writes extremely fast due to minimal disk seek overhead.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Retention-Based Storage<\/strong>:<\/p>\n<ul>\n<li>\n<p>Kafka retains messages for a configurable <strong>retention period<\/strong> (e.g., 7 days by default) or until the log reaches a certain size.<\/p>\n<\/li>\n<li>\n<p>Old messages are automatically deleted after they exceed the retention policy.<\/p>\n<\/li>\n<li>\n<p>Kafka is not designed for <strong>long-term data storage<\/strong> like HDFS; it is optimized for real-time streaming and temporary storage.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>File Organization<\/strong>:<\/p>\n<ul>\n<li>\n<p>Each partition is stored as a set of files on the broker&#8217;s disk.<\/p>\n<\/li>\n<li>\n<p>These files are segmented for efficient access (e.g., log-0, log-1).<\/p>\n<\/li>\n<li>\n<p>Kafka indexes these log files to enable fast lookups by message offset.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<h4>Kafka vs. HDFS Storage<\/h4>\n<div>\n<div class=\"table\">\n<table>\n<tbody>\n<tr>\n<th>\n<p>Feature<\/p>\n<\/th>\n<th>\n<p>Kafka<\/p>\n<\/th>\n<th>\n<p>HDFS<\/p>\n<\/th>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\"><strong>Purpose<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">Real-time streaming, message queue<\/p>\n<\/td>\n<td>\n<p align=\"left\">Long-term distributed file storage<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\"><strong>Data Retention<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">Configurable (e.g., time\/size-based)<\/p>\n<\/td>\n<td>\n<p align=\"left\">Permanent until explicitly deleted<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\"><strong>Read\/Write Pattern<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">Append-only log, sequential writes<\/p>\n<\/td>\n<td>\n<p align=\"left\">Random access, distributed writes<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\"><strong>Scalability<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">Horizontal scaling via brokers<\/p>\n<\/td>\n<td>\n<p align=\"left\">Horizontal scaling via DataNodes<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\"><strong>Fault Tolerance<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">Replication at partition level<\/p>\n<\/td>\n<td>\n<p align=\"left\">Replication across DataNodes<\/p>\n<\/td>\n<\/tr>\n<tr>\n<td>\n<p align=\"left\"><strong>Primary Use Case<\/strong><\/p>\n<\/td>\n<td>\n<p align=\"left\">Real-time data pipelines<\/p>\n<\/td>\n<td>\n<p align=\"left\">Storing large datasets for analysis<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n<figure class=\"full-width\"><\/figure>\n<h4>How Kafka Handles Storage<\/h4>\n<ol>\n<li>\n<p><strong>Persistence<\/strong>:<\/p>\n<ul>\n<li>\n<p>Kafka stores all data on disk, even though it&#8217;s a messaging system.<\/p>\n<\/li>\n<li>\n<p>This ensures durability and allows consumers to replay messages if needed.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Replication<\/strong>:<\/p>\n<ul>\n<li>\n<p>Messages in Kafka are replicated across brokers for fault tolerance.<\/p>\n<\/li>\n<li>\n<p>For example, a topic with a replication factor of 3 will have its data stored on three brokers.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Segmented Storage<\/strong>:<\/p>\n<ul>\n<li>\n<p>Kafka splits each partition&#8217;s log into <strong>segments<\/strong>.<\/p>\n<\/li>\n<li>\n<p>When a segment reaches a configured size, Kafka creates a new segment.<\/p>\n<\/li>\n<li>\n<p>Old segments are deleted according to the retention policy.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><strong>Compaction<\/strong> (Optional):<\/p>\n<ul>\n<li>\n<p>Kafka offers a <strong>log compaction<\/strong> feature, which retains only the latest value for each key.<\/p>\n<\/li>\n<li>\n<p>This is useful for scenarios like updating state or maintaining a compact view of data.<\/p>\n<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<p>Kafka does have its own storage, but it is designed for <strong>short-term storage<\/strong> and high-throughput message delivery. For long-term storage or big data analysis, Kafka is usually paired with systems like HDFS, S3, or other data lakes.<\/p>\n<\/p>\n<\/div>\n<\/div>\n<\/div>\n<p><!----><!----><\/div>\n<p><!----><!----><br \/> \u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438 <a href=\"https:\/\/habr.com\/ru\/articles\/872976\/\"> https:\/\/habr.com\/ru\/articles\/872976\/<\/a><br \/><\/br><\/br><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[],"class_list":["post-444765","post","type-post","status-publish","format-standard","hentry"],"_links":{"self":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/444765","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=444765"}],"version-history":[{"count":0,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/444765\/revisions"}],"wp:attachment":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=444765"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=444765"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=444765"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}