Release Notes - Kafka - Version 3.7.0

Below is a summary of the JIRA issues addressed in the 3.7.0 release of Kafka. For full documentation of the release, a guide to get started, and information about the project, see the Kafka project site.

Note about upgrades: Please carefully review the upgrade documentation for this release thoroughly before upgrading your cluster. The upgrade notes discuss any critical information about incompatibilities and breaking changes, performance changes, and any other changes that might impact your production deployment of Kafka.

The documentation for the most recent release can be found at https://kafka.apache.org/documentation.html.

New Feature

[KAFKA-9800] - [KIP-580] Client Exponential Backoff Implementation
[KAFKA-15215] - The default.dsl.store config is not compatible with custom state stores
[KAFKA-15428] - Cluster-wide dynamic log adjustments for Connect
[KAFKA-15445] - KIP-975: Docker Image for Apache Kafka
[KAFKA-15448] - Streams StandbyTaskUpdateListener
[KAFKA-15470] - Allow creating connectors in a stopped state

Improvement

[KAFKA-9693] - Kafka latency spikes caused by log segment flush on roll
[KAFKA-10199] - Separate state restoration into separate threads
[KAFKA-14127] - KIP-858: Handle JBOD broker disk failure in KRaft
[KAFKA-14780] - Make RefreshingHttpsJwksTest#testSecondaryRefreshAfterElapsedDelay deterministic
[KAFKA-14855] - Harden integration testing logic for asserting that a connector is deleted
[KAFKA-14912] - Introduce a configuration for remote index cache size, preferably a dynamic config.
[KAFKA-15022] - Support rack aware task assignment in Kafka streams
[KAFKA-15046] - Produce performance issue under high disk load
[KAFKA-15047] - Handle rolling segments when the active segment's retention is breached incase of tiered storage is enabled.
[KAFKA-15141] - High CPU usage with log4j2
[KAFKA-15147] - Measure pending and outstanding Remote Segment operations
[KAFKA-15208] - Upgrade Jackson dependencies to version 2.16.0
[KAFKA-15241] - Compute tiered offset by keeping the respective epochs in scope.
[KAFKA-15248] - Add BooleanConverter to Kafka Connect
[KAFKA-15273] - Log common name of expired client certificate
[KAFKA-15315] - Use getOrDefault rather than get
[KAFKA-15401] - Segment with corrupted index should not be uploaded to remote storage
[KAFKA-15415] - In Java-client, backoff should be skipped for retried producer-batch to a new leader
[KAFKA-15418] - Update statement on decompression location
[KAFKA-15432] - RLM Stop partitions should not be invoked for non-tiered storage topics
[KAFKA-15464] - Allow dynamic reloading of certificates with different DN / SANs
[KAFKA-15471] - Allow independently stop KRaft controllers or brokers
[KAFKA-15476] - Improve checkstyle performance
[KAFKA-15485] - Support building with Java 21 (LTS release)
[KAFKA-15492] - Enable spotbugs when building with Java 21
[KAFKA-15493] - Ensure system tests work with Java 21
[KAFKA-15521] - Refactor build.gradle to align gradle swagger plugin with swagger dependencies
[KAFKA-15527] - Add reverseRange and reverseAll query over kv-store in IQv2
[KAFKA-15536] - dynamically resize remoteIndexCache
[KAFKA-15542] - Release member assignments on errors
[KAFKA-15563] - Provide informative error messages when Connect REST requests time out
[KAFKA-15566] - Flaky tests in FetchRequestTest.scala in KRaft mode
[KAFKA-15596] - Upgrade ZooKeeper to 3.8.3
[KAFKA-15629] - proposal to introduce IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery
[KAFKA-15685] - Add missing compatibility for MinGW and MSYS2 (windows)
[KAFKA-15769] - Fix wrong log with exception
[KAFKA-15774] - Respect default.dsl.store Configuration Without Passing it to StreamsBuilder
[KAFKA-15831] - List Client Metrics Configuration Resources
[KAFKA-15837] - Throw error on use of Consumer.poll(long timeout)
[KAFKA-15866] - Refactor OffsetFetchRequestState Error handling to be more consistent with OffsetCommitRequestState
[KAFKA-15868] - KIP-951 - Leader discovery optimisations for the client
[KAFKA-15906] - Emit offset syncs more often than offset.lag.max for low-throughput/finite partitions
[KAFKA-15922] - Add MetadataVersion for JBOD
[KAFKA-15971] - Re-enable consumer integration tests for new consumer
[KAFKA-15980] - Add KIP-1001 CurrentControllerId metric
[KAFKA-16007] - ZK migrations can be slow for large clusters
[KAFKA-16210] - Upgrade jose4j to 0.9.4

Bug

[KAFKA-12679] - Rebalancing a restoring or running task may cause directory livelocking with newly created task
[KAFKA-13327] - Preflight validations of connectors leads to 500 responses
[KAFKA-13973] - block-cache-capacity metrics worth twice as much as normal
[KAFKA-13988] - Mirrormaker 2 auto.offset.reset=latest not working
[KAFKA-14067] - Sink connector override.consumer.group.id can conflict with worker group.id
[KAFKA-14616] - Topic recreation with offline broker causes permanent URPs
[KAFKA-14767] - Gradle build fails with missing commitId after git gc
[KAFKA-14927] - Prevent kafka-configs.sh from setting non-alphanumeric config key names
[KAFKA-15000] - High vulnerability PRISMA-2023-0067 reported in jackson-core
[KAFKA-15117] - SslTransportLayerTest.testValidEndpointIdentificationCN fails with Java 20 & 21
[KAFKA-15152] - Fix incorrect format specifiers when formatting string
[KAFKA-15221] - Potential race condition between requests from rebooted followers
[KAFKA-15230] - ApiVersions data between controllers is not reliable
[KAFKA-15311] - Fix docs about reverting to ZooKeeper mode during KRaft migration
[KAFKA-15372] - MM2 rolling restart can drop configuration changes silently
[KAFKA-15392] - RestServer starts but does not stop ServletContextHandler
[KAFKA-15412] - Reading an unknown version of quorum-state-file should trigger an error
[KAFKA-15465] - MM2 not working when its internal topics are pre-created on a cluster that disallows topic creation
[KAFKA-15473] - Connect connector-plugins endpoint shows duplicate plugins
[KAFKA-15481] - Concurrency bug in RemoteIndexCache leads to IOException
[KAFKA-15489] - split brain in KRaft cluster
[KAFKA-15491] - RackId doesn't exist error while running WordCountDemo
[KAFKA-15500] - Code bug in SslPrincipalMapper.java
[KAFKA-15502] - Handle large keystores in SslEngineValidator
[KAFKA-15507] - adminClient should not throw retriable exception when closing instance
[KAFKA-15510] - Follower's lastFetchedEpoch wrongly set when fetch response has no record
[KAFKA-15511] - Exception not handled correctly if indexFile is corrupted.
[KAFKA-15537] - Unsafe metadata.version downgrade is not supported
[KAFKA-15571] - StateRestoreListener#onRestoreSuspended is never called because wrapper DelegatingStateRestoreListener doesn't implement onRestoreSuspended
[KAFKA-15602] - Breaking change in 3.4.0 ByteBufferSerializer
[KAFKA-15605] - Topics marked for deletion in ZK are incorrectly migrated to KRaft
[KAFKA-15607] - Possible NPE is thrown in MirrorCheckpointTask
[KAFKA-15644] - Fix CVE-2023-4586 in netty:handler
[KAFKA-15653] - NPE in ChunkedByteStream
[KAFKA-15658] - Zookeeper.jar | CVE-2023-44981
[KAFKA-15680] - Partition-Count is not getting updated Correctly in the Incremental Co-operative Rebalancing(ICR) Mode of Rebalancing
[KAFKA-15689] - KRaftMigrationDriver not logging the skipped event when expected state is wrong
[KAFKA-15693] - Disabling scheduled rebalance delay in Connect can lead to indefinitely unassigned connectors and tasks
[KAFKA-15695] - Local log start offset is not updated on the follower after rebuilding remote log auxiliary state
[KAFKA-15704] - ControllerRegistrationRequest must set ZkMigrationReady field if appropriate
[KAFKA-15755] - LeaveGroupResponse v0-v2 should handle no members
[KAFKA-15771] - ProduceRequest#partitionSizes() is not an atomic operation
[KAFKA-15799] - ZK brokers incorrectly handle KRaft metadata snapshots
[KAFKA-15800] - Malformed connect source offsets corrupt other partitions with DataException
[KAFKA-15802] - Trying to access uncopied segments metadata on listOffsets
[KAFKA-15817] - Avoid reconnecting to the same IP address if multiple addresses are available
[KAFKA-15819] - KafkaServer leaks KafkaRaftManager when ZK migration enabled
[KAFKA-15824] - SubscriptionState's maybeValidatePositionForCurrentLeader should handle partition which isn't subscribed yet
[KAFKA-15825] - KRaft controller writes empty state to ZK after migration
[KAFKA-15836] - KafkaConsumer subscribes to multiple topics does not respect max.poll.records
[KAFKA-15860] - ControllerRegistration must be written out to the metadata image
[KAFKA-15888] - DistributedHerder log context should not use the same client ID for each Connect worker by default
[KAFKA-15890] - Consumer.poll with long timeout unaware of assigned partitions
[KAFKA-15904] - Downgrade tests are failing with directory.id
[KAFKA-15932] - Flaky test - PlaintextConsumerTest.testSeek("kraft+kip-848","consumer")
[KAFKA-15946] - AsyncKafkaConsumer should retry commits on the application thread instead of auto-retry
[KAFKA-15965] - Test failure: org.apache.kafka.common.requests.BrokerRegistrationRequestTest
[KAFKA-15967] - Fix revocation in reconcilation logic
[KAFKA-15978] - New consumer sends OffsetCommit with empty member ID
[KAFKA-15986] - New consumer group protocol integration test failures
[KAFKA-16005] - ZooKeeper to KRaft migration rollback missing disabling controller and migration configuration on brokers
[KAFKA-16012] - Incomplete range assignment in consumer
[KAFKA-16015] - kafka-leader-election timeout values always overwritten by default values
[KAFKA-16017] - Checkpointed offset is incorrect when task is revived and restoring
[KAFKA-16029] - Investigate cause of "Unable to find FetchSessionHandler for node X" in logs
[KAFKA-16046] - Stream Stream Joins fail after restoration with deserialization exceptions
[KAFKA-16078] - Be more consistent about getting the latest MetadataVersion
[KAFKA-16085] - remote copy lag bytes/segments metrics don't update all topic value
[KAFKA-16094] - BrokerRegistrationRequest.logDirs field must be ignorable
[KAFKA-16101] - KRaft migration rollback documentation is incorrect
[KAFKA-16120] - Fix partition reassignment during ZK migration
[KAFKA-16131] - Repeated UnsupportedVersionException logged when running Kafka 3.7.0-RC2 KRaft cluster with metadata version 3.6
[KAFKA-16133] - Commits during reconciliation always time out
[KAFKA-16141] - StreamsStandbyTask##test_standby_tasks_rebalanceArguments:{ “metadata_quorum”: “ISOLATED_KRAFT”, “use_new_coordinator”: false} fails consistently in 3.7
[KAFKA-16144] - Controller leader checkQuorum timer should skip only 1 controller case
[KAFKA-16157] - Topic recreation with offline disk doesn't update leadership/shrink ISR correctly
[KAFKA-16162] - New created topics are unavailable after upgrading to 3.7
[KAFKA-16216] - Reduce batch size for initial metadata load during ZK migration
[KAFKA-16221] - IllegalStateException from Producer

Task

[KAFKA-14438] - Throw error when consumer configured with empty/whitespace-only group.id for AsyncKafkaConsumer
[KAFKA-15093] - Add 3.5.0 to broker/client and streams upgrade/compatibility tests
[KAFKA-15378] - Rolling upgrade system tests are failing
[KAFKA-15479] - Remote log segments should be considered once for retention breach
[KAFKA-15535] - Add documentation of "remote.log.index.file.cache.total.size.bytes" configuration property.
[KAFKA-15661] - KIP-951: Server side and protocol changes
[KAFKA-15664] - Add 3.4.0 streams upgrade/compatibility tests
[KAFKA-15714] - KRaft support in DynamicNumNetworkThreadsTest
[KAFKA-15725] - KRaft support in FetchRequestTest
[KAFKA-15730] - KRaft support in ProducerFailureHandlingTest
[KAFKA-15731] - KRaft support in ListOffsetsIntegrationTest
[KAFKA-15732] - KRaft support in LogAppendTimeTest
[KAFKA-15733] - KRaft support in FetchRequestMaxBytesTest
[KAFKA-15765] - Remove task level metric "commit-latency"

Test

[KAFKA-15169] - Add tests for RemoteIndexCache
[KAFKA-15179] - Add integration tests for the FileStream Sink and Source connectors
[KAFKA-15439] - Add transaction tests enabled with tiered storage
[KAFKA-15547] - Thread leak in MirrorMakerConfigTest#testClientConfigProperties
[KAFKA-15659] - Flaky test RestoreIntegrationTest.shouldInvokeUserDefinedGlobalStateRestoreListener
[KAFKA-15671] - Flaky test RemoteIndexCacheTest.testClearCacheAndIndexFilesWhenResizeCache
[KAFKA-15780] - Wait for consistent kraft metadata when creating topics in tests
[KAFKA-15793] - Flaky test ZkMigrationIntegrationTest.testMigrateTopicDeletions
[KAFKA-15991] - Flaky new consumer test testGroupIdNotNullAndValid
[KAFKA-16035] - add integration test for ExpiresPerSec and RemoteLogSizeComputationTime metrics
[KAFKA-16139] - StreamsUpgradeTest fails consistently in 3.7.0

Sub-task

[KAFKA-14247] - Implement EventHandler interface and DefaultEventHandler for Consumer
[KAFKA-14252] - Create background thread skeleton for new Consumer threading model
[KAFKA-14274] - Introduce FetchRequestManager to integrate fetch into new consumer threading refactor
[KAFKA-14481] - Move LogSegment/LogSegments to storage module
[KAFKA-14499] - Implement OffsetFetch/OffsetCommit APIs
[KAFKA-14502] - Implement LeaveGroup API
[KAFKA-14503] - Implement ListGroups
[KAFKA-14515] - Implement optimized uniform broker side assignor
[KAFKA-14516] - Implement static membeship
[KAFKA-14519] - Add metrics to the new coordinator
[KAFKA-14581] - Move GetOffsetShell to tools
[KAFKA-14593] - Move LeaderElectionCommand to tools
[KAFKA-14595] - Move ReassignPartitionsCommand to tools
[KAFKA-14596] - Move TopicCommand to tools
[KAFKA-14675] - Extract metadata-related tasks from Fetcher into MetadataFetcher
[KAFKA-14684] - Replace EasyMock and PowerMock with Mockito in WorkerSinkTaskThreadedTest
[KAFKA-14724] - Port tests in FetcherTest to FetchRequestManagerTest
[KAFKA-14875] - Implement Wakeup()
[KAFKA-14960] - Metadata Request Manager and listTopics/partitionsFor API
[KAFKA-14965] - Introduce OffsetsRequestManager to integrate ListOffsets requests into new consumer threading refactor
[KAFKA-14987] - Implement Group/Offset expiration
[KAFKA-15026] - Implement min-cost flow balancing tasks for same subtopology
[KAFKA-15061] - CoordinatorPartitionWriter should reuse buffer
[KAFKA-15081] - Implement new consumer offsetsForTimes
[KAFKA-15115] - Implement resetPositions functionality in OffsetsRequestManager
[KAFKA-15158] - Add metrics for RemoteDeleteRequestsPerSec, RemoteDeleteErrorsPerSec, BuildRemoteLogAuxStateRequestsPerSec, BuildRemoteLogAuxStateErrorsPerSec
[KAFKA-15163] - Implement validatePositions functionality for new KafkaConsumer
[KAFKA-15164] - Extract reusable logic from OffsetsForLeaderEpochClient
[KAFKA-15174] - Ensure the correct thread is executing the callbacks
[KAFKA-15184] - New consumer internals refactoring and clean up
[KAFKA-15188] - Implement more of the remaining PrototypeAsyncConsumer APIs
[KAFKA-15200] - verify pre-requisite at start of release.py
[KAFKA-15237] - Implement write operation timeout
[KAFKA-15275] - Implement consumer group membership state machine
[KAFKA-15276] - Implement event plumbing for ConsumerRebalanceListener callbacks
[KAFKA-15277] - Design & implement support for internal Consumer delegates
[KAFKA-15278] - Implement client support for KIP-848 ConsumerGroupHeartbeat protocol RPC
[KAFKA-15280] - Implement client support for selecting KIP-848 server-side assignor
[KAFKA-15281] - Implement the groupMetadata Consumer API
[KAFKA-15304] - CompletableApplicationEvents aren't being completed when the consumer is closing
[KAFKA-15306] - Integrate committed offsets logic when updating fetching positions
[KAFKA-15316] - CommitRequestManager not calling RequestState callbacks
[KAFKA-15317] - Fix for async consumer access to committed offsets with multiple consumers
[KAFKA-15327] - Client consumer should commit offsets on close
[KAFKA-15346] - Single-Key_single-timestamp IQs with versioned state stores
[KAFKA-15347] - Single-Key_multi-timestamp IQs with versioned state stores
[KAFKA-15355] - Message schema changes
[KAFKA-15358] - QueuedReplicaToDirAssignments metric
[KAFKA-15365] - Broker-side replica management changes
[KAFKA-15366] - Log directory failure integration test
[KAFKA-15438] - Review exception caching logic used for reset/validate positions in async consumer
[KAFKA-15454] - Add support for OffsetCommit version 9 in admin client
[KAFKA-15455] - Add support for OffsetCommit version 9 in consumer
[KAFKA-15456] - Add support for OffsetFetch version 9 in consumer
[KAFKA-15457] - Add support for OffsetFetch version 9 in admin
[KAFKA-15461] - Add integration test for the ListGroup API
[KAFKA-15484] - Implement general uniform broker side assignor
[KAFKA-15508] - Method always return the same value ApplicationEventProcessor.java
[KAFKA-15531] - Ensure coordinator node is removed upon disconnection exception
[KAFKA-15533] - Ensure HeartbeatRequestManager only send out some fields once
[KAFKA-15534] - Propagate client response time when timeout to the request handler
[KAFKA-15544] - Enable existing client integration tests for new protocol
[KAFKA-15548] - Send GroupConsumerHeartbeatRequest on Consumer.close()
[KAFKA-15550] - OffsetsForTimes validation for negative timestamps in new consumer
[KAFKA-15554] - Update client state machine to align with protocol sending one assignment at a time
[KAFKA-15555] - Ensure wakeups are handled correctly in PrototypeAsyncConsumer.poll()
[KAFKA-15562] - Ensure fetch offset and commit offset handler handles both timeout and various error types
[KAFKA-15574] - Update states and transitions for membership manager state machine
[KAFKA-15582] - Clean shutdown detection, broker side
[KAFKA-15593] - Add 3.6.0 to broker/client upgrade/compatibility tests
[KAFKA-15594] - Add 3.6.0 to streams upgrade/compatibility tests
[KAFKA-15598] - Add integration tests for DescribeGroups API, DeleteGroups API and OffsetDelete API
[KAFKA-15604] - Add Telemetry RPCs Definitions
[KAFKA-15613] - Add Client API in interfaces and client configurations.
[KAFKA-15614] - Define interfaces/classes for capturing telemetry metrics
[KAFKA-15616] - Define client telemetry states and their transitions
[KAFKA-15618] - Implement MetricsCollector which collects metrics in OTLP format
[KAFKA-15628] - Refactor ConsumerRebalanceListener invocation for reuse
[KAFKA-15633] - Bug: Generated Persistent Directory IDs are overwritten on startup.
[KAFKA-15643] - Improve unloading logging
[KAFKA-15645] - Move ReplicationQuotasTestRig to tools
[KAFKA-15647] - Fix the different behavior in error handling between the old and new group coordinator
[KAFKA-15663] - Implement ClientTelemetryReporter which manages telemetry lifecyclye
[KAFKA-15668] - Add Opentelemetry Proto library with shadowed classes
[KAFKA-15669] - Implement telemetry naming strategy
[KAFKA-15673] - Define new client_metrics resource type configuration to store metric config.
[KAFKA-15679] - Client support for new consumer configs
[KAFKA-15681] - Add support of client-metrics in kafka-configs.sh
[KAFKA-15684] - Add support to describe all subscriptions through utility
[KAFKA-15696] - Revoke partitions on Consumer.close()
[KAFKA-15705] - Add integration tests for Heartbeat API and GroupLeave API
[KAFKA-15775] - Implement listTopics() and partitionFor() for the AsyncKafkaConsumer
[KAFKA-15778] - Implement ClientMetricsManager to process request
[KAFKA-15779] - Define broker configurations and exceptions
[KAFKA-15794] - Implement request and response handling for telemetry APIs
[KAFKA-15803] - Update last seen epoch during commit
[KAFKA-15818] - Implement max poll interval
[KAFKA-15830] - Add request/response handling in KafkaApis and update metrics plugin
[KAFKA-15833] - Restrict Consumer API to be used from one thread
[KAFKA-15842] - Correct handling of KafkaConsumer.committed for new consumer
[KAFKA-15849] - Fix ListGroups API when runtime partition size is zero
[KAFKA-15854] - Move Java classes from kafka.server to the server module
[KAFKA-15856] - Add integration tests for JoinGroup API and SyncGroup API
[KAFKA-15865] - Ensure consumer.poll() execute autocommit callback
[KAFKA-15870] - Move new group coordinator metrics from Yammer to Metrics
[KAFKA-15871] - Implement kafka-client-metrics.sh tool
[KAFKA-15874] - Add metric and request log attribute for deprecated request api versions (KIP-896)
[KAFKA-15879] - Add documentation and examples for the Docker image
[KAFKA-15880] - Add github actions workflow for promoting RC docker image
[KAFKA-15881] - Make changes in Release Process Wiki and Release Process
[KAFKA-15883] - Implement RemoteCopyLagBytes
[KAFKA-15887] - Autocommit during close consistently fails with exception in background thread
[KAFKA-15893] - Bump MetadataVersion for directory assignments
[KAFKA-15901] - Implement client changes to support telemetry APIs
[KAFKA-15903] - Add github actions workflow for building and pushing RC Docker Image
[KAFKA-15910] - New group coordinator needs to generate snapshots while loading
[KAFKA-15913] - Migrate AsyncKafkaConsumerTest away from ConsumerTestBuilder
[KAFKA-15953] - Refactor polling delays
[KAFKA-15981] - update Group size only when groups size changes
[KAFKA-16002] - Implement RemoteCopyLagSegments, RemoteDeleteLagBytes and RemoteDeleteLagSegments
[KAFKA-16013] - Implement ExpiresPerSec metric
[KAFKA-16014] - Implement RemoteLogSizeComputationTime, RemoteLogSizeBytes, RemoteLogMetadataCount
[KAFKA-16026] - AsyncConsumer does not send a poll event to the background thread
[KAFKA-16030] - new group coordinator should check if partition goes offline during load
[KAFKA-16036] - Add `group.coordinator.rebalance.protocols` and publish all new configs
[KAFKA-16040] - Rename `Generic` to `Classic`
[KAFKA-16053] - Fix leaked Default DirectoryService
[KAFKA-16058] - Fix leaked in ControllerApiTest
[KAFKA-16059] - Fix leak of ExpirationReaper-1-AlterAcls threads in :core:test
[KAFKA-16062] - Upgrade mockito to 5.8.0
[KAFKA-16065] - Fix leak in DelayedOperationTest
[KAFKA-16074] - Fix thread leaks in ReplicaManagerTest
[KAFKA-16079] - Fix leak in LocalLeaderEndPointTest/FinalizedFeatureChangeListenerTest/KafkaApisTest/ReplicaManagerConcurrencyTest
[KAFKA-16099] - Handle timeouts for AsyncKafkaConsumer.commitSync
[KAFKA-16118] - Coordinator unloading fails when replica is deleted