提问者:小点点

Hefty Streaming - kafka.internals.Handover$ClosedException


我正在使用Apache Flink v1.12.3。

最近我遇到了这个错误,我不知道它到底是什么意思。错误与 Kafka 或 Flink 有关吗?

错误日志:

2021-07-02 21:32:50,149 WARN  org.apache.kafka.clients.consumer.internals.ConsumerCoordinator [] - [Consumer clientId=consumer-myGroup-24, groupId=myGroup] Offset commit failed on partition my_topic-14 at offset 11328862986: The request timed out.
// ...
2021-07-02 21:32:50,150 INFO  org.apache.kafka.clients.consumer.internals.AbstractCoordinator [] - [Consumer clientId=consumer-myGroup-24, groupId=myGroup] Group coordinator 1.2.3.4:9092 (id: 2147483616 rack: null) is unavailable or invalid, will attempt rediscovery

// ...

2021-07-02 21:33:20,553 INFO  org.apache.kafka.clients.FetchSessionHandler                 [] - [Consumer clientId=consumer-myGroup-21, groupId=myGroup] Error sending fetch request (sessionId=1923351260, epoch=9902) to node 29: {}.

// ...

2021-07-02 21:33:19,457 INFO  org.apache.kafka.clients.FetchSessionHandler                 [] - [Consumer clientId=consumer-myGroup-15, groupId=myGroup] Error sending fetch request (sessionId=1427185435, epoch=10820) to node 29: {}.
org.apache.kafka.common.errors.DisconnectException: null

// ...

2021-07-02 21:34:10,157 WARN  org.apache.flink.runtime.taskmanager.Task                    [] - Source: my_topic_stream (4/15)#0 (2e2051d41edd606a093625783d844ba1) switched from RUNNING to FAILED.
org.apache.flink.streaming.connectors.kafka.internals.Handover$ClosedException: null
    at org.apache.flink.streaming.connectors.kafka.internals.Handover.close(Handover.java:177) ~[blob_p-a7919582483974414f9c0d4744bab53199b880d7-d9edc9d0741b403b3931269bf42a4f6b:?]
    at org.apache.flink.streaming.connectors.kafka.internals.KafkaFetcher.cancel(KafkaFetcher.java:164) ~[blob_p-a7919582483974414f9c0d4744bab53199b880d7-d9edc9d0741b403b3931269bf42a4f6b:?]
    at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.cancel(FlinkKafkaConsumerBase.java:945) ~[blob_p-a7919582483974414f9c0d4744bab53199b880d7-d9edc9d0741b403b3931269bf42a4f6b:?]
    at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.lambda$createAndStartDiscoveryLoop$2(FlinkKafkaConsumerBase.java:913) ~[blob_p-a7919582483974414f9c0d4744bab53199b880d7-d9edc9d0741b403b3931269bf42a4f6b:?]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]

共1个答案

匿名用户

这是Kafka的问题。Kafka 消费者客户端在将偏移量提交到 Kafka 集群时抛出错误(超时)。一个可能的原因是 Kafka 集群繁忙,无法及时响应。此错误使运行 Kafka 使用者的任务管理器失败。

尝试在从 Kafka 创建源流时向属性添加参数。可能的参数为:request.timeout.ms,将其设置为更长的时间,然后尝试。

引用:

  • https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/connectors/datastream/kafka/#kafka-consumer
  • https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html#consumerconfigs_request.timeout.ms