Discussion:
Missing kafkaOffset metric in storm-kafka-client trident spout
Add Reply
Jonathan Munz
2018-11-23 19:54:34 UTC
Reply
Permalink
Hi,

I was looking to upgrade from 1.0.5 to 1.2.2. One thing I noticed is that
the 'kafkaOffset' metric is no longer being reported by the Kafka Trident
spout. In 1.0.5 it was being registered here:
https://github.com/apache/storm/blob/v1.0.5/external/storm-kafka/src/jvm/org/apache/storm/kafka/trident/TridentKafkaEmitter.java#L60

But in 1.2.2. that's no longer the case:
https://github.com/apache/storm/blob/v1.2.2/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/trident/KafkaTridentSpoutEmitter.java#L90

I don't see any mention in the PR that added the new implementation of the
Kafka Trident spout about why this was removed:
https://github.com/apache/storm/pull/1687

Was there a particular reason for this? Would you have any recommendations
about how to get this back or alternative ways to get at the same metric?

Thanks!
Stig Rohde Døssing
2018-11-24 00:11:11 UTC
Reply
Permalink
Your first link points to storm-kafka, while your second link points to
storm-kafka-client. Storm-kafka-client is intended as a replacement for
storm-kafka, but the two codebases are unrelated. The metric is missing
because no one has implemented metrics for the new Trident spout yet.
There's an implementation for the regular spout at
https://github.com/apache/storm/blob/master/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/metrics/KafkaOffsetMetric.java,
so if you'd like to add metrics to the Trident spout it would be a good
starting point to look at that class.
Post by Jonathan Munz
Hi,
I was looking to upgrade from 1.0.5 to 1.2.2. One thing I noticed is that
the 'kafkaOffset' metric is no longer being reported by the Kafka Trident
https://github.com/apache/storm/blob/v1.0.5/external/storm-kafka/src/jvm/org/apache/storm/kafka/trident/TridentKafkaEmitter.java#L60
https://github.com/apache/storm/blob/v1.2.2/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/trident/KafkaTridentSpoutEmitter.java#L90
I don't see any mention in the PR that added the new implementation of the
https://github.com/apache/storm/pull/1687
Was there a particular reason for this? Would you have any recommendations
about how to get this back or alternative ways to get at the same metric?
Thanks!
Jonathan Munz
2018-11-26 15:13:10 UTC
Reply
Permalink
Okay I'll take a look that as a starting point, thank you!
Post by Stig Rohde Døssing
Your first link points to storm-kafka, while your second link points to
storm-kafka-client. Storm-kafka-client is intended as a replacement for
storm-kafka, but the two codebases are unrelated. The metric is missing
because no one has implemented metrics for the new Trident spout yet.
There's an implementation for the regular spout at
https://github.com/apache/storm/blob/master/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/metrics/KafkaOffsetMetric.java,
so if you'd like to add metrics to the Trident spout it would be a good
starting point to look at that class.
Post by Jonathan Munz
Hi,
I was looking to upgrade from 1.0.5 to 1.2.2. One thing I noticed is that
the 'kafkaOffset' metric is no longer being reported by the Kafka Trident
https://github.com/apache/storm/blob/v1.0.5/external/storm-kafka/src/jvm/org/apache/storm/kafka/trident/TridentKafkaEmitter.java#L60
https://github.com/apache/storm/blob/v1.2.2/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/trident/KafkaTridentSpoutEmitter.java#L90
I don't see any mention in the PR that added the new implementation of
https://github.com/apache/storm/pull/1687
Was there a particular reason for this? Would you have any
recommendations about how to get this back or alternative ways to get at
the same metric?
Thanks!
Loading...