Discussion:
how to set timestamp field in trident window?
Add Reply
731635762
2018-11-23 03:51:04 UTC
Reply
Permalink
Hi All, I use storm-core 1.2.2, and there is my code:


spoutStream.window(TumblingDurationWindow.of(new BaseWindowedBolt.Duration(10, TimeUnit.SECONDS)),
new Fields("value"), new CountAsAggregator(), new Fields("record"));


I can't find any API to specify window timestamp field.



Can anyone tell me how to set timestamp field in trident TumblingDurationWindow?





Any help is appreciated. Thanks,
ChenBo
Stig Rohde Døssing
2018-11-23 11:10:17 UTC
Reply
Permalink
I don't believe you can. The Trident windowing code seems to use
System.currentTimeMillis without allowing you to set a timestamp extractor.
If you want to add this functionality, you should look at
https://github.com/apache/storm/blob/21bb1388414d373572779289edc785c7e5aa52aa/storm-client/src/jvm/org/apache/storm/trident/windowing/InMemoryTridentWindowManager.java#L53,
and likely also the other implementation of ITridentWindowManager, and make
them use the windowManager.add(event, ts) method instead of
windowManager.add(event). You can consider using
https://github.com/apache/storm/blob/21bb1388414d373572779289edc785c7e5aa52aa/storm-client/src/jvm/org/apache/storm/topology/WindowedBoltExecutor.java#L310
for reference to how it works in regular Storm topologies.

Note that there doesn't seem to be a watermark event generator for the
Trident code either, so you may want to look at adding that as well.
Post by 731635762
spoutStream.window(TumblingDurationWindow.of(new
BaseWindowedBolt.Duration(10, TimeUnit.SECONDS)),
new Fields("value"), new CountAsAggregator(), new
Fields("record"));
I can't find any API to specify window timestamp field.
Can anyone tell me how to set timestamp field in trident
TumblingDurationWindow?
Any help is appreciated.
Thanks,
ChenBo
Stig Rohde Døssing
2018-11-23 11:11:48 UTC
Reply
Permalink
Also you should be aware of
https://issues.apache.org/jira/browse/STORM-3280 if you intend to use
Trident based windowing.

Den fre. 23. nov. 2018 kl. 12.10 skrev Stig Rohde DÞssing <
Post by Stig Rohde Døssing
I don't believe you can. The Trident windowing code seems to use
System.currentTimeMillis without allowing you to set a timestamp extractor.
If you want to add this functionality, you should look at
https://github.com/apache/storm/blob/21bb1388414d373572779289edc785c7e5aa52aa/storm-client/src/jvm/org/apache/storm/trident/windowing/InMemoryTridentWindowManager.java#L53,
and likely also the other implementation of ITridentWindowManager, and make
them use the windowManager.add(event, ts) method instead of
windowManager.add(event). You can consider using
https://github.com/apache/storm/blob/21bb1388414d373572779289edc785c7e5aa52aa/storm-client/src/jvm/org/apache/storm/topology/WindowedBoltExecutor.java#L310
for reference to how it works in regular Storm topologies.
Note that there doesn't seem to be a watermark event generator for the
Trident code either, so you may want to look at adding that as well.
Post by 731635762
spoutStream.window(TumblingDurationWindow.of(new
BaseWindowedBolt.Duration(10, TimeUnit.SECONDS)),
new Fields("value"), new CountAsAggregator(), new
Fields("record"));
I can't find any API to specify window timestamp field.
Can anyone tell me how to set timestamp field in trident
TumblingDurationWindow?
Any help is appreciated.
Thanks,
ChenBo
ChenBo
2018-11-26 02:27:31 UTC
Reply
Permalink
Strange problem, I can't receive reply of my question to email list. I read ansewer from strom website. Thanks for
Stig Rohde DÞssing. That's very helpful.




------------------ Original ------------------
From: "ChenBo"<***@qq.com>;
Date: Fri, Nov 23, 2018 11:51 AM
To: "user"<***@storm.apache.org>;

Subject: how to set timestamp field in trident window?



Hi All, I use storm-core 1.2.2, and there is my code:


spoutStream.window(TumblingDurationWindow.of(new BaseWindowedBolt.Duration(10, TimeUnit.SECONDS)),
new Fields("value"), new CountAsAggregator(), new Fields("record"));


I can't find any API to specify window timestamp field.



Can anyone tell me how to set timestamp field in trident TumblingDurationWindow?





Any help is appreciated. Thanks,
ChenBo

Loading...