红联Linux门户
Linux帮助

JStorm 2.2.1发布,分布式计算系统

发布时间:2017-01-10 21:29:38来源:红联作者:uineqa
JStorm 2.2.1 发布,更新内容如下:

新功能

Performance is improved by 200%~300%, compared to Release 2.1.1 and 0.9.8.1 in several testing scenarios, while 120%~200% compared to Flink and 300%~400% compared to Storm.

Restructure the batch solution

Improve serialization and deserialization to reduce the cost of cpu and network

Improve the cost of cpu on critical path and metrics

Improve the strategy of netty client and netty server

Support consume and publish of disruptor queue under batch mode

Introduce snapshot exactly once framework

Compared to Trident solution, the performance of new framework is increased by several times. Besides it.

The new framework also support "at least once" mode. Compared to the acker mechanism,it will reduce the cost of relative calculation in acker, and the cost of network, which will improve the performance singificantly.

Support JStorm on yarn

Currently, jstorm cluster is capable of fast deployments,and fast scale-in/scale-out. It will improve the utility of resource.

Re-design the solution of backpressure. Currently, the flow control is stage by stage。

The solution is simple and effective now. The response is much more faster when the exchange of switch on/off of backpressure.

The performance and stability is improved significantly, compared to the original solution.

Introduce Window API

Support tumbling window,sliding window

window support two collection mode, count and duration.

Support watermark mechanism

Introduce the support of Flux

Flux is a programing framework or component which is aim to help create and deploy the topology of jstorm quickly.

Isolate the dependencies of jstorm and user topology by maven shade plugin to fix the conflict problem.

Improve Shuffle grouping solution

Integrate shuffle, localOrShuffle and localFirst. The grouping solution will be auto adapted according to the assignment of topology.

Introduce load aware in shuffle to ensure the load balance of downstreams.

Support to configure blacklist in Nimbus to exclude some problematic nodes

Support batch mode in trident

Supervisors will synchronize cluster configuration from nimbus master automatically

Add buildTs to supervisor info and heartbeats

Add ext module for nimbus and supervisor to support external plugins

Add jstorm-elasticsearch support, thanks to @elloray for your contribution

改进

Restructure nimbus metrics implementation. Currently, the topology metrics runnable is event-driven.

Restructure topology master. Currently, the processor in TM is event-drive.

Add some examples to cover more scenarios

Disable stream metrics to reduce the cost of sending metrics to Nimbus

Support metrics in local mode

Improve the implementation of gauge by changing the instantaneous value of each minute,to the average value of some sample values in each minute.

Introduce an approximate histogram calculation to reduce memory usage of histogram metrics

Add Full GC and supervisor network related metrics

修复

Fix message disorder bug

Fix the bug that some connections to zookeeper are not closed by expected when encountering exception in supervisor.

The deactivate might be called by mistake when task init

The rootId might be duplicated occasionally. It will cause the unexpected message failure.

Fix the bug when local mode

Fix logwriter's bug

Some task metrics(RecvTps ProcessLatency) might not be aggregated correctly.

Fix the racing condition of AsmCounter during flushing

软件详情:https://github.com/alibaba/jstorm/

下载地址:https://github.com/alibaba/jstorm/releases/download/2.2.1/jstorm-2.2.1.zip

来自:开源中国社区
文章评论

共有 0 条评论