红联Linux门户
Linux帮助

Onyx 0.8.0发布,分布式计算系统

发布时间:2015-11-14 09:16:53来源:红联作者:empast
Onyx 0.8.0 发布,此版本最主要是增加了自动化状态管理功能,恢复,Windowing 和触发器。状态会被记录在 BookKeeper 日志记录,segments 可以使用嵌入式 RocksDB 删除处理。开箱支持多个类型的聚合。

升级注意事项:

Onyx 0.8.0 不兼容之前的版本

改进列表:

Breaking change :onyx.messaging/peer-port-range and :onyx.messaging/peer-ports are deprecated in favour of a single :onyx.messaging/peer-port port. The Aeron layer multiplexed all communication over a single port so multiple port selection is longer required.

Important change: default task batch timeout was reduced from 1000ms to 50ms.

New major feature: Windowing and Triggers.

New feature: Fixed windows

New feature: Sliding windows

New feature: Session windows

New feature: Global windows

New feature: Timer triggers

New feature: Segment-based triggers

New feature: Watermark triggers

New feature: Percentile Watermark triggers

New feature: Punctuation triggers

New feature: Accumulating refinement mode

New feature: Discarding refinement mode

New feature: BookKeeper & embedded RocksDB automatic "exactly once" filtering.

New feature: friendlier error messages. Added custom schema error handlers to print out documentation and required types in a more customized format.

New flux policy: Recover

New lifecycle: after-read-batch

New metric: peer-send-bytes

Bug fix: fixed an issue where the percentage job scheduler would misallocate.

Bug fix: fixed :data key being removed from exception messages in the logs.

Onyx 是什么?

Onyx 是一个无中心、支持云、容错的分布式计算系统

使用 Clojure 编写

支持批处理和流处理混合

提供信息模型用于描述和构建分布式工作流

竞争对手: Storm, Cascading, Map/Reduce, Dryad, Apache Sqoop, Twitter Crane 等等

用途?

实时的事件流处理

连续计算

ETL

在 MapReduce 进行数据转换

数据清理

数据摄入和存储媒介传输

软件详情:http://michaeldrogalis.github.io/jekyll/update/2015/11/12/Onyx-0.8.0-Automatic-State-Management.html

下载地址:https://github.com/onyx-platform/onyx

来自:开源中国社区
文章评论

共有 0 条评论