kafka是一种高吞吐量的分布式发布订阅消息系统,她有如下特性:
通过O(1)的磁盘数据结构提供消息的持久化,这种结构对于即使数以TB的消息存储也能够保持长时间的稳定性能。
高吞吐量:即使是非常普通的硬件kafka也可以支持每秒数十万的消息。
支持通过kafka服务器和消费机集群来分区消息。
支持Hadoop并行数据加载。
卡夫卡的目的是提供一个发布订阅解决方案,它可以处理消费者规模的网站中的所有动作流数据。 这种动作(网页浏览,搜索和其他用户的行动)是在现代网络上的许多社会功能的一个关键因素。 这些数据通常是由于吞吐量的要求而通过处理日志和日志聚合来解决。 对于像Hadoop的一样的日志数据和离线分析系统,但又要求实时处理的限制,这是一个可行的解决方案。kafka的目的是通过Hadoop的并行加载机制来统一线上和离线的消息处理,也是为了通过集群机来提供实时的消费。
2014-03-11发布0.8.1 上个版本是2013-12-10的0.8.0
完全改进:Release Notes - Kafka - Version 0.8.1
New Feature
[KAFKA-330] - Add delete topic support
[KAFKA-554] - Move all per-topic configuration into ZK and add to the CreateTopicCommand
[KAFKA-615] - Avoid fsync on log segment roll
[KAFKA-657] - Add an API to commit offsets
[KAFKA-925] - Add optional partition key override in producer
[KAFKA-1092] - Add server config parameter to separate bind address and ZK hostname
[KAFKA-1117] - tool for checking the consistency among replicas
Improvement
[KAFKA-347] - change number of partitions of a topic online
[KAFKA-561] - Rebuild index file for a log segment if there is none
[KAFKA-918] - Change log.retention.hours to be log.retention.mins
[KAFKA-995] - Enforce that the value for replica.fetch.max.bytes is always >= the value for message.max.bytes
[KAFKA-1001] - Handle follower transition in batch
[KAFKA-1084] - Validate properties for custom serializers
[KAFKA-1127] - kafka and zookeeper server should start in daemon mode and log to correct position
[KAFKA-1131] - copy some more files into the release tar and zip that are needed/desired
[KAFKA-1136] - Add subAppend in Log4jAppender for generic usage
[KAFKA-1158] - remove bin/run-rat.sh
[KAFKA-1186] - Add topic regex to the kafka-topics tool
[KAFKA-1232] - make TopicCommand more consistent
Task
[KAFKA-823] - merge 0.8 (51421fcc0111031bb77f779a6f6c00520d526a34) to trunk
[KAFKA-896] - merge 0.8 (988d4d8e65a14390abd748318a64e281e4a37c19) to trunk
[KAFKA-965] - merge c39d37e9dd97bf2462ffbd1a96c0b2cb05034bae from 0.8 to trunk
[KAFKA-1051] - merge from 0.8 da4512174b6f7c395ffe053a86d2c6bb19d2538a to trunk
Sub-task
[KAFKA-121] - pom should include standard maven niceties
[KAFKA-1244] - The LICENSE and NOTICE are missing from the jar files
[KAFKA-1245] - the jar files and pom are not being signed so nexus is failing to publish them
[KAFKA-1246] - The 2.10 version is showing up as 2.10.1
[KAFKA-1249] - release tar name is different than 0.8.0
[KAFKA-1254] - remove vestigial sbt
[KAFKA-1274] - gradle.properties needs the variables used in the build.gradle
Bug
[KAFKA-184] - Log retention size and file size should be a long
[KAFKA-515] - Log cleanup can close a file channel opnened by Log.read before the transfer completes
[KAFKA-648] - Use uniform convention for naming properties keys
[KAFKA-671] - DelayedProduce requests should not hold full producer request data
[KAFKA-739] - Handle null values in Message payload
[KAFKA-759] - Commit/FetchOffset APIs should not return versionId
[KAFKA-852] - Remove clientId from OffsetFetchResponse and OffsetCommitResponse
[KAFKA-897] - NullPointerException in ConsoleConsumer
[KAFKA-930] - Integrate preferred replica election logic into kafka
[KAFKA-933] - Hadoop example running DataGenerator causes kafka.message.Message cannot be cast to [B exception
[KAFKA-985] - Increasing log retention quickly overflows scala Int
[KAFKA-1004] - Handle topic event for trivial whitelist topic filters
[KAFKA-1009] - DumpLogSegments tool should return error on non-existing files
[KAFKA-1020] - Remove getAllReplicasOnBroker from KafkaController
[KAFKA-1036] - Unable to rename replication offset checkpoint in windows
[KAFKA-1052] - integrate add-partitions command into topicCommand
[KAFKA-1055] - BrokerTopicStats is updated before checking for MessageSizeTooLarge
[KAFKA-1060] - Break-down sendTime into responseQueueTime and the real sendTime
[KAFKA-1074] - Reassign partitions should delete the old replicas from disk
[KAFKA-1090] - testPipelinedRequestOrdering has transient failures
[KAFKA-1091] - full topic list can be read from metadata cache in the broker instead of ZK
[KAFKA-1097] - Race condition while reassigning low throughput partition leads to incorrect ISR information in zookeeper
[KAFKA-1103] - Consumer uses two zkclients
[KAFKA-1104] - Consumer uses two zkclients
[KAFKA-1112] - broker can not start itself after kafka is killed with -9
[KAFKA-1121] - DumpLogSegments tool should print absolute file name to report inconsistencies
[KAFKA-1128] - Github is still showing 0.7 as the default branch
[KAFKA-1133] - LICENSE and NOTICE files need to get into META-INF when jars are built before they're signed for publishing to maven
[KAFKA-1135] - Code cleanup - use Json.encode() to write json data to zk
[KAFKA-1140] - Move the decoding logic from ConsumerIterator.makeNext to next
[KAFKA-1141] - make changes to downloads for the archive old releases to new old_releases folder
[KAFKA-1151] - The Hadoop consumer API doc is not referencing the contrib consumer
[KAFKA-1152] - ReplicaManager's handling of the leaderAndIsrRequest should gracefully handle leader == -1
[KAFKA-1154] - replicas may not have consistent data after becoming follower
[KAFKA-1156] - Improve reassignment tool to output the existing assignment to facilitate rollbacks
[KAFKA-1157] - Clean up Per-topic Configuration from Kafka properties
[KAFKA-1164] - kafka should depend on snappy 1.0.5 (instead of 1.0.4.1)
[KAFKA-1168] - OfflinePartitionCount in JMX can be incorrect during controlled shutdown
[KAFKA-1169] - missing synchronization in access to leaderCache in KafkaApis
[KAFKA-1170] - ISR can be inconsistent during partition reassignment for low throughput partitions
[KAFKA-1198] - NullPointerException in describe topic
[KAFKA-1200] - inconsistent log levels when consumed offset is reset
[KAFKA-1202] - optimize ZK access in KafkaController
[KAFKA-1205] - README in examples not update
[KAFKA-1208] - Update system test still to use kafka-topics instead of kafka-create-topics shell
[KAFKA-1228] - Socket Leak on ReplicaFetcherThread
[KAFKA-1243] - Gradle issues for release
[KAFKA-1263] - Snazzy up the README markdown for better visibility on github
[KAFKA-1271] - controller logs exceptions during ZK session expiration
[KAFKA-1275] - fixes for quickstart documentation
[KAFKA-1288] - add enclosing dir in release tar gz
[KAFKA-1289] - Misc. nitpicks in log cleaner
下载:http://apache.org/dist/kafka/0.8.1/kafka-0.8.1-src.tgz
来自:oschina开源中国社区

