红联Linux门户
Linux帮助

Apache MRUnit 1.1发布,MapReduce单元测试

发布时间:2014-06-09 09:06:01来源:红联作者:empast
Apache MRUnit 1.1 发布了,MRUnit是由Couldera公司开发的专门针对 Hadoop中 编写MapReduce单元测试的框架,基本原理是JUnit4和 EasyMock。MR就是Map和Reduce的缩写。MRUnit框架非常精简,其核心的单元测试依赖于JUnit。而且MRUnit实现了一套 Mock对象来控制OutputCollector的操作,从而可以拦截OutputCollector的输出,和我们的期望结果进行比较,达到自动断言 的目的。

有了MRUnit,对MR程序做重构的时候,只要明确输入和输出,就可以写出单元测试,并且在放到群集校验前进行试验,从而节省时间和资源,也 能更快的定位到问题。而进行重构的话,只要写得足够详细的单元测试都是绿色的话,那么基本就可以保证在群集运行的结果也是正常的。

改进记录包括:

Bug
[MRUNIT-129] - Key object re-use in Reducer is inconsistent with MapReduce behaviour (new API)

[MRUNIT-160] - DistributedCache unit tests fail on Mac OS X 10.8.2

[MRUNIT-185] - IllegalFormatException for '%' in Output

[MRUNIT-186] - method context.getNumReduceTasks() does not work

[MRUNIT-188] - Throw assertion error if drivers are reused

[MRUNIT-189] - MapDriverBase.withInput comments state same as setInput but uses addInput

[MRUNIT-190] - Use mvn enforcer plugin to ensure that a non-vulnerable version of java is used for javadoc generation

[MRUNIT-192] - MRUNIT-180 requires values be comparable

[MRUNIT-193] - Serialization.copy throws NPE instead of ISE (missing serilization impl) for Hadoop 2.x

[MRUNIT-198] - Serialization is missing in MockMultipleOutputs

[MRUNIT-199] - Mapper.Context and Reducer.Context are not fully mocked

[MRUNIT-203] - tarballs should extract to dirs with the same name as tarball

[MRUNIT-204] - Jira and Wiki tutorial are outdated

Dependency upgrade
[MRUNIT-209] - Can't use the maven dependency plugin with maven 3 on mrunit

Improvement
[MRUNIT-164] - Remove duplicated InputPathStoringMapper classes

[MRUNIT-180] - Being able to use a comparator on runTest

[MRUNIT-182] - TemporaryPath should support test data in subdirectories

[MRUNIT-194] - Update versions of Hadoop we compile against

[MRUNIT-208] - mrunit unnecessarily depends on mockito-all

New Feature
[MRUNIT-13] - Add support for MultipleOutputs

[MRUNIT-162] - MapReduce testing utilties

[MRUNIT-205] - Add support for MultipleInputs

Task
[MRUNIT-201] - Update list of committers everywhere (pom/site)

[MRUNIT-207] - Test failure on trunk-hadoop1 branch

项目主页:http://mrunit.apache.org/

下载地址:http://mrunit.apache.org/general/downloads.html

来自:开源中国社区
文章评论

共有 0 条评论