红联Linux门户
Linux帮助

Scrapy 1.0发布,Web爬虫框架

发布时间:2015-06-30 15:02:10来源:红联作者:empast
Scrapy 1.0 发布,该版本有多项新的特性和 bug 修复,部分内容如下:

新特性和增强

Python logging (:issue:`1060`, :issue:`1235`, :issue:`1236`, :issue:`1240`,:issue:`1259`, :issue:`1278`, :issue:`1286`)

FEED_EXPORT_FIELDS option (:issue:`1159`, :issue:`1224`)

Dns cache size and timeout options (:issue:`1132`)

support namespace prefix in xmliter_lxml (:issue:`963`)

Reactor threadpool max size setting (:issue:`1123`)

Allow spiders to return dicts. (:issue:`1081`)

Add Response.urljoin() helper (:issue:`1086`)

look in ~/.config/scrapy.cfg for user config (:issue:`1098`)

handle TLS SNI (:issue:`1101`)

Selectorlist extract first (:issue:`624`, :issue:`1145`)

Added JmesSelect (:issue:`1016`)

add gzip compression to filesystem http cache backend (:issue:`1020`)

CSS support in link extractors (:issue:`983`)

httpcache dont_cache meta #19 #689 (:issue:`821`)

add signal to be sent when request is dropped by the scheduler (:issue:`961`)

avoid download large response (:issue:`946`)

Allow to specify the quotechar in CSVFeedSpider (:issue:`882`)

Add referer to "Spider error processing" log message (:issue:`795`)

process robots.txt once (:issue:`896`)

GSoC Per-spider settings (:issue:`854`)

Add project name validation (:issue:`817`)

GSoC API cleanup (:issue:`816`, :issue:`1128`, :issue:`1147`,:issue:`1148`, :issue:`1156`, :issue:`1185`, :issue:`1187`, :issue:`1258`,:issue:`1268`, :issue:`1276`, :issue:`1285`, :issue:`1284`)

Be more responsive with IO operations (:issue:`1074` and :issue:`1075`)

Do leveldb compaction for httpcache on closing (:issue:`1297`)

弃用和清除:

Deprecate htmlparser link extractor (:issue:`1205`)

remove deprecated code from FeedExporter (:issue:`1155`)

a leftover for.15 compatibility (:issue:`925`)

drop support for CONCURRENT_REQUESTS_PER_SPIDER (:issue:`895`)

Drop old engine code (:issue:`911`)

Deprecate SgmlLinkExtractor (:issue:`777`)

Scrapy 是一套基于基于Twisted的异步处理框架,纯python实现的爬虫框架,用户只需要定制开发几个模块就可以轻松的实现一个爬虫,用来抓取网页内容以及各种图片,非常之方便。

软件详情:https://github.com/scrapy/scrapy/blob/af97c373eac5f6c137f632e2db844bde3abb2f49/docs/news.rst#id4

下载地址:https://github.com/scrapy/scrapy/archive/1.0.0.zip

来自:开源中国社区
文章评论

共有 0 条评论