ºìÁªLinuxÃÅ»§
Linux°ïÖú

Pholcus£¨ÓÄÁéÖ룩ÅÀ³æÈí¼þv1.0Îȶ¨°æÕýʽ·¢²¼

·¢²¼Ê±¼ä:2016-06-16 14:52:38À´Ô´:ºìÁª×÷Õß:baihuo
×Ô2015Äê5Ô Pholcus ·¢²¼µÚÒ»¸ö°æ±¾ÆðÀú¾­Ò»ÄêµÄ°æ±¾µü´ú£¬½ñÈÕ v1.0 Îȶ¨°æ±¾ÖÕÓÚÕýʽ·¢²¼ÁË¡£Ôڴ˸ÐлËùÓйØ×¢¹ýPholcus¡¢·´À¡¹ý¿ª·¢½¨ÒéÒÔ¼°ÔËÐÐbugµÄÅóÓÑÃÇ¡£v1.0 °æ±¾ÊÇÒ»¸ö¾­¹ýÖî¶àÓû§Ê¹ÓÃÓëÏîÄ¿ÑéÖ¤¹ýµÄÎȶ¨°æ±¾£¬´ó¼Ò¿ÉÒÔ·ÅÐÄÏÂÔØÊ¹Óá£Í¬Ê±ÐèҪ˵Ã÷µÄÊÇ£¬´ËǰijЩÓû§Ìá³öµÄ²¿·Ö¿ª·¢½¨Ò鲢δ¼ÓÈëv1.0ÖУ¬µ«½«»áÔÚÖ®ºóµÄ°æ±¾Öð²½ÊµÏÖ¡£

PholcusµÄ¼ò½é£º

Pholcus£¨ÓÄÁéÖ룩ÊÇÒ»¿î´¿GoÓïÑÔ±àдµÄ¸ß²¢·¢¡¢·Ö²¼Ê½¡¢ÖØÁ¿¼¶ÅÀ³æÈí¼þ£¬Ö§³Öµ¥»ú¡¢·þÎñ¶Ë¡¢¿Í»§¶ËÈýÖÖÔËÐÐģʽ£¬ÓµÓÐWeb¡¢GUI¡¢ÃüÁîÐÐÈýÖÖ²Ù×÷½çÃæ£»¹æÔò¼òµ¥Áé»î¡¢ÅúÁ¿ÈÎÎñ²¢·¢¡¢Êä³ö·½Ê½·á¸»£¨mysql/mongodb/csv/excelµÈ£©¡¢ÓдóÁ¿Demo¹²Ïí£»Í¬Ê±Ëý»¹Ö§³Öºá×ÝÏòÁ½ÖÖץȡģʽ£¬Ö§³ÖÄ£ÄâµÇ¼ºÍÈÎÎñÔÝÍ£¡¢È¡ÏûµÈһϵÁи߼¶¹¦ÄÜ¡£

Ïà¶ÔÉÏÒ»°æ±¾£¨v0.8.5£©µÄ¸üÐÂÏêÇ飺

Ò»¡¢Êä³öÄ£¿é

1.Ö§³Ö¹æÔòÖÐÉèÖÃSpider.NotDefaultField=trueÀ´½ûÖ¹Êä³öϵͳĬÈÏÌí¼ÓµÄ×Ö¶Î Url¡¢ParentUrl ºÍ DownloadTime£»

2. ¼õÉÙmysqlµÄ±í´´½¨²Ù×÷£»

3. ÐÞ¸´mysql±íÃûºÍ×Ö¶ÎÄÚÈÝÔÚijЩÇé¿öÏ»á³öÏÖÂÒÂëµÄÏÖÏó£»

4. ÐÞ¸´LinuxÏ£¬Îļþ´´½¨È¨ÏÞÎÊÌ⣻

5. ÐÞ¸´Êä³öÎļþµÄ·¾¶ÒÔ"\"·Ö¸îʱ£¬Ä¿Â¼´´½¨Ê§Ð§µÄbug£»

6. ÐÞ¸´Output()·½·¨Ö¸¶¨ruleNameÎÞЧµÄbug£»

7. Ö±½ÓÏÂÔØÎļþʱ£¬ÔÚÈÕÖ¾ÖдòÓ¡Îļþ´óС¡£

¶þ¡¢ÄÚ´æÓÅ»¯

1.DateCell¡¢FileCell¡¢Context¾ùÔö¼ÓPool£¬¼õÉÙ¶ÔÏó´´½¨£»

2. ÿ2·ÖÖÓ¼ì²éÒ»´Î¿ÉÖØÓÃÄڴ棬Èô´óÓÚ50M£¬ÔòÊÖ¶¯ÊÍ·ÅÒ»´ÎÄڴ棻

3. ¼ò»¯½á¹ûÊý¾Ýת´¢Á÷³Ì£¬Ò»¶¨³Ì¶È½µµÍÄÚ´æÏûºÄ¡£

Èý¡¢¶¯Ì¬¹æÔò

1.ÐÞ¸´½âÎö¶à¸ö¶¯Ì¬¹æÔòʱµÄ±Õ°üÎÊÌ⣻

2. ÐÞ¸´¶¯Ì¬¹æÔòÖÐÉèÖÃHeaderÎÞЧµÄbug¡£

ËÄ¡¢ÈÕ־ģ¿é

1.ÔÚÅäÖÃÎļþÖÐÔö¼Ó[log]¶ÎÂä²ÎÊý£º

2. level -> È«¾ÖÈÕÖ¾´òÓ¡¼¶±ð£¨ÒàÊÇÈÕÖ¾ÎļþÊä³ö¼¶±ð£©£»

3. consolelevel -> ÈÕÖ¾ÔÚ¿ØÖÆÌ¨µÄÏÔʾ¼¶±ð£»

4. feedbacklevel -> ¿Í»§¶Ë·´À¡ÖÁ·þÎñ¶ËµÄÈÕÖ¾¼¶±ð£»

5. lineinfo -> ÈÕÖ¾ÊÇ·ñ´òÓ¡ÐÐÐÅÏ¢£»

6. save -> ÊÇ·ñ±£´æËùÓÐÈÕÖ¾µ½±¾µØÎļþ¡£

Îå¡¢ÅäÖÃÄ£¿é

1. ÅäÖÃÎļþÖжÎÂäÓëÅäÖÃÏî·Ö±ð°´×ÖĸÅÅÐò£»

2. ÅäÖÃÎļþÖÐ mysqlmaxallowedpacket -> mysqlmaxallowedpacketmb£¬µ¥Î»ÎªMB£»

3. mysqlÔö¼Ómax_allowed_packetµÄÅäÖã»

4. Ôö¼ÓconngcsecondÅäÖã¬Ö§³Ö×ÔÖ÷É趨mgoÁ¬½Ó³ØGCʱ¼ä¡£

Áù¡¢²Ù×÷½çÃæ

1. cmd°æµÄ·þÎñ¶Ëģʽ֧³ÖÁ¬ÐøÌí¼ÓÈÎÎñ¡®

2. ÐÞ¸´web°æ³¤Ê±¼äÔËÐУ¬ÒòÒâÍⶪʧcookie¶ø¿ÉÄÜÒý·¢panicµÄÎÊÌâ¡®

3. ÐÞ¸´web°æ´óÁ¿ÈÕÖ¾Êä³öʱ¡°send on closed channel¡°µÄbug¡£

Æß¡¢ÆäËû

1. Ôö¼Ó¶ÔÖ§³ÖgzipµÄÒ³Ãæ½øÐÐ×Ô¶¯½âÂëµÄ¹¦ÄÜ£»

2. ÐÞ¸´CrawlPool.Use()ÓëCrawlPool.Free()µÄд¾ºÕù£»

3. µ÷ÕûÔËÐÐÒÀÀµÄ¿Â¼£»

4. Éý¼¶surferÏÂÔØÆ÷¡£

Èí¼þÏêÇ飺http://pholcus.gitbooks.io/docs

ÏÂÔØµØÖ·£ºhttps://github.com/henrylee2cn/pholcus

À´×Ô:¿ªÔ´ÖйúÉçÇø
ÎÄÕÂÆÀÂÛ

¹²ÓÐ 0 ÌõÆÀÂÛ