python - 关于Scrapy中TwistedPipline报错
问题描述
> python3.5 / pycharm
crawlspider 返回的 item 是绝对没有问题的,接上TwistedPipline就报错 请哪位老师帮忙分析下报错代码
# TwistedPiplineclass MysqlTwistedPipline(object): def __init__(self, dbpool):self.dbpool = dbpool @classmethod def from_settings(cls, settings):dbparms = dict( host = settings['MYSQL_HOST'], db = settings['MYSQL_DBNAME'], user = settings['MYSQL_USER'], passwd = settings['MYSQL_PASSWORD'], charset=’utf8’, cursorclass=MySQLdb.cursors.DictCursor, use_unicode=True,)dbpool = adbapi.ConnectionPool('MySQLdb', **dbparms)return cls(dbpool) def process_item(self, item, spider):query = self.dbpool.runInteraction(self.do_insert, item)query.addErrback(self.handle_error, item, spider) def handle_error(self, failure, item, spider):print (failure) def do_insert(self, cursor, item):insert_sql, params = item.insert_values()print (insert_sql, params)cursor.execute(insert_sql, params)return item
Error code
2017-05-01 00:06:16 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.lagou.com/jobs/2108656.html> (referer: https://www.lagou.com/zhaopin/Python/)2017-05-01 00:06:16 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.lagou.com/jobs/2108656.html>None[Failure instance: Traceback: <class ’TypeError’>: connect() argument 3 must be str, not NoneD:Python35libthreading.py:914:_bootstrap_innerD:Python35libthreading.py:862:runD:Python35libsite-packagestwisted_threads_threadworker.py:46:workD:Python35libsite-packagestwisted_threads_team.py:190:doWork--- <exception caught here> ---D:Python35libsite-packagestwistedpythonthreadpool.py:250:inContextD:Python35libsite-packagestwistedpythonthreadpool.py:266:<lambda>D:Python35libsite-packagestwistedpythoncontext.py:122:callWithContextD:Python35libsite-packagestwistedpythoncontext.py:85:callWithContextD:Python35libsite-packagestwistedenterpriseadbapi.py:464:_runInteractionD:Python35libsite-packagestwistedenterpriseadbapi.py:36:__init__D:Python35libsite-packagestwistedenterpriseadbapi.py:76:reconnectD:Python35libsite-packagestwistedenterpriseadbapi.py:431:connectD:Python35libsite-packagesMySQLdb__init__.py:86:ConnectD:Python35libsite-packagesMySQLdbconnections.py:204:__init__]
问题解答
回答1:从提示看出,你的配置(settings)的时候有1个参数是None,而不是字符串
相关文章:
1. python - 关于matplotlib的x轴显示的问题2. mysql优化 - mysql慢查询copying to tmp table3. android 文件File删除问题4. javascript - vue生成一维码?求助!!!!!急5. ueditor上传服务器提示后端配置项没有正常加载,求助!!!!!6. css - .clearfix:after中为什么设置display: table7. 请教: 关于 python 反斜杠转义的疑问8. css - transform: translateY(-50%)在360浏览器极速模式下使得文字变模糊了9. angular.js - 怎样在使用ng-repeat属性的标签里面监听单个事件,使其能够单个改变CSS。10. nginx英文文档的WebSocket proxying部分没看太明白,麻烦推荐一点中文文章
