python - scrapy获取网页指定内容，后翻到下一页继续，固定循环次数。。问题

问题描述

import scrapyfrom movie.items import MovieItem class MeijuSpider(scrapy.Spider): name = 'meiju' allowed_domains = ['alexa.cn'] start_urls = [’www.alexa.cn/siterank’] def parse(self, response): movies = response.xpath(’//ul[@class='siterank-sitelist']/li’) for each_movie in movies:item = MovieItem()item[’name’] =each_movie.xpath(’.//p[@class='infos']’).extract()[0]yield item

代码是这样的。我想循环抓取的是：

www.alexa.cn/siterank/2www.alexa.cn/siterank/3www.alexa.cn/siterank/4.....

我看循环应该是这样的for i in range(2,10):yield scrapy.Request(’www.alexa.cn/siterank/%d’%i)，但是我不知道怎么填进去。求助

问题解答

若你範圍都確定的話，不如從start_urls 下手

start_urls = [’http://www.alexa.cn/siterank/{n}’.format(n=x) for x in range(2,10)] 回答2：

官网上有例子，关于追踪下一页，官网上面的例子用的是递归，官网的代码如下：

import scrapyclass QuotesSpider(scrapy.Spider): name = 'quotes' start_urls = [’http://quotes.toscrape.com/page/1/’, ] def parse(self, response):for quote in response.css(’p.quote’): yield {’text’: quote.css(’span.text::text’).extract_first(),’author’: quote.css(’small.author::text’).extract_first(),’tags’: quote.css(’p.tags a.tag::text’).extract(), }# next_page是用css选择器获取到的下一页， 在下面它递归地调用了parse方法来不断地追踪下一页next_page = response.css(’li.next a::attr(href)’).extract_first()if next_page is not None: next_page = response.urljoin(next_page) yield scrapy.Request(next_page, callback=self.parse)

我自己用Scrapy写了一个贴吧的爬虫，获取下一页用的也是这种递归的方法，代码如下：

import scrapyfrom tieba_crawler.items import ImageItemclass TiebaSpider(scrapy.Spider): name = ’tbimg’ def start_requests(self):url = ’http://tieba.baidu.com/f?kw=%E6%B8%A1%E8%BE%B9%E9%BA%BB%E5%8F%8B’yield scrapy.Request(url=url, callback=self.parse_post) def parse_post(self, response):post_list = response.css(’ul#thread_list li.j_thread_list’)for item in post_list: title = item.css(’a.j_th_tit::text’).extract_first() url = ’http://tieba.baidu.com’ + item.css(’a.j_th_tit::attr(href)’).extract_first() yield scrapy.Request(url=url, callback=self.parse_image)page_list = response.css(’p#frs_list_pager a::attr(href)’).extract()if not page_list: returnelse: next_page = page_list[-2] if next_page:yield response.follow(next_page, callback=self.parse_post) def parse_image(self, response):img_urls = response.css(’p#j_p_postlist img.BDE_Image::attr(src)’).extract()yield ImageItem(image_urls=img_urls)page_list = response.css(’ul.l_posts_num li.pb_list_pager a::attr(href)’).extract()if not page_list: returnelse: next_page = page_list[-2] if next_page:yield response.follow(next_page, callback=self.parse_image)