Scrapy简介

Scrapy是Python开发的一个快速,高层次的屏幕抓取和Web抓取框架，用于抓取Web站点并从页面中提取结构化的数据。

实例

新建工程

这次我们来用爬虫获取豆瓣电影Top 250的电影信息吧。开始之前，我们新建一个Scrapy工程。

1	scrapy startproject doubanmoive

这些文件主要为：

items.py：定义需要获取的内容字段，类似于实体类。
pipelines.py：项目流水线文件，用来处理Spider抓取的数据。
settings.py：项目配置文件
spiders：放置spider的目录

定义项目(Item)

Item 是保存爬取到的数据的容器；其使用方法和python字典类似，并且提供了额外保护机制来避免拼写错误导致的未定义字段错误。

# -*- coding: utf-8 -*-
# Define here the models for your scraped items
#
# See documentation in:
# http://doc.scrapy.org/en/latest/topics/items.html
import scrapy
class DoubanmoiveItem(scrapy.Item):
    name = scrapy.Field() #电影名
    year = scrapy.Field() #上映年份
    score = scrapy.Field() #豆瓣分数
    director = scrapy.Field() #导演
    classification = scrapy.Field() #分类
    actor = scrapy.Field() #演员

编写爬虫(Spider)

Spider是整个项目中最核心的类，在这个类里我们会定义抓取对象（域名、URL）以及抓取规则。Scrapy官方文档中的教程是基于BaseSpider的，但BaseSpider只能爬取给定的URL列表，无法根据一个初始的URL向外拓展。不过除了BaseSpider，还有很多可以直接继承Spider的类，比如scrapy.contrib.spiders.CrawlSpider。

在doubanmoive/spiders目录下新建moive_spider.py文件，并填写代码。

# -*- coding: utf-8 -*-
import scrapy
from scrapy.selector import Selector
from scrapy.contrib.spiders import CrawlSpider,Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from tutorial.items import DoubanmoiveItem
class MoiveSpider(CrawlSpider):
    name="doubanmoive"
    allowed_domains=["movie.douban.com"]
    start_urls=["http://movie.douban.com/top250"]
    rules=[
        Rule(SgmlLinkExtractor(allow=(r'http://movie.douban.com/top250\?start=\d+.*'))),
        Rule(SgmlLinkExtractor(allow=(r'http://movie.douban.com/subject/\d+')),callback="parse_item"),      
    ]
    def parse_item(self,response):
        sel=Selector(response)
        item=DoubanmoiveItem()
        
        item['name']=sel.xpath('//*[@id="content"]/h1/span[1]/text()').extract()
        item['year']=sel.xpath('//*[@id="content"]/h1/span[2]/text()').re(r'\((\d+)\)')
        item['score']=sel.xpath('//*[@id="interest_sectl"]/div[1]/p[1]/strong/text()').extract()
        item['director']=sel.xpath('//*[@id="info"]/span[1]/span[2]/a/text()').extract()
        item['classification']= sel.xpath('//span[@property="v:genre"]/text()').extract() 
        item['actor']= sel.xpath('//a[@rel="v:starring"]/text()').extract()
        print "debug------------------"
        print 'name:',
        print item['name'][0]
        
        print 'year:',
        print item['year'][0]
        print 'score:',
        print item['score'][0]
        print 'director:',
        print item['director'][0]
        print 'classification:'
        for cls in item['classification']:
            print cls
        print 'actor:'
        for actor in item['actor']:
            print actor 
        
        return item

存储数据

爬虫获取到数据以后我们需要将其存储到数据库中，之前我们提到该操作需要靠项目管道（pipeline）来处理。

# -*- coding: utf-8 -*-
from scrapy import log
from twisted.enterprise import adbapi
from scrapy.http import Request
import MySQLdb
import MySQLdb.cursors
# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: http://doc.scrapy.org/en/latest/topics/item-pipeline.html
class DoubanmoivePipeline(object):
    
    def __init__(self):
        self.dbpool = adbapi.ConnectionPool('MySQLdb',
                db = 'python',
                user = 'root',
                passwd = 'toor',
                cursorclass = MySQLdb.cursors.DictCursor,
                charset = 'utf8',
                use_unicode = False
        )
    def process_item(self, item, spider):
        query = self.dbpool.runInteraction(self._conditional_insert, item)
        query.addErrback(self.handle_error)
        return item
    def _conditional_insert(self,tx,item):
        tx.execute("select * from doubanmoive where m_name= %s",(item['name'][0],))
        result=tx.fetchone()
        log.msg(result,level=log.DEBUG)
        print result
        if result:
            log.msg("Item already stored in db:%s" % item,level=log.DEBUG)
        else:
            classification=actor=''
            lenClassification=len(item['classification'])
            lenActor=len(item['actor'])
            for n in xrange(lenClassification):
                classification+=item['classification'][n]
                if n<lenClassification-1:
                    classification+='/'
            for n in xrange(lenActor):
                actor+=item['actor'][n]
                if n<lenActor-1:
                    actor+='/'
            tx.execute(\
                "insert into doubanmoive (m_name,m_year,m_score,m_director,m_classification,m_actor) values (%s,%s,%s,%s,%s,%s)",\
                (item['name'][0],item['year'][0],item['score'][0],item['director'][0],classification,actor))
            log.msg("Item stored in db: %s" % item, level=log.DEBUG)
    def handle_error(self, e):
        log.err(e)

配置文件

在运行爬虫之前还需要将在settings.py中增加一些配置信息。

ITEM_PIPELINES={
    'tutorial.pipelines.DoubanmoivePipeline':400,
}
LOG_LEVEL='DEBUG'
DOWNLOAD_DELAY = 2
RANDOMIZE_DOWNLOAD_DELAY = True
USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.54 Safari/536.5'
COOKIES_ENABLED = True