scrapy 爬虫，为异步io框架;因此此处选择，先用requests请求，保存cookie文件，然后scrapy爬取前，在入口处加载cookie。
* 登录，保存cookie方法见前两节，此处展示的是scrapy读取cookie
* 首先要明确，scrapy框架关于登录的一些基础知识
1. scrapy 爬取有一个入口，在入口处加载cookie，后续的爬取会自动加载cookie信息
2. scrapy cookie的加载时接收字典形式的，因此，和requests加载cookie不同，要先转化为字典

加载cookie文件，并转化为字典形式

#!/usr/bin/env python3.6
# -*- coding: utf-8 -*-
# @Time    : 2018/4/18 18:10
# @Author  : ysj
import os
import http.cookiejar as cookielib
import requests
from scrapy_article.utl.ZhiHuLogIn import ZhiHu


cookie_file = os.path.join(os.path.dirname(__file__), 'zhihu_cookie.txt')


def load_cookie(filename=cookie_file):
    """加载cookie文件"""

    cookie = cookielib.LWPCookieJar()
    cookie.load(filename, ignore_discard=True, ignore_expires=True)
    return requests.utils.dict_from_cookiejar(cookie)


def get_dict_cookie(account='18516157608', password='******'):
    """" 调用前一节中的登录，"""
    login = ZhiHu(account, password)
    return requests.utils.dict_from_cookiejar(login.session.cookies)


if __name__ == '__main__':
    print(load_cookie())
    print(get_dict_cookie('18516157608', '*****'))

scrapy 入口加载cookie

主要是重写scrapy入口请求，添加cookie和headers
篇幅问题，此处指显示了重写的入口函数和一个登录是否成功的验证函数；需要提前加载上文load_cookie函数

class ZhihuSpider(scrapy.Spider):
    name = 'zhihu'
    allowed_domains = ['www.zhihu.com']
    start_urls = ['https://www.zhihu.com/']

    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36',
        'HOST': 'www.zhihu.com',
        'Referer': 'https://www.zhihu.com/signin?next=%2F',
        'Authorization': 'oauth c3cef7c66a1843f8b3a9e6a1e3160e20'
    }
    login_url = 'https://www.zhihu.com/api/v3/oauth/sign_in'
    captcha_url = 'https://www.zhihu.com/api/v3/oauth/captcha?lang=en'
    check_url = 'https://www.zhihu.com/inbox'
    # check_url = 'https://www.zhihu.com/'
    def start_requests(self):
        self.post_data['signature'] = self.get_signature()

        return [scrapy.Request(url=self.check_url, headers=self.headers, method='GET', meta={
   'dont_redirect': True},
                               cookies=load_cookie(), callback=self.check_login)]

    def check_login(self, response):
        """用requests登录过，会保存cookie，传入scrapy使用，暂时不考虑scrapy登录"""
        if response.status < 300:
            print('登录成功')
            for url in self.start_urls:
                yield scrapy.Request(url, dont_filter=True, headers=self.headers)
        else:
            print('未登录,即将拉取验证码，重新登录')
            yield [scrapy.Request(url=self.check_url, headers=self.headers, method='GET', meta={
   'dont_redirect': True},
                                  cookies=get_dict_cookie(), callback=self.check_login)]