如何抓取简书、博客园及CSDN个人主页目录

本文将介绍如何使用Python编写爬虫脚本，抓取简书、博客园及CSDN等平台用户个人主页的文章目录。基于实际使用需求，代码注释详细，直接阅读即可理解实现逻辑。

简书个人主页目录抓取

代码概述

# -*- coding:utf-8 -*-import urllib.requestfrom lxml import etreeclass CrawlJs:    def getArticle(self, url):        print('开始爬取数据')        headers = {            'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.104 Safari/537.36'        }        request = urllib.request.Request(url, headers=headers)        content = urllib.request.urlopen(request).read()        return content    def save(self, content):        xml = etree.HTML(content)        title = xml.xpath('//div[@class="content"]/a[@class="title"]/text()')        link = xml.xpath('//div[@class="content"]/a[@class="title"]/@href')        for t, l in zip(title, link):            with open('JsIndex.txt', 'a') as f:                f.write(f'[{t}]({l})\n')        print('爬取完成')if __name__ == '__main__':    page = int(input('请输入要抓取的页码总数：'))    for num in range(page):        url = f'http://www.jianshu.com/u/c475403112ce?order_by=shared_at&page={num + 1}'        js = CrawlJs()        content = js.getArticle(url)        js.save(content)

代码功能说明

getArticle函数：负责向指定URL发送请求，获取网页内容

save函数：解析获取的网页内容，提取文章标题和链接，并将数据保存到本地文件中

主程序：接收用户输入的页码总数，循环爬取每一页面的内容并保存

博客园个人主页目录抓取

# -*- coding:utf-8 -*-import urllib2from lxml import etreeclass CrawlJs:    def getArticle(self, url):        print('开始爬取数据')        headers = {            'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.104 Safari/537.36'        }        request = urllib2.Request(url, headers=headers)        content = urllib2.urlopen(request).read()        return content    def save(self, content):        xml = etree.HTML(content)        title = xml.xpath('//div[@class="postTitle"]/a/text()')        link = xml.xpath('//div[@class="postTitle"]/a/@href')        for t, l in zip(title, link):            with open('bokeyuan.txt', 'a') as f:                f.write(f'{t} {l}\n')        print('爬取完成')if __name__ == '__main__':    page = int(input('请输入要抓取的页码总数：'))    for num in range(page):        url = f'http://www.cnblogs.com/zhouxinfei/default.html?page={num + 1}'        js = CrawlJs()        content = js.getArticle(url)        js.save(content)

代码特点

URL请求处理：使用urllib2库发送请求，模拟浏览器访问

HTML解析：通过lxml.etree库解析网页内容，提取需要的文章信息

文件保存：将爬取到的文章标题和链接保存到本地文本文件中

CSDN个人文章目录抓取

# -*- coding:utf-8 -*-import urllib.requestfrom lxml import etreeclass CrawlJs:    def getArticle(self, url):        print('开始爬取数据')        headers = {            'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.104 Safari/537.36'        }        request = urllib.request.Request(url, headers=headers)        content = urllib.request.urlopen(request).read()        return content    def save(self, content):        xml = etree.HTML(content)        title = xml.xpath('//div[@class="article-list"]/div/h4/a/text()[2]')        link = xml.xpath('//div[@class="article-list"]/div/h4/a/@href')        if not title:            return        for t, l in zip(title, link):            with open('csdn.txt', 'a') as f:                f.write(f'{t.strip()} {l}\n')        print('爬取完成')if __name__ == '__main__':    page = int(input('请输入要抓取的页码总数：'))    for num in range(page):        url = f'https://blog.csdn.net/xc_zhou/article/list/{num + 1}'        js = CrawlJs()        content = js.getArticle(url)        js.save(content)