如何抓取简书、博客园及CSDN个人主页目录

本文将介绍如何使用Python编写爬虫脚本，抓取简书、博客园及CSDN等平台用户个人主页的文章目录。基于实际使用需求，代码注释详细，直接阅读即可理解实现逻辑。

简书个人主页目录抓取

代码概述

# -*- coding:utf-8 -*-
import urllib.request
from lxml import etree
class CrawlJs:
    def getArticle(self, url):
        print('开始爬取数据')
        headers = {
            'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.104 Safari/537.36'
        }
        request = urllib.request.Request(url, headers=headers)
        content = urllib.request.urlopen(request).read()
        return content
    def save(self, content):
        xml = etree.HTML(content)
        title = xml.xpath('//div[@class="content"]/a[@class="title"]/text()')
        link = xml.xpath('//div[@class="content"]/a[@class="title"]/@href')
        for t, l in zip(title, link):
            with open('JsIndex.txt', 'a') as f:
                f.write(f'[{t}]({l})\n')
        print('爬取完成')
if __name__ == '__main__':
    page = int(input('请输入要抓取的页码总数：'))
    for num in range(page):
        url = f'http://www.jianshu.com/u/c475403112ce?order_by=shared_at&page={num + 1}'
        js = CrawlJs()
        content = js.getArticle(url)
        js.save(content)

代码功能说明

getArticle函数：负责向指定URL发送请求，获取网页内容

save函数：解析获取的网页内容，提取文章标题和链接，并将数据保存到本地文件中

主程序：接收用户输入的页码总数，循环爬取每一页面的内容并保存

博客园个人主页目录抓取

# -*- coding:utf-8 -*-
import urllib2
from lxml import etree
class CrawlJs:
    def getArticle(self, url):
        print('开始爬取数据')
        headers = {
            'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.104 Safari/537.36'
        }
        request = urllib2.Request(url, headers=headers)
        content = urllib2.urlopen(request).read()
        return content
    def save(self, content):
        xml = etree.HTML(content)
        title = xml.xpath('//div[@class="postTitle"]/a/text()')
        link = xml.xpath('//div[@class="postTitle"]/a/@href')
        for t, l in zip(title, link):
            with open('bokeyuan.txt', 'a') as f:
                f.write(f'{t} {l}\n')
        print('爬取完成')
if __name__ == '__main__':
    page = int(input('请输入要抓取的页码总数：'))
    for num in range(page):
        url = f'http://www.cnblogs.com/zhouxinfei/default.html?page={num + 1}'
        js = CrawlJs()
        content = js.getArticle(url)
        js.save(content)

代码特点

URL请求处理：使用urllib2库发送请求，模拟浏览器访问

HTML解析：通过lxml.etree库解析网页内容，提取需要的文章信息

文件保存：将爬取到的文章标题和链接保存到本地文本文件中

CSDN个人文章目录抓取

# -*- coding:utf-8 -*-
import urllib.request
from lxml import etree
class CrawlJs:
    def getArticle(self, url):
        print('开始爬取数据')
        headers = {
            'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.104 Safari/537.36'
        }
        request = urllib.request.Request(url, headers=headers)
        content = urllib.request.urlopen(request).read()
        return content
    def save(self, content):
        xml = etree.HTML(content)
        title = xml.xpath('//div[@class="article-list"]/div/h4/a/text()[2]')
        link = xml.xpath('//div[@class="article-list"]/div/h4/a/@href')
        if not title:
            return
        for t, l in zip(title, link):
            with open('csdn.txt', 'a') as f:
                f.write(f'{t.strip()} {l}\n')
        print('爬取完成')
if __name__ == '__main__':
    page = int(input('请输入要抓取的页码总数：'))
    for num in range(page):
        url = f'https://blog.csdn.net/xc_zhou/article/list/{num + 1}'
        js = CrawlJs()
        content = js.getArticle(url)
        js.save(content)