[07]抓取简书|博客园|CSDN个人主页制作目录
本文共 6433 字，大约阅读时间需要 21 分钟。
文章目录
抓取简书个人主页制作目录
本文代码运行环境pyhton2，代码注释的很详细，直接看代码即可。
#-*- coding:utf-8 -*-import urllib2from lxml import etreeclass CrawlJs(): #定义函数，爬取对应的数据 def getArticle(self,url): print '█████████████◣开始爬取数据' my_headers = { 'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.104 Safari/537.36', } request = urllib2.Request(url,headers=my_headers) content = urllib2.urlopen(request).read() return content #定义函数，筛选和保存爬取到的数据 def save(self,content): xml = etree.HTML(content) title = xml.xpath('//div[@class="content"]/a[@class="title"]/text()') link = xml.xpath('//div[@class="content"]/a[@class="title"]/@href') print link i=-1 for data in title: print data i+=1 with open('JsIndex.txt','a+') as f: f.write('['+data.encode('utf-8')+']'+'('+'http://www.jianshu.com'+link[i]+')'+ '\n') print '█████████████◣爬取完成！'#定义主程序接口if __name__ == '__main__': page = int(raw_input('请输入你要抓取的页码总数：')) for num in range(page): #这里输入个人主页，如：u/c475403112ce url = 'http://www.jianshu.com/u/c475403112ce?order_by=shared_at&page=%s'%(num+1) #调用上边的函数 js = CrawlJs() #获取页面内容 content = js.getArticle(url) #保存内容到文本中 js.save(content) 运行结果
python3代码
#-*- coding:utf-8 -*-import urllib.requestfrom lxml import etreeclass CrawlJs(): #定义函数，爬取对应的数据 def getArticle(self,url): print ('█████████████◣开始爬取数据') my_headers = { 'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.104 Safari/537.36', } request = urllib.request.Request(url,headers=my_headers) content = urllib.request.urlopen(request).read() return content #定义函数，筛选和保存爬取到的数据 def save(self,content): xml = etree.HTML(content) title = xml.xpath('//div[@class="content"]/a[@class="title"]/text()') link = xml.xpath('//div[@class="content"]/a[@class="title"]/@href') print (link) i=-1 for data in title: print (data) i+=1 with open('JsIndex.txt','a+') as f: f.write('['+data+']'+'('+'http://www.jianshu.com'+link[i]+')'+ '\n') print ('█████████████◣爬取完成！')#定义主程序接口if __name__ == '__main__': page = int(input('请输入你要抓取的页码总数：')) for num in range(page): #这里输入个人主页，如：u/c475403112ce url = 'http://www.jianshu.com/u/c475403112ce?order_by=shared_at&page=%s'%(num+1) js = CrawlJs() content = js.getArticle(url) js.save(content) 抓取博客园个人主页制作目录
python2代码
#-*- coding:utf-8 -*-import urllib2from lxml import etreeclass CrawlJs(): #定义函数，爬取对应的数据 def getArticle(self,url): print '█████████████◣开始爬取数据' my_headers = { 'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.104 Safari/537.36', } request = urllib2.Request(url,headers=my_headers) content = urllib2.urlopen(request).read() return content #定义函数，筛选和保存爬取到的数据 def save(self,content): xml = etree.HTML(content) title = xml.xpath('//*[@class="postTitle"]/a/text()') link = xml.xpath('//*[@class="postTitle"]/a/@href') print (title,link) # print(zip(title,link)) # print(map(lambda x,y:[x,y], title,link)) for t,li in zip(title,link): print(t+li) with open('bokeyuan.txt','a+') as f: f.write(t.encode('utf-8')+li+ '\n') print '█████████████◣爬取完成！'#定义主程序接口if __name__ == '__main__': page = int(raw_input('请输入你要抓取的页码总数：')) for num in range(page): #这里输入个人主页， url = 'http://www.cnblogs.com/zhouxinfei/default.html?page=%s'%(num+1) js = CrawlJs() content = js.getArticle(url) js.save(content) python3代码
#-*- coding:utf-8 -*-import urllib.requestfrom lxml import etreeclass CrawlJs(): #定义函数，爬取对应的数据 def getArticle(self,url): print ('█████████████◣开始爬取数据') my_headers = { 'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.104 Safari/537.36', } request = urllib.request.Request(url,headers=my_headers) content = urllib.request.urlopen(request).read() return content #定义函数，筛选和保存爬取到的数据 def save(self,content): xml = etree.HTML(content) title = xml.xpath('//*[@class="postTitle"]/a/text()') link = xml.xpath('//*[@class="postTitle"]/a/@href') print (title,link) # print(zip(title,link)) # print(map(lambda x,y:[x,y], title,link)) for t,li in zip(title,link): print(t+li) with open('bokeyuan.txt','a+') as f: f.write(t+' '+li+ '\n') print('█████████████◣爬取完成！')#定义主程序接口if __name__ == '__main__': page = int(input('请输入你要抓取的页码总数：')) for num in range(page): #这里输入个人主页， url = 'http://www.cnblogs.com/zhouxinfei/default.html?page=%s'%(num+1) js = CrawlJs() content = js.getArticle(url) js.save(content) CSDN个人目录制作
#-*- coding:utf-8 -*-import urllib.requestfrom lxml import etreeclass CrawlJs(): #定义函数，爬取对应的数据 def getArticle(self,url): print ('█████████████◣开始爬取数据') my_headers = { 'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.104 Safari/537.36', } request = urllib.request.Request(url,headers=my_headers) content = urllib.request.urlopen(request).read() return content #定义函数，筛选和保存爬取到的数据 def save(self,content): xml = etree.HTML(content) title = xml.xpath('//div[@class="article-list"]/div/h4/a/text()[2]') link = xml.xpath('//div[@class="article-list"]/div/h4/a/@href') if title==None: return # print(map(lambda x,y:[x,y], title,link)) for t,li in zip(title,link): print(t+li) with open('csdn.txt','a+') as f: f.write(t.strip()+' '+li+ '\n') print('█████████████◣爬取完成！')#定义主程序接口if __name__ == '__main__': page = int(input('请输入你要抓取的页码总数：')) for num in range(page): #这里输入个人主页， url = 'https://blog.csdn.net/xc_zhou/article/list/%s'%(num+1) js = CrawlJs() content = js.getArticle(url) js.save(content)
你可能感兴趣的文章
MySQL集群解决方案（5）：PXC集群
查看>>
MySQL面试宝典
查看>>
WAP短信：融合传统短信和互联网的新型通信方式
查看>>
mysql面试题学校三表查询_mysql三表查询分组后取每组最大值，mysql面试题。
查看>>
Mysql面试题精选
查看>>
MySQL面试题集锦
查看>>
mysql面试题，存储引擎InnoDB和MyISAM
查看>>
mysql面试题：为什么MySQL单表不能超过2000W条数据？
查看>>
mysql面试题：创建索引时会不会锁表？
查看>>
mysql面试题：高度为3的B+树可以存放多少数据？
查看>>
mysql颠覆实战笔记(八)--mysql的自定义异常处理怎么破
查看>>
mysql驱动、durid、mybatis之间的关系
查看>>
mysql驱动支持中文_mysql 驱动包-Go语言中文社区
查看>>
MySQL高可用之——keepalived+互为主从
查看>>
MySQL高可用切换_（5.9）mysql高可用系列——正常主从切换测试
查看>>
MySQL高可用解决方案
查看>>
MySQL高可用解决方案详解
查看>>
MYSQL高可用集群MHA架构
查看>>
MySQL高可用集群架构MHA企业级实战
查看>>
MySQL高级-MySQL存储引擎
查看>>