想必知乎这个网站大家都很熟悉吧,每天上午可能都会摸摸鱼来看看热榜上又有哪些热点 。但是要是明目张胆的看网站时突然发现背后站着一个领导,那时可能就会很尴尬了,哈哈!
【30行代码完成对知乎热榜的抓取!】今天我就教大家如何快速而又优雅的获取知乎热榜上的信息,只需要30行代码即可完成 。话不多说,咱们直接开始!
首先导入我们需要的库以及定义我们的,注意这里里面的要替换为自己网站上的值哦!
import requestsfrom bs4 import BeautifulSoupimport jsonimport reheaders = {'X-Requested-With': 'XMLHttpRequest','User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36','cookie': ''} #cookie值替换为自己网站上的cookie值
![30行代码完成对知乎热榜的抓取!](http://img.gansulong.com/231105/23053US1-0.png)
文章插图
之后来获取关于知乎热榜上的所有问题链接:
url = 'https://www.zhihu.com/hot'res = requests.get(url, headers=headers)soup = BeautifulSoup(res.text, 'html.parser')all_a = soup.select('div.HotItem-content a')
最后对每个问题链接进行请求,并获取该问题下的所有回答:
for a in all_a:link = a['href']print(link + a['title'])question_id = link.replace('https://www.zhihu.com/question/', '')res2 = requests.get(link, headers=headers)soup2 = BeautifulSoup(res2.text, 'html.parser')counts_content = soup2.select('h4.List-headerText')[0].get_text()counts_re = re.search('\d+\.?\d*', counts_content)counts = int(counts_re.group())print('共有{}个回答'.format(counts))for i in range(0, counts, 20):href = http://www.kingceram.com/post/'https://www.zhihu.com/api/v4/questions/{}/answers?include=data%5B%2A%5D.is_normal%2Cadmin_closed_comment%2Creward_info%2Cis_collapsed%2Cannotation_action%2Cannotation_detail%2Ccollapse_reason%2Cis_sticky%2Ccollapsed_by%2Csuggest_edit%2Ccomment_count%2Ccan_comment%2Ccontent%2Ceditable_content%2Cattachment%2Cvoteup_count%2Creshipment_settings%2Ccomment_permission%2Ccreated_time%2Cupdated_time%2Creview_info%2Crelevant_info%2Cquestion%2Cexcerpt%2Cis_labeled%2Cpaid_info%2Cpaid_info_content%2Crelationship.is_authorized%2Cis_author%2Cvoting%2Cis_thanked%2Cis_nothelp%2Cis_recognized%3Bdata%5B%2A%5D.mark_infos%5B%2A%5D.url%3Bdata%5B%2A%5D.author.follower_count%2Cbadge%5B%2A%5D.topics%3Bdata%5B%2A%5D.settings.table_of_content.enabled&limit=20&offset={}&platform=desktop&sort_by=default'.format(question_id, str(i))print(href)res3 = requests.get(href, headers=headers)datas = json.loads(res3.text).get('data')for data in datas:content = data.get('content')soup = BeautifulSoup(content, 'html.parser')print(soup.text)
![30行代码完成对知乎热榜的抓取!](http://img.gansulong.com/231105/23053S152-1.png)
文章插图
关于知乎热榜的抓取就完成啦,抓取结果大致如下,数量很多,这里只展示前面一部分:
怎么样,是不是还是挺简单的!当然如果你还想获取其它信息的话也可以针对程序进行修改来获取你想要的信息,我们要学会举一反三 。
完整程序如下:
import requestsfrom bs4 import BeautifulSoupimport jsonimport reheaders = {'X-Requested-With': 'XMLHttpRequest','User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36','cookie': ''} #cookie值替换为自己网站上的cookie值url = 'https://www.zhihu.com/hot'res = requests.get(url, headers=headers)soup = BeautifulSoup(res.text, 'html.parser')all_a = soup.select('div.HotItem-content a')for a in all_a:link = a['href']print(link + a['title'])question_id = link.replace('https://www.zhihu.com/question/', '')res2 = requests.get(link, headers=headers)soup2 = BeautifulSoup(res2.text, 'html.parser')counts_content = soup2.select('h4.List-headerText')[0].get_text()counts_re = re.search('\d+\.?\d*', counts_content)counts = int(counts_re.group())print('共有{}个回答'.format(counts))for i in range(0, counts, 20):href = http://www.kingceram.com/post/'https://www.zhihu.com/api/v4/questions/{}/answers?include=data%5B%2A%5D.is_normal%2Cadmin_closed_comment%2Creward_info%2Cis_collapsed%2Cannotation_action%2Cannotation_detail%2Ccollapse_reason%2Cis_sticky%2Ccollapsed_by%2Csuggest_edit%2Ccomment_count%2Ccan_comment%2Ccontent%2Ceditable_content%2Cattachment%2Cvoteup_count%2Creshipment_settings%2Ccomment_permission%2Ccreated_time%2Cupdated_time%2Creview_info%2Crelevant_info%2Cquestion%2Cexcerpt%2Cis_labeled%2Cpaid_info%2Cpaid_info_content%2Crelationship.is_authorized%2Cis_author%2Cvoting%2Cis_thanked%2Cis_nothelp%2Cis_recognized%3Bdata%5B%2A%5D.mark_infos%5B%2A%5D.url%3Bdata%5B%2A%5D.author.follower_count%2Cbadge%5B%2A%5D.topics%3Bdata%5B%2A%5D.settings.table_of_content.enabled&limit=20&offset={}&platform=desktop&sort_by=default'.format(question_id, str(i))print(href)res3 = requests.get(href, headers=headers)datas = json.loads(res3.text).get('data')for data in datas:content = data.get('content')soup = BeautifulSoup(content, 'html.parser')print(soup.text)
- 人事管理 | 如何用一套系统完成“入转调离”全生命周期管理?
- lstm股票走势matlab代码,用 LSTM 预测股票价格
- python 爬取名字分数
- 浅谈怎么防止源代码泄漏出去
- 5个强大到没朋友的自学网站,帮你轻松月入过万,完成人生蜕变
- VS2019创建项目和添加代码
- 高梁打完药水叶子发红怎么补救?
- 【研究生】像人一样编辑源代码,谷歌大脑提出神经网络也可以学「编程」
- 完美解决mac因为中文输入法导致的卡顿问题
- 上 构建Lua解释器Part7:构建完整的语法分析器