python如何设计爬虫界面软件_Python+PyQt5设计B站视频数据爬虫小程序

开发环境

Python3

Pycharm

PyQt5

Qt Designer

requests

准备工作

首先，新建一个bilibili.py文件。获取B站视频数据的爬虫代码相对简单，只要在设置好url的keyword和page参数，即可轻松获取数据。以下贴出完整代码：

importrequestsfrom lxml importetreeimporttimeimportrandomimportcsvdefget_target(keyword,page):for i in range(1,page + 1):

headers= {‘User-Agent’:’Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36′}

url= ‘https://search.bilibili.com/allyword={0}&from_source=nav_suggest_new0&page={1}’.format(keyword,page)

html= requests.get(url.format(i),headers =headers)

bs=etree.HTML(html.text)

items= bs.xpath(‘//li[@class = “video-item matrix”]’)for item initems:

title= item.xpath(‘div[@class = “info”]/div/a/@title’)[0]

click= item.xpath(‘div[@class = “info”]/div[3]/span[1]/text()’)[0].strip(‘n’).replace(“万”,””)

danmu= item.xpath(‘div[@class = “info”]/div[3]/span[2]/text()’)[0].strip(‘n’)

date= item.xpath(‘div[@class = “info”]/div[3]/span[3]/text()’)[0].strip(‘n’)

up= item.xpath(‘div[@class = “info”]/div[3]/span[4]/a/text()’)[0].strip(‘n’)

data= {‘标题’: title, ‘播放量(万)’: click, ‘弹幕’: danmu, ‘日期’: date, ‘UP主’: up}

with open(‘B站数据.csv’, ‘a+’, encoding=’utf_8_sig’, newline=”) as fp:

fieldnames= [‘标题’, ‘播放量(万)’, ‘弹幕’, ‘日期’, ‘UP主’]

writer= csv.DictWriter(fp, fieldnames=fieldnames)

writer.writerow(data)

time.sleep(random.random()+ 1)print(‘已经完成b站第 {} 页爬取’.format(i))if __name__ == “__main__”:

keyword= input(“请输入要搜索的关键词：”)

page= int(input(“请输入要爬取的页数：”))

get_target(keyword,page)

比如keyword输入”打工人”，page输入5，运行爬虫代码，即可生成打工人相关的B站视频数据csv文件，数据预览如下：

添加QT Desinger

Name: QT Designer

Program: designer.exe所在的路径#注意：该路径为你Python安装路径下Libsite-packagespyqt5_tools文件夹里

Working directory: $FileDir$

添加Py UIC

Name: Py UIC

Program: pyuic5.exe所在路径#该路径为你Python安装路径下Scripts文件夹里

Arguments: $FileName$ -o $FileNameWithoutExtension$.py

Working directory: $FileDir$

界面介绍

将ui文件转为py代码

界面设计好以后，保存为run.ui文件，选中“run.ui”，在其上点击鼠标右键，到“External Tools”中点击“PyUIC”，即可生成run.py文件：

爬虫代码与界面相结合

完善run.py，赋给“开始下载”按钮以‘pushButton_click’函数，赋给“保存路径”按钮以‘setBrowerPath’函数：

#链接爬虫函数

self.pushButton.clicked.connect(Form.pushButton_click)

self.pushButton_2.clicked.connect(Form.setBrowerPath)

完善爬虫代码，将界面整体框架融入其中：

importsysfrom run import Ui_Form #继承run.py

from PyQt5.QtWidgets import *

from PyQt5 importQtCore, QtGui, QtWidgetsclassMyPyQT_Form(QWidget, Ui_Form):def __init__(self):

super(MyPyQT_Form, self).__init__()

self.setupUi(self)defpushButton_click(self):

······defsetBrowerPath(self):

······if __name__ == ‘__main__’:

app=QApplication(sys.argv)

my_pyqt_form=MyPyQT_Form()

my_pyqt_form.show()

sys.exit(app.exec_())

然后在pushButton_click函数中添加代码， 接收界面输入的keyword和page参数，导入接口url中，代码如下：

defpushButton_click(self):

self.textEdit_4.setText(“下载中……”)

self.keyword= self.lineEdit.text() #关键字

self.page = int(self.lineEdit_2.text()) #页数

#result = pd.DataFrame()

for i in range(1,self.page + 1):

time.sleep(random.random()+ 0.5)

headers= {‘User-Agent’:’Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36′}

url= ‘https://search.bilibili.com/allyword={0}&from_source=nav_suggest_new0&page={1}’.format(self.keyword,self.page)

html= requests.get(url.format(i),headers =headers)

bs=etree.HTML(html.text)

items= bs.xpath(‘//li[@class = “video-item matrix”]’)for item initems:

title= item.xpath(‘div[@class = “info”]/div/a/@title’)[0]

click= item.xpath(‘div[@class = “info”]/div[3]/span[1]/text()’)[0].strip(‘n’).replace(“万”,””)

danmu= item.xpath(‘div[@class = “info”]/div[3]/span[2]/text()’)[0].strip(‘n’)

date= item.xpath(‘div[@class = “info”]/div[3]/span[3]/text()’)[0].strip(‘n’)

up= item.xpath(‘div[@class = “info”]/div[3]/span[4]/a/text()’)[0].strip(‘n’)

data= {‘标题’: title, ‘播放量(万)’: click, ‘弹幕’: danmu, ‘日期’: date, ‘UP主’: up}

with open(‘{0}/{1}视频数据.csv’.format(self.download_path,self.keyword),’a+’, encoding=’utf_8_sig’, newline=”) as fp:

fieldnames= [‘标题’, ‘播放量(万)’, ‘弹幕’, ‘日期’, ‘UP主’]

writer= csv.DictWriter(fp, fieldnames=fieldnames)

writer.writerow(data)

self.textEdit_4.append(“第{0}页下载完成”.format(i))

self.textEdit_4.append(“保存路径:{}”.format(self.download_path))

self.textEdit_4.append(“文件名称:{}”.format(‘{0}共{1}个视频数据.csv’.format(self.keyword,self.page*20)))

充实setBrowerPath函数，获取数据存储路径，将路径在输出框显示出来：

defsetBrowerPath(self):

self.download_path=QFileDialog.getExistingDirectory(self)

self.textEdit_3.setText(self.download_path)

工具打包

在cmd控制台使用cd进入py文件所在目录:

cd D:GUI工具B站视频数据下载器

使用pyinstaller打包:

pyinstaller -F -w bilibili.py

运行完成后，会在当前目录下生成1个dist文件夹和build文件夹，打开dist文件夹，双击bilibili.exe，工具即可运行。mac环境下可以使用py2app进行打包。

文章知识点与官方知识档案匹配，可进一步学习相关知识Python入门技能树首页概览211389 人正在系统学习中 相关资源：开源的爬虫软件Heritrix3.1.0_开源爬虫-Java工具类资源-CSDN文库