Python 动态网页Fetch/XHR爬虫——以获取NBA球员信息为例

动态网页抓取信息，一般利用F12开发者工具-网络-Fetch/XHR获取信息，实现难点有：

成都创新互联是一家专业提供新民企业网站建设,专注与成都做网站、网站建设、H5网站设计、小程序制作等业务。10年已为新民众多企业、政府机构等服务。创新互联专业网站设计公司优惠进行中。

1. 1. 动态网页的加载方式

   2. 获取请求Url

   3. 编排处理Headers

   4. 分析返回的数据Json

   5. pandas DataFrame的处理

我们本次想获取的信息如下：

成功获取到的csv一共506位球员，具体如下：

实现代码：

import requests
import pandas as pd
​
def get_headers(header_raw):
  return dict(line.split(": ", 1) for line in header_raw.split("\n") if line != '')
​
# 设置headers
headers_str = '''
accept: application/json, text/plain, */*
accept-encoding: gzip, deflate, br
accept-language: zh-CN,zh;q=0.9
referer: https://china.nba.cn/playerindex/
sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="96", "Google Chrome";v="96"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
sec-fetch-dest: empty
sec-fetch-mode: cors
sec-fetch-site: same-origin
cookie: sensorsdata2015jssdkcross=%7B%22distinct_id%22%3A%d0029f842fc-0d281a685dd4e08---182d0029f%22%2C%22first_id%22%3A%22%22%2C%22props%22%3A%7B%22%24latest_traffic_source_type%22%3A%22%E7%9B%B4%E6%8E%A5%E6%B5%81%E9%87%8F%22%2C%22%24latest_search_keyword%22%3A%22%E6%9C%AA%E5%8F%96%E5%88%B0%E5%80%BC_%E7%9B%B4%E6%8E%A5%E6%89%93%E5%BC%80%22%2C%22%24latest_referrer%22%3A%22%22%7D%2C%22identities%22%3A%22eyIkaWRlbnRpdHlfY29va2llX2lkIjoiMTgyZDAwMjlmODQyZmMtMGQyODFhNjg1ZGQ0ZTA4LTQzMDMwNjYtMjQwMDY5Mi0xODJkMDAyOWY4NTQwNiJ9%22%2C%22history_login_id%22%3A%7B%22name%22%3A%22%22%2C%22value%22%3A%22%22%7D%2C%22%24device_id%22%3A%d0029f842fc-0d281a685dd4e08---182d0029f%22%7D; privacyV2=true; i18next=zh_CN; locale=zh_CN
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36
'''
headers = get_headers(headers_str)
# print(headers)
​
# requests请求
param = {'locale': 'zh_CN'}
url = 'https://china.nba.cn/stats2/league/playerlist.json'
response = requests.get(url=url, headers=headers, params=param)
​
print('返回状态码：', response.status_code)
print('编码：', response.encoding)
​
# json解码成字典
myjson = response.json()
​
# 保存为pandas DataFrame
# print(players_dicts['playerProfile'])
# print(players_dicts['teamProfile'])
​
# 遍历选手信息
players_info = []
for players_dicts in myjson['payload']['players']:
  players_info.append(pd.DataFrame([players_dicts['playerProfile']]))
​
# 遍历队伍简介信息
teams_info = []
for players_dicts in myjson['payload']['players']:
  teams_info.append(pd.DataFrame([players_dicts['teamProfile']]))
​
# 得到两个DataFrame
players_pandas = pd.concat(players_info)
teams_pandas = pd.concat(teams_info)
​
# 合并得到最终DataFrame
result = pd.concat([players_pandas, teams_pandas], axis=1)
result.to_csv(r'C:\Users\WeiRonbbin\Desktop\NBA_Players1.csv')