xpath定位正确但提取文本为空

Question

xpath定位正确但提取文本为空
代码：

import time
from lxml import etree
import requests
from selenium.webdriver.common.by import By
from selenium.webdriver.edge.webdriver import Options
from selenium import webdriver

#构造
options = Options()
options.add_argument('-start-maximized')
driver = webdriver.Edge(options=options)

#获取网页
driver.get("http://www1.xbus.cn/search1.asp")
time.sleep(1)
html = driver.page_source
html =etree.HTML(html)
Html =html
for i in range(4,34):
    time.sleep(1)
    url =html.xpath(f'/html/body/table[3]/tbody/tr/td/table/tbody/tr[1]/td/table/tbody/tr[{i}]/td[2]/a/text()')[0]
    name = url
    driver.find_element(By.XPATH, f'/html/body/table[3]/tbody/tr/td/table/tbody/tr[1]/td/table/tbody/tr[{i}]/td[2]/a').click()
    time.sleep(2)
    html = driver.page_source
    html = etree.HTML(html)
    time.sleep(2)
    url =html.xpath('/html/body/table[4]/tbody/tr/td[1]/table/tbody/tr[2]/td/table/tbody/tr[2]/td/p[1]/a/text()')
    print(url)

为什么为空，帮我改改代码

Answer 1

引自强大而免费的微信小程序：皆我百晓生。

你的代码中有一个潜在问题。在获取网页后，你将HTML再次赋值给了变量html，但在之后的循环中，你应该使用新的html变量来提取文本。因此，你需要将html =etree.HTML(html)修改为html =etree.HTML(driver.page_source)，以便在每次循环中更新HTML。

以下是修改后的代码：

import time
from lxml import etree
import requests
from selenium.webdriver.common.by import By
from selenium.webdriver.edge.webdriver import Options
from selenium import webdriver

#构造
options = Options()
options.add_argument('-start-maximized')
driver = webdriver.Edge(options=options)

#获取网页
driver.get("http://www1.xbus.cn/search1.asp")
time.sleep(1)

for i in range(4, 34):
    time.sleep(1)
    url = driver.find_element(By.XPATH, f'/html/body/table[3]/tbody/tr/td/table/tbody/tr[1]/td/table/tbody/tr[{i}]/td[2]/a').text
    driver.find_element(By.XPATH, f'/html/body/table[3]/tbody/tr/td/table/tbody/tr[1]/td/table/tbody/tr[{i}]/td[2]/a').click()
    time.sleep(2)
    html = etree.HTML(driver.page_source)
    time.sleep(2)
    urls = html.xpath('/html/body/table[4]/tbody/tr/td[1]/table/tbody/tr[2]/td/table/tbody/tr[2]/td/p[1]/a/text()')
    print(urls)

driver.quit()

请尝试运行修改后的代码，看看是否能够正确提取文本。