Python Web Scraping: Beautiful Soup

  • A+
Category:Languages

I have a problem with the scraping of a web page. I'm trying to get the difference of points (Ex: +2,+1,...) between two teams but when I apply the find_all method it returns an empty list...

from bs4 import BeautifulSoup from requests import get url='https://www.mismarcadores.com/partido/Q942gje8/#punto-a-punto;1' response=get(url) html_soup=BeautifulSoup(response.text,'html.parser')   html_soup.find_all('span',class_='match-history-diff-score-inc') 


The problem you have is the web content is being generated dynamically through JavaScript. As such, requests is unable to handle it, and so you'd be better off using something like Selenium:

from bs4 import BeautifulSoup from selenium import webdriver  driver = webdriver.Chrome() driver.get("https://www.mismarcadores.com/partido/Q942gje8/#punto-a-punto;1")  html = driver.page_source soup = BeautifulSoup(html, "lxml")  table = soup.find_all('span',{'class' : 'match-history-diff-score-inc'}) results = [] for tag in table:     results.append(tag.text) print(results) 

This outputs:

['+2', '+1', '+2', '+2', '+2', '+4', '+2', '+4', '+7', '+8', '+7', '+5', '+5', '+5', '+6', '+6', '+7', '+4', '+5', '+7', '+8', '+1', '+2', '+4', '+7', '+8', '+9', '+11', '+11', '+10', '+11', '+1', '+3', '+3', '+3', '+4', '+2', '+3', '+6', '+5', '+1', '+1', '+2', '+4', '+4', '+3', '+5', '+7', '+8', '+4', '+6', '+9', '+9', '+10', '+11', '+12'] 

Comment

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen: