All Questions
733
questions
0
votes
1
answer
29
views
How to exclude div classes 'modal-content' and 'modal-body' from pyppeteer web scraper?
I'm building a scraper that gets text data from a list of articles. A common specimen in the text content I'm scraping at the minute is that at the bottom there is this message:
"As a subscriber, ...
0
votes
1
answer
36
views
Extracting the text between span tags in a Javascript-rendered page using Selenium in Python
I am trying to scrape all instances of text between tags with a particular class on a web page that dynamically updates. I am using selenium with a chrome WebDriver in Python.
In a normal browser, ...
- 5
2
votes
3
answers
79
views
scraping table from web page
I'm trying to scrape a table from a webpage using Selenium and BeautifulSoup but I'm not sure how to get to the actual data using BeautifulSoup.
webpage: https://leetify.com/app/match-details/5c438e85-...
- 23
0
votes
1
answer
67
views
Using Python with Selenium and BeautifulSoup4 how can i get data after Javascript has loaded all elements on the page?
I'm trying to scrape data from a sandbox website just to practice and start using python to scrape web data.
I have managed to extract a lot of data using the basics however I have found an element ...
0
votes
0
answers
62
views
Weird API response with <script> JavaScript tag
I am working on a project in Python that scrapes a university portal website to retrieve a weekly schedule. I see from the developer tools that the schedule page makes a API call and receives a JSON ...
- 13
0
votes
1
answer
53
views
make client send http request for backend flask
I am trying to avoid rate limiting and ip blacklisting while accessing an external api. I want to deploy a flask web app on google app engine.
I need a way to have the client send http requests to the ...
- 11
0
votes
0
answers
19
views
Is there a way to mimic the Element.closest() function from javascript in Scrapy python?
I am trying to convert my web-scraper I built in JavaScript using the puppeteer library into a python-based web-scraper running on Scrapy. I want to be able to do something similar to JavaScript's ...
1
vote
1
answer
38
views
How to web scrape a page that has collapsible fields using Selenium?
There is this website - https://www.flashscore.com/ and I'm looking to extract fixture data, including the competition name, team names, and dates. While I've already developed a scraping solution, ...
0
votes
1
answer
173
views
Unable to bypass rotate captcha using 2captcha API
I'm currently working on bypassing the rotate captcha on the 2captcha API's demo page. The issue I'm encountering isn't related to the integration aspect; I've already figured out how to integrate it ...
- 97
0
votes
1
answer
58
views
Beautiful Soup returns script language instead of HTML
I made a python program to scrape data from a couple shopping sites, which was working fine on both, until recently.
URL1 - https://www.auchan.pt/pt/alimentacao/alimentacao-bebe-e-crianca/papa-e-...
1
vote
1
answer
72
views
Can't Scrape a webpage whose contents are dynamically generated through JavaScript
I am trying to scrape table data from a webpage but it's not a normal webpage that can be scraped using its html tags and CSS class or ID. The contents of the webpage are dynamically generated using ...
- 11
0
votes
0
answers
23
views
How to navigate to next page using selenium by finding the javscript URL within div
Using selenium and bs4 in Python, if I run the following code
driver = webdriver.Chrome()
driver.get(url) # url = https://www.marathon.tokyo/2023/result/index.php, choose Men Marathon in first ...
- 2,998
0
votes
1
answer
55
views
How to obtain and click the URL of a dynamically loaded website?
I would like to scrape the marathon results from the link (call it page A): https://www.marathon.tokyo/2023/result/index.php
Suppose I choose the 'Marathon Men' in the first option and then search, I ...
- 2,998
0
votes
0
answers
56
views
Console Messages in Pyppeteer
I am trying to get all the console messages given a website URL using pyppeteer. However, All I can get are the responses with the 200-status code. I am using a website which has 4xx and 5xx responses ...
0
votes
1
answer
78
views
How to interact with a button using selenium
I am trying to scrape some information from this page (Binance announcements)
I am trying to go through all the announcements and scrape the data from the announcements. however, I am having trouble ...
- 47