Webscraping Amazon with BeautifulSoup

32
November 20, 2018, at 06:00 AM

I am trying to webscrape Amazon's reviews: https://www.amazon.com/Python-Crash-Course-Hands-Project-Based/dp/1593276036/ref=sr_1_3?ie=UTF8&qid=1541450645&sr=8-3&keywords=python

Here is my code:

import requests as req
from bs4 import BeautifulSoup
headers = {'User-Agent': 'Kevin\'s_request'}
r = req.get('https://www.amazon.com/Python-Crash-Course-Hands-Project-Based/dp/1593276036/ref=sr_1_3?ie=UTF8&qid=1541450645&sr=8-3&keywords=python', headers=headers)
soup = BeautifulSoup(r.text, "html.parser")
soup.find(class_="a-expander-content a-expander-partial-collapse-content")

I only end up with an empty list. I am using Python 3.6.4 in Jupyter Notebooks and BS 4

Answer 1

Not sure what's happening on your side, but this code works fine. Here it goes (python 3.6, BSP 4.6.3):

import requests
from bs4 import BeautifulSoup
def s_comments(url):
    headers = {'User-Agent': 'Bob\'s_request'}
    response = requests.get(url, headers=headers )
    if response.status_code != 200:
        raise ConnectionError
    soup = BeautifulSoup(response.content)
    return soup.find_all(class_="a-expander-content a-expander-partial- collapse-content")

url = 'https://www.amazon.com/dp/1593276036'    
reviews = s_comments(url)
for i, review in enumerate(reviews):
    print('---- {} ----'.format(i))
    print(review.text)
READ ALSO
MERN App's Redux functionality not working after build

MERN App's Redux functionality not working after build

So I recently pushed my MERN app to AWS's EC2, this is the 1st time I've tried hosting and allBut when I ran the build script, shortly after that, none of the redux actions seem to work

50
Input range connected

Input range connected

I can't figure out how I can connect two input range

32
How can we define 'PlacesServiceStatus.OK' for adding markers of locations on customized google maps in laravel 5.7?

How can we define 'PlacesServiceStatus.OK' for adding markers of locations on customized google maps in laravel 5.7?

Have there any alternative defined function in laravel 57? I have tried my best but fail to get an answer

19
Include PHP and CSS files -> relative path

Include PHP and CSS files -> relative path

Hello how can I include PHP and CSS files from another directory to the current file? Open Image

36