Scrape data to a text file

241
December 01, 2017, at 09:18 AM

I'm wanting to scrape data to a text file instead of a csv. I feel like I am making this a lot more complicated than it should be and have always used csv's.

As an example lets say I wanted to scrape .question-hyperlink off stackoverflow homepage

Txt: (something like)

With open('C:/dwadawd.txt', 'a', newline='',
                  encoding="utf-8") as outfile:
            writer = csv.writer(outfile)
            for row in zip(gz_text, ga_href):
                writer.writerow(row)
                print(row) 

I have also tried the following:

#Scrapes from website
    # EXTRACT
    elems = driver.find_elements_by_css_selector('.question-hyperlink')
    ag_href = []
    for ag in elems:
        ag_href.append(elem.get_attribute("href"))

Something like:

    f = open('helloworld.txt', 'a')
    w.write(str(ag_href))
    f.write(' ')

The output just appears to be blank.

I have tried different methods but the results are the same. Any help?

Answer 1

Possible duplicate of this question.

You have to flush the file after each write function call:

f.flush()

To avoid this, you can open the file using with, this ensures the file is closed:

with open('file.txt', 'w') as f:
    f.write('some text')
Rent Charter Buses Company
READ ALSO
How can i plot line plot of retweets(y-axis) with timestamp(x-axis) according to the label(another column)?

How can i plot line plot of retweets(y-axis) with timestamp(x-axis) according to the label(another column)?

This is a snippet of twitter data I have, the labels are 0,1,20 means rumor-debunking tweet

220
KNeighborsClassifier query data dimension must match training data dimension

KNeighborsClassifier query data dimension must match training data dimension

I'm trying to find the training error on a model across different values for k

943
Dropping rows in Dataframe: labels not contained in axis

Dropping rows in Dataframe: labels not contained in axis

I have a table with three columns: user_id, book_id and ratingSo, one row shows what rating a user gave to a book

710
Python Message Encryption/Decryption Program… How hard is it to decrypt?

Python Message Encryption/Decryption Program… How hard is it to decrypt?

Here's a program I wrote to encrypt a message and decrypt the message

246