How to scrape page requiring cookies and javascript in PHP

52
July 16, 2019, at 12:50 PM

Is there an easy way to emulate cookies and javascript with a php script scraping a web page requiring it?

The current response shows:

<body><noscript>This site requires JavaScript and Cookies to be enabled. Please change your browser settings or upgrade your browser.</noscript></body>

I put this in the code and it made no difference:

$strCookie = 'PHPSESSID=' . $_COOKIE['PHPSESSID'] . '; path=/';
curl_setopt( $ch, CURLOPT_COOKIE, $strCookie );
Answer 1

HTML inside the <noscript> </noscript> will only get displayed if Javascript is disabled. So this is the test for whether Javascript is enabled.

There is jQuery-cookie-plugin for reading, writing and deleting cookies. But it will not work if Javascript is disabled.

For checking if cookie is enabled with PHP code, you need to set the cookie and read the cookie with another request. This post have some guides on it.

Answer 2

Is there an easy way to emulate cookies and javascript with a php script scraping a web page requiring it?

No.

READ ALSO
How to implement auto assign order in slim framework?

How to implement auto assign order in slim framework?

In our mobile application we want to add auto assign functionality for agent regarding end customer order

45
How to convert json value database to html

How to convert json value database to html

I have notification table on my database where the value is using like json

70
How to detect the anomaly in a time series data(specifically) with trend and seasonality present in it?

How to detect the anomaly in a time series data(specifically) with trend and seasonality present in it?

I want to detect the outliers in a "time series data" which contains the trend and seasonality componentsI want to leave out the peaks which are seasonal and only consider only the other peaks and label them as outliers

52
killing py4j request and java processes associated with it

killing py4j request and java processes associated with it

I am calling java function from python using py4j java gateway

57