Python selenium print frame source

528

March 29, 2017, at 11:52 PM

This is my first foray into Selenium. Apologies in advance if this is a stupid/trivial question.

I am trying to scrape information from a webpage. With Python/Selenium I am able to log on to the site and get to the page with the information I need. After the page I need is displayed, I am issuing

time.sleep(20)
html_source = driver.page_source
print html_source

The "source" that gets printed is different from both the right click and select view page source and right click and select This Frame, View Frame source

The required information is in the View Frame source. All of this is in Firefox.

What do I need to do to get to the Frame Source? There is no frame name in the Frame Source.

Additional information below:

When I right click and select view page source I get the below:

<!DOCTYPE html><html>
   <head>
      <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
      <meta http-equiv="X-UA-Compatible" content="IE=edge">
      <title>xxxxxxx Portal</title>
      <base href="https://website.org/page/">
      <link rel="shortcut icon" href="images/logos/xxxxxxx.ico">
      <meta http-equiv="Pragma" content="no-cache">
      <meta http-equiv="Expires" content="-1"><script type="text/javascript" src="https://website.org/page/security/csrf.js"> </script><script type="text/javascript" src="https://website.org/page/security/csrf/execute.js"> </script><script>
                    function pushFocus()
                    {
                        frameDetail.focus();
                    }
                    function addInProgressPanel(doc)
                    {
                        var d = doc.createElement('div');
                        d.id="inProgressPane";
                        d.className="freezeOn";
                        var tbl = doc.createElement("table");
                        var row = tbl.insertRow(-1);
                        var oi = doc.createElement("img");
                        oi.src= 'https://website.org/page/'+ "images/actions/loading2.gif";
                        var td = doc.createElement("td");
                        td.className="detailFormField";
                        td.bgcolor="red";
                        td.appendChild(oi);
                        row.appendChild(td);
                        td = doc.createElement("td");
                        td.className="inProcessing";
                        td.appendChild(doc.createTextNode("Your Request is Being Processed  ..."));
                        row.appendChild(td);
                        d.appendChild(tbl);
                        doc.body.appendChild(d);
                        return d;
                    }
                    function inProgressScreen(type)
                    {
                        var ws = frames["frameDetail"];
                        if(!ws) return true;
                        var ips = ws.document.getElementById("inProgressPane");
                        if(ips)
                        {
                            if(type)  ips.className = 'freezeOn';
                            else      ips.className = 'freezeOff';
                        }else if(type)
                            ips = addInProgressPanel(ws.document);
                    }
                </script></head>
   <frameset id="main" framespacing="0" frameborder="0">
      <frame id="frameDetail" name="frameDetail" scrolling="auto" marginwidth="0" marginheight="0" src="portal/portal.xsl?x=portal.PortalOutline&amp;lang=en&amp;mode=notices">
   </frameset>
</html>

When I right click and select This Frame, View Frame source I get

<!DOCTYPE html><html>
   <head>
      <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
      <base href="https://website.org/xxxxxx/">
      <meta http-equiv="Content-Language" content="en-us">
      <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
      <meta http-equiv="X-UA-Compatible" content="IE=edge">
      <meta http-equiv="Pragma" content="no-cache">
      <meta http-equiv="Expires" content="-1">
      <title>xxxxxxxx Portal</title>
      <link rel="stylesheet" type="text/css" href="styles/portal/menu.css">
      <link rel="stylesheet" type="text/css" href="styles/portal/header.css">
      <link rel="stylesheet" type="text/css" href="styles/portal/footer.css">
      <link rel="stylesheet" type="text/css" href="styles/portal/jquery-ui-1.8.7.portal.css">
      <link rel="stylesheet" type="text/css" href="styles/portal/fg.menu.css">
      <link rel="stylesheet" type="text/css" href="styles/portal/portal.css">
      <link rel="stylesheet" type="text/css" href="styles/icons.css">
      <link rel="stylesheet" type="text/css" href="styles/portal/notifications.css"><script type="text/javascript" src="https://website.org/xxxxxxxx/security/csrf.js"> </script><script type="text/javascript" src="https://website.org/xxxxxxxx/security/csrf/execute.js"> </script><script src="scripts/widgets/common.js"></script><script src="scripts/controller.js"></script><script src="scripts/portal.js"></script><script src="scripts/jquery/jquery-1.7.2.min.js"></script><script type="text/javascript" src="https://website.org/xxxxxxxx/security/csrf/jquery.js"> </script><script src="scripts/jquery/jquery-ui-1.8.16.min.js"></script><script src="scripts/jquery/fg.menu.js"></script><script src="portal/lang/datePickerLanguage.jsp?lang=en"></script><script src="portal/portal.js"></script><script src="portal/portalNoShim.js"></script><script>

Lots more code here. Did not paste as it was too long. There is no frame name other than the reference to iSessionFrame below:

    </script><script language="javascript" src="portal/grades.js"></script></div>
            </div>
         </div>
         <div id="footer">
            <table id="language"><select id="locale" style="width:175px"></select></table>
         </div>
      </div><iframe id="iSessionFrame" name="iSessionFrame" width="0" height="0" src="https://website.org/xxxxxx/white.jsp" style="visibility:hidden;"></iframe></body>
</html>

Answer 1

Q: What do I need to do to get to the Frame Source?

A: First you must switch to the wanted frame using the switch_to command and then you should use .page_source to get the html source.

Obs.: take a look at Selenium Docs, more specifically at Moving between windows and frames.

Code:

driver.switch_to_frame(driver.find_element_by_tag_name("frameDetail"))
driver.page_source

POPULAR ONLINE

Html input is not selected on click

Laravel Valet not working- as if it's missing a web server

Hibernate @One-To-Many using @JoinTable does not work properly for update: Duplicate entry '1478-49547' for key 'PRIMARY'

DevExpress checkbox tree view search error

Add a gradient to the Easy Pie Chart plugin

Python's tab spacing

So I have a program that prints out two "fish" per line, and the distance between two fish is a tab character "\t", the output looks like this:

432

Python

Save regular expression result in variable

I would like to split a string into separated, single strings and save each in a new variableThat's the use case:

427

Python

Find (a^b)%c in python

I am trying to create a Python program which will print the value of (a^b)%c quickly even for very big values of a,b and cThe logic which I am using to do this is here: https://www

456

Python

Python: Finding length of the repeating sequence of numbers in a string of floats

I am trying to write a function which takes in a list of floats and determines whether or not there is a repeating sequence of number within that listIf there is a repeating sequence then it counts the numbers in that sequence

539