find broken links on webpage using selenium webdriver by avoiding dynamically generated links

Question

I am finding broken links on webpage by finding all anchor tags.
But there are some dynamically generated href's through javascript.
When I print the list of all links , i get StaleElementReferenceException due to dynamically generated link.
Why do I get StaleElementReferenceExceptionfor the below twitter link ?

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>
    <head>
        <title></title>
    </head>
<body>
    <div style="padding-top:10px;">
        <a href="https://twitter.com/url" class="twitter-follow-button" data-show-count="false" data-size="large" data-show-screen-name="false">Follow @url</a>
            <script>!function(d, s, id) {
                var js, fjs = d.getElementsByTagName(s)[0], p = /^http:/.test(d.location) ? 'http' : 'https';
                if (!d.getElementById(id)) {
                    js = d.createElement(s);
                    js.id = id;
                    js.src = p + '://platform.twitter.com/widgets.js';
                    fjs.parentNode.insertBefore(js, fjs);
                }
            }(document, 'script', 'twitter-wjs');</script>
    </div>
</body>

Selenium code :

     List<WebElement> links=driver.findElements(By.tagName("a"));    for(WebElement link: links)   {       System.out.println(link.getAttribute("href"));   }

It would help if you could paste your code, but I suspect my answer to a post from yesterday may explain the problem: http://stackoverflow.com/q/35130727/954442 — Andrew Regan, Feb 02 '16 at 13:04
Where are you actually getting the error? In Java test code? If so, could I see that? There's no Selenium in your JS, and presumably that works OK. — Andrew Regan, Feb 08 '16 at 11:05

score 0 · Answer 1 · edited Feb 16 '16 at 15:32

Almost certainly what's happening is:

get(<url>) is called, page starts to load
findElements(By.tagName("a")) is called. This sends event to browser, which starts to build list of matches and return them to WebDriver caller.
Meanwhile, the <script> tag has been inserted, with a request to platform.twitter.com/widgets.js, which starts to generate new / modify existing <a> elements.

In a nutshell, because your WebElements potentially come from two different states (before load, after load), and because some from "before" no longer match the same in "after", the "before" elements are determined to be stale.

There's a number of things you can do:

Delay your findElements(By.tagName("a")) until you know that the Twitter widgets have finished loading, i.e. by waiting for the existence of one of the generated links.
Catch the StaleElementReferenceException if it happens, and redo the findElements lookup in your exception handler.

(2) is probably simpler. If you only get the exception 20% of the time and the lookup is quick, performance won't be greatly affected.

Can you explain this further- In a nutshell, because your WebElements potentially come from two different states (before load, after load), and because some from "before" no longer match the same in "after", the "before" elements are determined to be stale. — Grishma Oswal, Feb 11 '16 at 08:47
Sure: the elements are tied to the page they were created on, so if the page refreshes or redirects, or a script removes them from the DOM, the references become invalid, and accessing them gives an error. You can catch that error (option 2 above) and simply re-do the `findElements` to get fresh references. — Andrew Regan, Feb 11 '16 at 12:51

find broken links on webpage using selenium webdriver by avoiding dynamically generated links

1 Answers1