Java selenium 4 || Scrapping Titles from each WebElement link

Question

I have a list of Anchor tags. I want to click on each tag , scrap the page's title and print each item. However it returns error after the second item.

public static void main(String[] args) {
    // TODO Auto-generated method stub
    System.setProperty("webdriver.chrome.driver", "D:\\ronjg\\Desktop\\DEV - LANG\\chromewebdriver\\chromedriver.exe");
    WebDriver driver = new ChromeDriver();
    driver.get("https://petstore.octoperf.com/actions/Catalog.action");
    

    List<WebElement> myList=driver.findElements(By.id("SidebarContent"));



    
    
    
     for (WebElement element : myList) {
            List<WebElement> listofAtags = element.findElements(By.tagName("a"));
            int size_of_links = listofAtags.size();
            System.out.println(size_of_links);

             for (WebElement lnk : listofAtags) {
                 lnk.click();
                 String res = driver.findElement(By.xpath("/html/body/div[2]/div[2]/h2")).getText();
                 System.out.println(res);
                 WebElement reutrn_home = driver.findElement(By.xpath("//*[@id=\"BackLink\"]/a"));
                 reutrn_home.click();
                 
             }
            
            
                
           
            
            }
     
     driver.close();
    


     }}

The Links of the group are : Fish, Dogs, Cats, Reptiles, Birds . So I want to to print each item , so the end print out would be like this : Fish Dogs Cats Reptiles Birds

score 0 · Answer 1 · answered Jul 07 '22 at 07:58

First of all you can use Jsoup instead of Selenium. The specified website is static and you don't need selenium: Selenium gives you the ability to work with dynamic pages.

If you want to implement this functionality with Jsoup, here is the code snippet:

public class Main {


    public static void main(String[] args) {

        Document document = null;
        try {
            document = Jsoup.connect("https://petstore.octoperf.com/actions/Catalog.action").get();
        } catch (IOException ex) {
            e.printStackTrace();
        }
        Elements anchorElements = document.select("#SidebarContent a")
        for (Element anchor : anchorElements) {
            String url = anchor.attr("abs:href");
            parseData(url);
        }
        
    }

    public static void parseData(String url) {
        Document document = null;
        try {
            document = Jsoup.connect(url).get();
        } catch (IOException ex) {
            e.printStackTrace();
        }
        String title = document.title();
        System.out.println(title);
        String h2Tag = document.select("#SidebarContent h2").text();
        System.out.println(h2Tag);
    }

}

Also you don't need to select h2 element by xpath. You can select by #SidebarContent h2

If you decided to use Selenium, so what kind of exception or error you are getting? May be there is no any element on the page specified by the xpath

The error I get is " stale element reference: element is not attached to the page document ". I want to click on each Element to print it's title to check if each link has the expected result so h2 is not good on this case . — Ron Goldstein, Jul 07 '22 at 08:04
What is the meaning: "to check if each link has the expected result"? By the way you can add try catch. Then catch the StaleElementReferenceException and try to click element once more. Here is the duplication question - https://stackoverflow.com/questions/18225997/stale-element-reference-element-is-not-attached-to-the-page-document — Orkhan Hasanli, Jul 07 '22 at 20:01

score 0 · Answer 2 · answered Jul 12 '22 at 06:36

Try the following code -

    try {
        List<WebElement> listOfAnchors = driver.findElements(By.xpath("//div[@id='SidebarContent']/a"));
        int anchorTagsCount = listOfAnchors.size();

        for (int i = 0; i <= anchorTagsCount; i++) {
            listOfAnchors = driver.findElements(By.xpath("//div[@id='SidebarContent']/a"));

            // Store all the text in a List if you want to.
            System.out.println(listOfAnchors.get(i).getText().trim());

            listOfAnchors.get(i).click();
            Thread.sleep(2000);

            // Store all the titles in a separate List if you want to.
            System.out.println(driver.getTitle());

            driver.navigate().back();
            Thread.sleep(2000);
        }

    } catch (Exception e) {
        e.printStackTrace();
    }

Java selenium 4 || Scrapping Titles from each WebElement link

2 Answers2