How can I hold the driver of a specific page until a specific drop-down menu list save and entire list request using Selenium Python?

Question

I made a crawler for this page (http://www.bobaedream.co.kr/cyber/CyberCar.php?gubun=I) to collect the stock list of specific manufacturers. My code starts from selecting the drop-down menu in the first row of search menu.

Each right drop-down menu is child menu of its left drop-down menu. What I would like to do is to select each first item in each drop-down menu and click the "search" button for the first run. After crawling of its stock list, then I set the second item of the last drop-down menu and click the "search" button. (If there is only one item in the last drop-down menu, then go-up to the second drop-down menu from the last and pick the second item in it.)

But the problem is occurred here. I saved each items of each drop-down menu as tuple. When I try to call the second item of the last drop-down menu for the second round of crawling, "StaleElementReferenceException" or "NoSuchElementException" is occurred with the message of "Element is no longer attached to the DOM". Thus, I would like to make the drive until each drop-down menu iteration is completed.

Below is my code, but still have the error message. My error usually occurs at the second while loop. At this moment, I added some "wait" function and "refresh" function. Sometimes it can help to get out of while loop in my code, but usually it got stuck in the while loop and infinite repeat. I guess some type of "wait.until(EC.~)" code in the second "try" function can work this out, but I have no specific idea for this. Please help or give me any advice.

def option2_menu_loaded(inDriver):
    path = '//select[@id="level2_no"]'
    return inDriver.find_element_by_xpath(path)

self.wait.until(option2_menu_loaded)

while True:
    try:
        select_option2_values = [
            ('%s' % o.get_attribute('text'), '%s' % o.get_attribute('value'))
            for o
            in Select(self.driver.find_element_by_css_selector("#level2_no")).options
            if o.get_attribute('text') != '세부등급']
    except (StaleElementReferenceException, NoSuchElementException):
        print("=======Exception Found - Option2 Save=====")
        self.driver.refresh()
        self.driver.implicitly_wait(1.5)
        continue
    break

for option2 in select_option2_values:
    self.csv.setCarTitle(ma, mo, de, option1[0], option2[0])

    print(option2[0], option2[1])
    self.driver.implicitly_wait(0.5)

    while True:
        try:
            Select(self.driver.find_element_by_css_selector("#level2_no")).select_by_value(option2[1])

        except (StaleElementReferenceException, NoSuchElementException):
            print("=======Exception Found - Option2 Request=====")
            self.driver.refresh()
            self.driver.implicitly_wait(1.5)
            self.driver.refresh()
            continue
        break

score 0 · Answer 1 · answered Feb 05 '17 at 10:24

0

This means the bags reloads between finding the element and using the element object, make sure the page is loaded first and search for the element each time you need to use it.

An extra check would be to add an extra wait for the element before doing the select since find_element_by_css_selector will not wait at all for element to be available.

The sequence should be: page is loaded > wait until element exists > use action on element

answered Feb 05 '17 at 10:24

lauda

4,153
2
14
28

Thanks for the comment. Then what is the "wait until element exists" function I need to apply? Do you have any idea? – 신종원 Feb 05 '17 at 11:13
Check here for a list with python waits for selenium https://selenium-python.readthedocs.io/waits.html – lauda Feb 05 '17 at 11:26
Here is a question with some examples https://stackoverflow.com/questions/7781792/selenium-waitforelement – lauda Feb 05 '17 at 11:27

How can I hold the driver of a specific page until a specific drop-down menu list save and entire list request using Selenium Python?

1 Answers1