Python web scraping with requests sign in

Question

I am working with www.freightquote.com and at some point I need to sign in otherwise not allowed me to get freight rates for more than 45 pairs.

I would like to enter sign in information for this website but for some reason it is not working. I could not understand the problem.

You can directly use this website: https://account.chrobinson.com/

I have problem to enter the information that I am asked. Here is what I did:

 from selenium import webdriver
 from time import sleep
  import pandas as pd

 from selenium.webdriver.support.ui import WebDriverWait
 from selenium.webdriver.common.by import By
 from selenium.webdriver.support import expected_conditions as EC
 from selenium.webdriver.common.keys import Keys
 from selenium.webdriver.support.ui import Select
 from selenium.webdriver.chrome.service import Service 

PATH = r'C:\Users\b\Desktop\Webscraping\chromedriver.exe'
s= Service(PATH )
driver = webdriver.Chrome(service=s)
link = "https://www.freightquote.com/book/#/free-quote/pickup"

driver.get(link)

sleep(2)

driver.maximize_window()
sleep(2)
 driver.find_elements(by=By.XPATH, value = '//button[@type="button"]')[0].click()
sleep(3)
#Username:
   driver.find_element(by=By.XPATH, value='//input[@type="email"]').send_keys('USERNAME')
   driver.find_elements(by=By.XPATH, value = '//input[@class="button button-primary" and @type="submit"]')[0].click()
 #password
  driver.find_element(by=By.XPATH, value='//input[@type="password"]').send_keys('PASSWORD')


  driver.find_elements(by=By.XPATH, value = '//input[@class="button button-primary" and @type="submit"]')[0].click()

  sleep(2)

Hamid Heravi · Answer 1 · 2022-04-27T06:49:19.647

your code and your technic have too many problems, you should learn how to code in selenium completely and then start writing code. I modified your code to the point of entering the email, please complete the code accordingly.

driver = webdriver.Chrome()
link = "https://www.freightquote.com/book/#/free-quote/pickup"

driver.get(link)

driver.maximize_window()
WebDriverWait(driver, 30).until(
    EC.presence_of_element_located((By.XPATH, 
'(//button[@type="button"])[1]'))).click()
WebDriverWait(driver, 30).until(
    EC.presence_of_element_located((By.XPATH, 
'//input[@type="email"]'))).send_keys('USERNAME')

also, you don't need to add chromedriver path in your code. if you use Windows or Linux you should add it into your virtualenv, in the /bin folder and if you use from mac you should add it to this path /usr/local/bin

score 0 · Answer 2 · answered Apr 27 '22 at 07:52

To enter sign in information for the website you need to induce WebDriverWait for the element_to_be_clickable() and you can use the following locator strategies:

Using CSS_SELECTOR:

driver.get("https://account.chrobinson.com/")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[name='username']"))).send_keys("Ribella")
driver.find_element(By.CSS_SELECTOR, "input[name='password']").send_keys("Ribella")
driver.find_element(By.CSS_SELECTOR, "input[value='Sign In']").click()

Note: You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Browser Snapshot:

chrobinson

Python web scraping with requests sign in

2 Answers2