0

So i need to scrap a page like this for example or this and i am using Scrapy + Seleninum to interact with a datepicker calendar.

Basically i need to check for the availabilities of the rooms for each day in a month for example, so my idea was trying to click the not disabled links/dates on the datepicker and check if an error message appears or not for me to know if its available or not.

I have no idea if this is the proper way or even how to do this with this tools. I have been playing around with them but still looks like i am far from a final solution.

Does anyone know how i can tackle this or even provide the code for me to achieve this?

psychok7
  • 5,373
  • 9
  • 63
  • 101

1 Answers1

1

Hi please find the answer below 

WebDriver driver = new FirefoxDriver();
driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
driver.manage().window().maximize();
driver.get("https://www.airbnb.pt/rooms/516301?check_in=2016-04-14&guests=1&check_out=2016-04-17");

// selecting firstdate picker -- check in

driver.findElement(By.xpath("//*[@class='col-sm-6']/input")).click();
// note calendar is not completely visible hence to make it visible 
// scroll a little bit down
((JavascriptExecutor) driver).executeScript("window.scrollBy(0,200)");
// take all calendar dates inside the list
// here i have update my code to skip stale element exception 
List<WebElement> alldates = driver.findElements(By.xpath("//*[@class='ui-datepicker-calendar']/tbody/tr/td/a[contains(@class, 'ui-state-default')]"));
System.out.println("Size is : "+ alldates.size());
// suppose you want to select 27 form the all dates
// keeping it as a parameter 
String dateToBeSelected = "19";  // check in
for(int i=0;i<alldates.size();i++){
    alldates = driver.findElements(By.xpath("//*[@class='ui-state-default']"));
System.out.println("dates is : " + alldates.get(i).getText());
// selects a check-in date
if( alldates.get(i).getText().equals(dateToBeSelected)){
    alldates.get(i).click();
    break;
            }
        }
// on selection of Checkin date check out calender automatically pop ups
System.out.println("--------------------------");
String datetobeselected = "27";
for(int i=0;i<alldates.size();i++){
    alldates = driver.findElements(By.xpath("//*[@class='ui-state-default']"));
System.out.println("dates is : " + alldates.get(i).getText());
// selects a check-in date
if( alldates.get(i).getText().equals(datetobeselected)){
    alldates.get(i).click();
    break;
            }
        }
// actually after calendar selection whatever text is shown in red color
// i am not sure what is it (sorry only English)

// identify the text with red color is present or not 
boolean errorMsg = driver.findElement(By.xpath("//*[@class='panel-body panel-light']/div[2]/p")).isDisplayed();
if(errorMsg){
    System.out.println("error msg is displayed");
}else{
System.out.println("error msg is not - displayed");
        }

Hope this helps you also note this above example takes check-in and checkout date as parameter if you want code to take non disabled date automatically please feel free to ask i will update my answer.

Rajnish Kumar
  • 2,828
  • 5
  • 25
  • 39
  • thanks, i will test this as soon as possible and let you know (mark you answer as correct) if it works – psychok7 Apr 06 '16 at 10:07
  • i think i am almost there, but i am running into this issue sometimes right before checking if the element is the one i want to click `StaleElementReferenceException: Message: Element not found in the cache - perhaps the page has changed since it was looked up` and i also am running into a `NoSuchElementException: Message: Unable to locate element` when looking for the message to check if its available or not. By the way you didn't add the click to change months right? – psychok7 Apr 06 '16 at 14:10
  • just to add how to deal with stale element exception : you can see above code in the for loops of check-in and check-out date selection (dateToBeSelected = "19"; datetobeselected = "27";) i have called alldates again just to avoid stale element ,so you can do it like that in your code as well. – Rajnish Kumar Apr 06 '16 at 14:16
  • this is how my code is right now https://dpaste.de/UDyD it looks similar to yours and i added some sleeps to see if i could fix the stale issue but still nothing.. do you see anything wrong? – psychok7 Apr 06 '16 at 14:20
  • please tell me exactly which line you are getting stale element exception in your code it will help form my pre-spective code looks good – Rajnish Kumar Apr 06 '16 at 14:22
  • i am having the stale issue in line 31 `if alldates[i].text == dateToBeSelected:` and the unable to locate in 63 `errorMsg = self.driver.find_element(By.XPATH, "//*[@class='panel-body panel-light']/div[2]/p")` – psychok7 Apr 06 '16 at 14:25
  • i realized that for the locate issue its because i am using a diferent url and its valid so the element wich is dynamic does not exit. i cant work arount it with exceptions. but the stale issue remains – psychok7 Apr 06 '16 at 14:29
  • the stale issue seems to be random... sometimes it works, but sometimes e throws the error... i am using `self.driver.implicitly_wait(10)` after `alldates = self.driver.find_elements(By.XPATH, "//*[@class='ui-state-default']")` but it doesnt seem to be working.. should i changed it to python `time.sleep()` ?? – psychok7 Apr 06 '16 at 14:39
  • yeah you are true some times its working and some times its not m looking into it hopefully i will came with a solution to that ASAP and sorry for the problem you are facing – Rajnish Kumar Apr 06 '16 at 14:45
  • Someone gave me another approach maybe you can help me, so instead of doing it like this, how can i get the prices and if its available for each of the days just by hovering the mouse? thats the information i need, so you know how i can simulate that behaviour it would be much more efficient – psychok7 Apr 06 '16 at 14:52
  • when you hover the mouse a tooltip with the price appears on the calendar if the date is available. and if its not available its disabled.. should i create another question for this and mark this one as resolved? – psychok7 Apr 06 '16 at 14:53
  • yeah that can be done but not inside this question please raise a separate question for that and also to add on chrome above solution works 8-9 times out of 10 .please try the same code in chrome at your side ,if i gate any better way then i will update this – Rajnish Kumar Apr 06 '16 at 14:55
  • and also if you think my solution helped you what you are looking at then plz mark it as your answer thanks – Rajnish Kumar Apr 06 '16 at 15:09
  • ok, i will create the new question and post it here in 30min. i marked your question here as correct as it will point ppl in the right direction. thanks – psychok7 Apr 06 '16 at 15:16
  • here is the link for the new question please have a look http://stackoverflow.com/questions/36455974/screenscraping-a-datepicker-with-scrapy-and-selenium – psychok7 Apr 06 '16 at 15:43
  • hi i have updated the above answer and it never gives any stale element exception anymore please update line 29 as given by you in above comments – Rajnish Kumar Apr 07 '16 at 06:17