0

I am scraping a webpage. The webpage consists of 48 entries. After 48 entries it gives a Load more results button. I need to get all the products from this page. How can I do it? For scraping, I am using Python, LXML, and Requests Library.

I will be thankful for your Help.

My Code :

import requests
from lxml import html
home_page = requests.get('https://www.anntaylor.com/')
website_tree = html.fromstring(home_page.content)
categories = website_tree.xpath('//a[@role="menuitem"]')
def my_function(urls):
    category_page = requests.get(urls)
    category_tree = html.fromstring(category_page.content)
    product_data = category_tree.xpath('//div[@class="product-wrap"]') #//div[@class='product-wrap']
    
    for title in product_data:
        print(title.xpath('.//@href')[0])
        # //link[@rel="next"]
    
    print(len(product_data))
    print(urls)
for i in categories:
    urls = i.xpath('./@href')
    # print(urls[0])
    my_function(urls[0])
print(len(categories))
DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22
Haris Gul
  • 1
  • 1
  • 4

1 Answers1

1

The page you requested with your code has not a button, but a "loading more" info bar is shown when you scroll down. I think you want to load the full content of the page to parse the data of the products.

This post may help you: Scrape entire scrolling-load page with Python Requests

turkishce1
  • 31
  • 3