Jsoup with JavaScript dependent web page?

Question

So I'm having trouble formulating the correct syntax for selecting this element from a webpage. Here is what the path looks like on the Inspect Element Interface on Firefox

And here's what my current code looks like:

    Element prices = doc.select("body[class =en page-type-search page-type-group-shelf og ress] " +
                    "div#wrap " +
                    "div#main-wrap " +
                    "div#jalapeno-template " +
                    "div[class=zone zone3 wgrid-10of12 wgrid-6of8 wgrid-4of4] " +
                    "section#shelf-page " +
                    "div#shelf-thumbs " +
                    "div.shelf-thumbs " +
                    "div.price-current " +
                    "span.product-price-analytics").first();

            String priceOne = prices.attr("data-analytics-value");

And just to be incredibly clear, the attribute that I'm wanting is the 'data-analytics-value' because it gives an exact price.

I think that I have all the correct syntax so what am I doing wrong? When I run the program it gives me a nullPointerException. Any help is appreciated!

[Update] I changed princeOne to doc.toString() and its saying the the web browser is not running javascript and that JavaScript is required to view the walmart website, any work arounds?

Use a WebView to grab the rendered page, then parse with jsoup. — Frederic Klein, Nov 17 '16 at 07:04
Possible duplicate of [Android - Parse JS generated urls with JSOUP](http://stackoverflow.com/questions/39140121/android-parse-js-generated-urls-with-jsoup) — Frederic Klein, Nov 17 '16 at 07:04

score 0 · Accepted Answer · answered Nov 19 '16 at 00:04

After trying with no luck using Android's WebView, I accidentally found a solution in setting my userAgent, all I did was change the

    Jsoup.connect(url).get();

line to

    Jsoup.connect(url).userAgent("YOUR_USER_AGENT_HERE").get();

and it worked like a charm. Thanks for the reply anyway Fred!

Jsoup with JavaScript dependent web page?

1 Answers1