Scraping web pages (including AJAX) from a .NET solution

I hope this helps you . If you use the developer feature of your web browser (Firebug, Inspector, Developer Tools, ...) you should able to see the connections being made to retrieve the data through Ajax and the request and response headers being sent and received.
The request headers will contain the data being sent as well as the URL that's been request. The query string of the URL or the POST data would most likely contain a "start" or "next" or some time of parameter that identifies the start and number of results to return.

I hope this helps you . In general, in order to fake the ASP.NET web site to think that you actually pressed a button (in more general terms - performed a postback), you need to do the following:

I hope this helps you . First of all, scrapy docs are available at https://scrapy.readthedocs.org/en/latest/.
Speaking about handling ajax while web scraping. Basically, the idea is rather simple:

hop of those help? I'm having a challenge: , That situation doesn't exists.
Explanation:

it fixes the issue Alternatively to the selenium or webkit based approach, you can parse the javascript with a javascript code parser, like slimit. It definitely raises the complexity and reliability of the web-scraping since you go down to a bare hardcore metal with it - think about it as a "white box" approach as opposed to selenium based high-level "black box" one.
Here's the answer I've given for an exact same topic/problem you are asking about: