On August 29, 2009, in PHP, by Vaibhav

Last night I was trying to use curl for data mining in PHP. The website was built in ASPX and used Session. As we all know has server side controls whose state is stored in the hidden __VIEWSTATE variable in the form, I had to send this VIEWSTATE variable as my post variable to get the desired response.

After trying for a few hours with no success I was wondering what was wrong. After some research I came across some article that suggested using LiveHttpHeaders plugin in Firefox to view what exact request is being sent. Then after looking at the Request Header I found that the VIEWSTATE needs to be URL encoded !! and there I got my desired response instantly.

To conclude if we send POST parameters with CURL make sure you encode them and send in the proper order.

Checkout my other post CURL Class that will help you write code to send requests to pages that need cookies.

Also checkout Convert HTML to well-formed XML document (Clean HTML) with SgmlReader or php_tidy post to get insight on how to extract information from HTML.