Last night I was trying to use curl for data mining in PHP. The website was built in ASPX and used Session. As we all know ASP.net has server side controls whose state is stored in the hidden __VIEWSTATE variable in the form, I had to send this VIEWSTATE variable as my post variable to get the desired response.
After trying for a few hours with no success I was wondering what was wrong. After some research I came across some article that suggested using LiveHttpHeaders plugin in Firefox to view what exact request is being sent. Then after looking at the Request Header I found that the VIEWSTATE needs to be URL encoded !! and there I got my desired response instantly.
To conclude if we send POST parameters with CURL make sure you encode them and send in the proper order.
Checkout my other post CURL Class that will help you write code to send requests to pages that need cookies.
Also checkout Convert HTML to well-formed XML document (Clean HTML) with SgmlReader or php_tidy post to get insight on how to extract information from HTML.
I had a similar problem when using CURL but I’m one step behind. How do you capture the initial VIEWSTATE and EVENTVALIDATION fields to url encode back in the POST response?
In my senario I used hard-coded VIEWSTATE variable for the first request and then I read the VIEWSTATE value from the response.
If you dont know the VIEWSTATE value or it is dynamic then in this senario you simply send the GET request without VIEWSTATE. You will get the VIEWSTATE in the response and all other parameters. You can use php_tidy extension to parse+clean the html and use xquery to fetch the required variables. Hope that helps.
Vaibhav,
Could you tell me the exact version of cURL/php you use. I started to experience a problem with my script once I upgraded my OS, so I supposed that’s an issue with cURL’s POST and 302 redirect after that. Also, could you paste me some part of your code. Thanks!
Hi!
Im interested in the same as Opala. A brief of the code would be nice! 🙂
Hello,
thanks for your hints, I have exactly the same problem. A code example would be very appreciated. Thank you
I would love to see a demo of you code. Have you some examples of how you have done the encode on _viewstate and others?
Thanks.
I ran into an error: “Validation of viewstate MAC failed”. Basically, an invalid viewstate, although i have rawurlencoded all data.
Have you run into this??
I have seen this error usually when the website is served from a Web Farm (multiple IIS Server). It doesnt recognize the ViewState because of incorrect keys configured on the server. You have to make sure you send all parameters : __VIEWSTATE, __EVENTTARGET, __EVENTARGUMENT.
Yes, I see what you’re saying now, and I’m sending the value(s) sent by their __doPostBack js function, but the result is the same.
Maybe there’s just something I’m doing wrong, I’ll try using your CURL class to help clear out anything I might be doing incorrectly.
Thanks