World of Warcraft Armory xml Grabber with cURL - Page 4

sketchMedia · 03-09-2009, 12:49 PM

I posted some code to do this a few posts back:
World of Warcraft Armory xml Grabber with cURL

I'll explain it.
First off i believe you need to send a user-agent through to the armory otherwise it wont return the correct XML file (correct me if I'm wrong) so we need to send out the correct HTTP header.
By far the easiest way to archive this is by creating a stream context and file_get_contents (as its 3rd param accepts a stream context).

I suppose you could do it by opening up a socket on port 80(fsockopen) and passing the headers through (fputs etc), but its all a bit of a faff.

Tanax · 03-09-2009, 01:53 PM

Quote:

Originally Posted by sketchMedia

An alternative to using curl would be:

PHP Code:


			
/* $ch = curl_init();

 curl_setopt ($ch, CURLOPT_URL, $url);

 curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);

 curl_setopt ($ch, CURLOPT_CONNECTTIMEOUT, 15);

 curl_setopt ($ch, CURLOPT_USERAGENT,  self::BROWSER);



 $url_string = curl_exec($ch);

 curl_close($ch);*/

$opts = array(

    'http'=>array(

        'method'=>"GET",

        'header'=>"User-Agent: ".self::BROWSER." \r\n"

    )

);

$url_string = file_get_contents($url,false,stream_context_create($opts));

return simplexml_load_string($url_string);

But stream context create doesn't work on secured pages, such as for example the guild-login part or the account management page?

Or am I wrong?

Salathe · 03-09-2009, 03:02 PM

Quote:

Originally Posted by Tanax

But stream context create doesn't work on secured pages, such as for example the guild-login part or the account management page?

Or am I wrong?

The stream contexts will work on any of PHPs stream wrappers (file, http, ftp, php, phar, to name a few) provided you give the proper options/parameters.

Tanax · 03-09-2009, 08:30 PM

Quote:

Originally Posted by Salathe

The stream contexts will work on any of PHPs stream wrappers (file, http, ftp, php, phar, to name a few) provided you give the proper options/parameters.

Are you sure it works with secured pages(https)?

Check this thread, it's kinda related to this:
Stream problem

Salathe · 03-09-2009, 10:00 PM

Quote:

Originally Posted by Tanax

Are you sure it works with secured pages(https)?

Yes, I regularly access remote pages over SSL with cURL and file_get_contents. Provided your server has OpenSSL, there's really not all that much difference between using HTTP and HTTPS (in the context of retrieving web documents with PHP).

Tanax · 03-09-2009, 10:27 PM

Quote:

Originally Posted by Salathe

Yes, I regularly access remote pages over SSL with cURL and file_get_contents. Provided your server has OpenSSL, there's really not all that much difference between using HTTP and HTTPS (in the context of retrieving web documents with PHP).

Ah yes - with cURL. But sketchmedia said you could do this without cURL..

And if I misunderstood you, and you meant that you could do this without cURL(logging in on a secured page and retrieve the content), then what is the error with my code?

Salathe · 03-10-2009, 12:20 AM

You misunderstood, it's perfectly possible with file_get_contents and other such non-curl functions. As for the error you're experiencing, it's quite plain; the connection cannot be established, probably because you don't have OpenSLL compiled into PHP (the --with-openssl flag).

Tanax · 03-10-2009, 01:05 AM

Oh I see.. I checked out php.net on how to install it, but it only says how to install it on PHP, but I use WAMP(which has PHP build-in), so how would I install it on that?

Thanks

SawGore · 03-10-2009, 09:41 AM

PHP Code:


			
$opts = array(
            'http'=>array(
            'timeout'=> 5,
            'method'=>"GET",
            'header'=>"User-Agent: ".self::BROWSER." \r\n"
            )
        );
        $this->content = @file_get_contents($url,false,stream_context_create($opts));

This is a part of class. After the most dificult way, is to cached datas. return error with cache, with no cache, no error cache must be update, no update, …

nota: short timeout for speed page loading and @ for no warning

commonmind · 03-10-2009, 06:09 PM

Okay, I'm unable to view the data from using this bit of code:

PHP Code:


			
$url = 'http://www.wowarmory.com/guild-info.xml?r=YOUR+SERVER&n=YOUR+GUILD&p=1';

    $ch = curl_init();

    curl_setopt ($ch, CURLOPT_URL, $url);

    curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);

    curl_setopt ($ch, CURLOPT_USERAGENT,  "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1");

    

    $rosterxml = curl_exec($ch);

I'm guessing I'm making some fundamental mistake here, so let me explain exactly what I'm doing.

I'm inserting the above code within php tags, using Dreamweaver, so that it appears as follows:

PHP Code:


			
<?php 

$url = 'http://www.wowarmory.com/guild-info.xml?r=YOUR+SERVER&n=YOUR+GUILD&p=1';

    $ch = curl_init();

    curl_setopt ($ch, CURLOPT_URL, $url);

    curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);

    curl_setopt ($ch, CURLOPT_USERAGENT,  "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1");

    

    $rosterxml = curl_exec($ch);

?>

After which point I save the document and view it in a browser. Now, I don't see any information displayed (which I suppose is the way it works) but even after viewing the source, I still see nothing. No data of any sort, save my raw HTML tags.

I'm still very new at this, so I'm almost 100% certain that I'm simply doing something wrong, and missing what is probably an obvious step for the rest of you.

codeguy · 03-10-2009, 06:42 PM

Your using php code to save the contents of $rosterxml to another document on the webserver? Post the full code if you can.

Are you getting any errors? Was php compiled with curl?

commonmind · 03-10-2009, 07:00 PM

And that's probably where I'm going wrong, because I'm literally just inputting that code (with the correct URL of my guild/server, of course) into the editor, saving it, and trying to view it.

To say I'm a novice would be an understatement. And again, excuse the extreme lack of basic knowledge here, I'm really trying to understand how this works, beyond simply getting it to function as others have on their respective guild sites.

codeguy · 03-10-2009, 07:19 PM

Quote:

Originally Posted by commonmind

And that's probably where I'm going wrong, because I'm literally just inputting that code (with the correct URL of my guild/server, of course) into the editor, saving it, and trying to view it.

To say I'm a novice would be an understatement. And again, excuse the extreme lack of basic knowledge here, I'm really trying to understand how this works, beyond simply getting it to function as others have on their respective guild sites.

If your not getting an error message when browsing to your php page, I'll assume you have curl compiled into php.

Your code doesn't look wrong, however, you never output the variable holding the results from the webserver. If the code you posted is exactly and completely your code, I understand why you see nothing.

Try this:

Code:

<?php 
$url = 'http://www.wowarmory.com/guild-info.xml?r=YOUR+SERVER&n=YOUR+GUILD&p=1';
    $ch = curl_init();
    curl_setopt ($ch, CURLOPT_URL, $url);
    curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt ($ch, CURLOPT_USERAGENT,  "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1");
    
    $rosterxml = curl_exec($ch);
    
    echo $rosterxml;  #ADDED
?>

commonmind · 03-10-2009, 09:36 PM

Worked beautifully. Viewing the source gives me the expected data; now, getting it to display on the page...that's the next obstacle...

Yinx · 03-10-2009, 09:46 PM

As mentioned before, all the info you need is in this forum topic.
Read through it, there are numerous examples.

The way I see it, you are trying to get someone to make it entirely complete for you.

Tanax · 03-10-2009, 09:59 PM

Quote:

Originally Posted by Yinx

The way I see it, you are trying to get someone to make it entirely complete for you.

And that's the difference between a newbie and a noob

commonmind · 03-10-2009, 11:37 PM

Quote:

Originally Posted by Yinx

As mentioned before, all the info you need is in this forum topic.
Read through it, there are numerous examples.

The way I see it, you are trying to get someone to make it entirely complete for you.

I'm sorry you feel that way; I've spent a good deal of time attempting to learn as much as possible, with the intention of doing it myself. Unfortunately, my aptitude for this sort of thing began to waver after 30, and even some of the explanations provided aren't making a lot of sense to me. I'm guessing this comes from the fact that the majority of folks posting here have some general knowledge which I happen to lack.

If I could make an analogy, it's similar to someone listening in on a group of musicians discussing advanced musical theory, without knowing the rudiments. If you don't understand the fundamentals, the other concepts are confusing. In other words, despite the information being there (and maybe I did miss something which was more clear; my brain is a honestly a little fatigued) I don't understand what to do with it.

As I said before, I apologize if I derailed the thread or if I'm coming off as lazy or expectant, because at this point I do genuinely want to understand how this works, and I have spent the last couple of days with my nose buried in PHP tutorials (hence the fatigue) trying to do just that.

codeguy · 03-11-2009, 12:01 AM

From my perspective, commonmind, you made a very decent attempt at doing this yourself, using the help of this thread.

Disregard the disrespectful flaming. You did quite well. You had a very simple omission in your code, and asked for help, appropriately.

Unfortunately, some people think stating your objective is asking someone to do it for you. From your posts, I got the impression you really wanted to know how to do it and how its working, not for someone to do it for you. I appreciate that.

Tanax · 03-11-2009, 12:12 AM

Quote:

Originally Posted by codeguy

From my perspective, commonmind, you made a very decent attempt at doing this yourself, using the help of this thread.

Disregard the disrespectful flaming. You did quite well. You had a very simple omission in your code, and asked for help, appropriately.

Unfortunately, some people think stating your objective is asking someone to do it for you. From your posts, I got the impression you really wanted to know how to do it and how its working, not for someone to do it for you. I appreciate that.

I hope you don't include me in those people xD

commonmind · 03-11-2009, 01:35 AM

Thanks codeguy. And no harm done, really. I wasn't trying to ruffle feathers or anything like that, and hopefully when all is said and done I can add something constructive to the community in some way. This is a great site, with lots of fantastic resources, and I definitely don't want to outstay my welcome.