World of Warcraft Armory xml Grabber with cURL - Page 2

sketchMedia · 01-30-2009, 09:53 PM

Looks good mate :)

dont forget the quotes on the string arguments:

PHP Code:


			
$armory = new armory(character, $realmname, NULL, $charname, NULL); 
//should be
$armory = new armory('character', $realmname, NULL, $charname, NULL);

It works eitherway but the latter doesnt throw a PHP warning about an undefined constant.

SaintIsaiah · 01-30-2009, 10:15 PM

Good point. Took care of that. Here's a skinned version where it's part of the site. Just need to do some layout work and it should be good to go.

http://hellfireclub.org/character.php?name=Haiasi

Krik · 02-02-2009, 06:22 AM

Well this looks to be the best place to get an answer to a question that has been plaguing me.

While I did find some interesting stuff prior to this most of it is fairly basic and straight forward and what I am looking to do is take it one step further.

The question is how to get at the bank logs?

Obviously this requires logging in. I have tried using CURLOPT_POSTFIELDS, CURLOPT_HTTPAUTH, CURLOPT_COOKIEJAR, CURLOPT_COOKIEFILE and then submitting the data. As well I have tried submitting data straight to the log in page to see if it would let me in and no luck there either. Of course I was forced to get an authenticator so that may be complicating it.

I am hoping a few of you have played around with it more and have a trick up your sleeves.

SaintIsaiah · 02-02-2009, 12:05 PM

It is highly possible to get the guild bank logs as that is something I've wanted for myself. However, unless you're logged into the armory and your account has a character in the guild, you cant see it. You could hypothetically embed your username and password in the script to do this, but that will create a security issue for yourself. Unless you make your own form to curl the guild bank requiring a login each time, you're kind of SOL.

Krik · 02-02-2009, 06:54 PM

Yes the idea is to make it so an administrator logs in to the guild web site. Then goes to bank log update page and enters the WoW login info and, bang, it grabs the bank logs automatically and stores them on the data base.

I have used cURL for ton of featured on the site. And I can go and manually download a copy of the XML from the bank log and parse it into the data base. Just cannot get the login to work so it can be automated for the less computer literate administrators.

I used cURL to view the non-XML source for their login page so as to see their forms as well have used the XML data they generate as the page source to find out what all the form fields are that need to be submitted. And every thing I have so far tried to submit is not allowing the login.

I am hoping some one has done this and can share their tricks with us.

SaintIsaiah · 02-03-2009, 01:50 AM

Unfortunately, this goes a bit passed my current knowledge of PHP. But I will try to work with some functions and post a solution here if I can find one.

Krik · 02-04-2009, 11:31 PM

EDIT
Posted a few hours too early.

PHP Code:


			
<html>

<body>

<?php

if (isset($_POST['submit'])) {



$username = $_POST['username'];

$password = $_POST['password'];

$authenticate = $_POST['authenticate'];



$postvars = "accountName=" . $username . "&password=" . $password;



define(US_LOGIN_URL, 'https://www.blizzard.com/login/login.xml?referer=http%3A%2F%2Fwww.wowarmory.com%2Fvault%2Fguild-bank-log.xml%3Fr%3DMy%2BRealm%26n%3DMy%2BGuild&loginType=com');





    $ch = curl_init(US_LOGIN_URL);

    curl_setopt($ch, CURLOPT_POST, 1);

    curl_setopt($ch, CURLOPT_POSTFIELDS, $postvars);

    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);

    curl_setopt($ch, CURLOPT_HEADER, 0);

    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

    curl_setopt ($ch, CURLOPT_COOKIEJAR, 'cookie.txt');

    $returnedPage = curl_exec($ch);

    curl_close($ch);





    if (strstr($returnedPage, "Invalid account name or password")) {

        echo "FAIL: Invalid account name or password<br>";

    }

    elseif (strstr($returnedPage, "Account name required")) {

        echo "FAIL: Invalid account name or password<br>";

    }

    elseif (strstr($returnedPage, "Password required")) {

        echo "FAIL: Invalid account name or password<br>";

    }

    elseif (strstr($returnedPage, "Authenticator code required")) {

        echo "FAIL: Invalid Authenticator code<br>";

    }

    elseif (strstr($returnedPage, "Login failed")) {

        echo "FAIL: Invalid account name or password<br>";

    }

    elseif (strstr($returnedPage, "Authenticator Code")) {

        $postauth = "authValue=" . $authenticate;

        

        $ch = curl_init(US_LOGIN_URL);

        curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");

        curl_setopt($ch, CURLOPT_POST, 1);

        curl_setopt($ch, CURLOPT_POSTFIELDS, $postauth);

        curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);

        curl_setopt($ch, CURLOPT_HEADER, 0);

        curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

        curl_setopt ($ch, CURLOPT_COOKIEJAR, 'cookie.txt');

        $returnedPage = curl_exec($ch);

        curl_close($ch);

    

        echo $returnedPage;

    }

    else {

        echo $returnedPage;

    }

}

else {

?>

<form action="banklog.php" id="loginForm" method="post">



Account Name<input id="username" name="username" type="text" /><br>



Password<input id="password" name="password" type="password" /><br>



Authenticator<input id="authenticate" name="authenticate" type="text" /><br>



<input type="submit" name="submit" value="Submit">



</form>

<?php

}

?>

</body>

</html>

It's a bit sloppy but it works to get the bank logs open, I will clean it up more later. But I am not having any luck getting the XML out of it. I have parsed XML off other WoW pages but I can't off this I keep getting HTML and javascript. Any thoughts here.

Krik · 02-05-2009, 07:51 AM

Well it seems I keep progressing on this.

PHP Code:


			
<?php

if (isset($_POST['submit'])) {



$username = $_POST['username'];

$password = $_POST['password'];

$authenticate = $_POST['authenticate'];



$loginvars = "accountName=" . $username . "&password=" . $password;



define(WOW_LOGIN_URL, 'https://www.blizzard.com/login/login.xml?referer=http%3A%2F%2Fwww.wowarmory.com%2Fvault%2Fguild-bank-log.xml%3Fr%3DMy%2BRealm%26n%3DMy%2BGuild&loginType=com');



    $ch = curl_init(WOW_LOGIN_URL);

    curl_setopt($ch, CURLOPT_POST, 1);

    curl_setopt($ch, CURLOPT_POSTFIELDS, $loginvars);

    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);

    curl_setopt($ch, CURLOPT_HEADER, 0);

    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

    curl_setopt($ch, CURLOPT_COOKIEJAR, '../hidden/cookie.txt');

    $results = curl_exec($ch);

    curl_close($ch);





    if (strstr($results, "Invalid account name or password")) {

        echo "Invalid account name or password<br>";

    }

    elseif (strstr($results, "Account name required")) {

        echo "Invalid account name or password<br>";

    }

    elseif (strstr($results, "Password required")) {

        echo "Invalid account name or password<br>";

    }

    elseif (strstr($results, "Authenticator code required")) {

        echo "Invalid Authenticator code<br>";

    }

    elseif (strstr($results, "Login failed")) {

        echo "Invalid account name or password<br>";

    }

    elseif (strstr($results, "Authenticator Code")) {

        $loginauth = "authValue=" . $authenticate;

        

        $ch = curl_init(WOW_LOGIN_URL);

        curl_setopt($ch, CURLOPT_COOKIEFILE, '../hidden/cookie.txt');

        curl_setopt($ch, CURLOPT_POST, 1);

        curl_setopt($ch, CURLOPT_POSTFIELDS, $loginauth);

        curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1");

        curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);

        curl_setopt($ch, CURLOPT_HEADER, 0);

        curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

        curl_setopt($ch, CURLOPT_COOKIEJAR, '../hidden/cookie.txt');

        $results = curl_exec($ch);

        curl_close($ch);

    

        echo $results;

    }

    else {

        echo $results;

    }

}

else {

?>

<html>

<body>

<form action="banklog.php" id="loginForm" method="post">



Account Name<input id="username"name="username" type="text" /><br>



Password<input id="password" name="password" type="password" /><br>



Authenticator<input id="authenticate" name="authenticate" type="text" /><br>



<input type="submit" name="submit" value="Submit">



</form>

</body>

</html>

<?php

}

?>

My earlier mistake was in forgetting the CURLOPT_USERAGENT.

Now I am on the other side of this I can get it to work with my authenticator but I don't think it will work for those that don't have one as the CURLOPT_USERAGENT is not set in the first curl options. When I put it in there it I can't login with my authenticator.

Yinx · 02-06-2009, 11:59 PM

Anybody familiar with caching?
Wondering how to cache the displayed data for when the armory is offline as a example.

And to speed up the process of showing, like after it has reached a certain time it shows the cached information.

Krik · 02-07-2009, 07:00 AM

You parse the XML and store it in a database. Right now I have the last 5 months of my guilds bank logs stored in a database and the page for viewing them has many of the features of the armory with a few other options for sorting and displaying it added.

Actually the above scripts main purpose will be to allow administrators to log in and update the stored data as often as needed without needing know anything more than their WoW loging info.

On a side note if your just looking to have your browser hold the data you can set up you browser to store websites for use offline. Of course if the armory is just slow to get that data you will need to disconnect your PC from your ISP for it to work.

xtsu · 02-08-2009, 09:12 PM

This was just what I was looking for! However I do have a question or two. I was wondering how I can parse the achievement pages? I need to grab the Cooking, First Aid and Fishing level of my members and yet I can't seem to get it to work by back engineering the scripts on here.

This is what I have tried to do so far.

First I created a new condition in the function pull_xml If you insert the c=132 into the URL it will take you to that characters info page outlining certain achievements.

Code:

elseif( $this->query === 'skills' ){
        $url = 'http://www.wowarmory.com/character-statistics.xml?c=132&' . urlencode($this->server) . '&n=' . $this->guildie;
        
        }

After that I tried creating the call to that function:

Code:

$armory = new armory('skills', 'steamwheedle cartel', 'fallen legacy', 'xtsu', NULL);
$xml = $armory->pull_xml();

foreach($xml->category as $catg)
{
    $catg = reset($catg);
    echo $catg['name'],'<br />';
}

The above was to see if I could pull and list the different categories but no luck. I am not too sure what to try next and am seeking advice. Like I said I would like to be able to pull the numbers off this page.

Here is a hard link to the XML page that blizzard puts out and that would need to be parsed.

http://www.wowarmory.com/character-s...+Cartel&n=Xtsu

xtsu · 02-10-2009, 01:50 AM

PHP Code:


			
        $armory = new armory('roster', 'steamwheedle cartel', 'fallen legacy', NULL, NULL);

        $xml = $armory->pull_xml();  



foreach ($xml->guildInfo->guild->members->character as $char) {

    echo $char['name'] . "<br>";

}

The returns this error now:

Warning: Invalid argument supplied for foreach() in .../fallenlegacy/test3.php on line 96

so I thought it might be the cURL stuff soo I tried the other way and in addition to the above warning I also got this:

Warning: file_get_contents(http://www.wowarmory.com/guild-info....llen+legacy&p=) [function.file-get-contents]: failed to open stream: HTTP request failed! HTTP/1.1 503 Service Temporarily Unavailable in .../fallenlegacy/test3.php on line 81

Does that mean my webserver has a temp ban on it? Or where my webserver is located can't get to the armory?

Tanax · 02-11-2009, 02:09 PM

Quote:

Originally Posted by Krik

Well it seems I keep progressing on this.

PHP Code:


			
<?php

if (isset($_POST['submit'])) {



$username = $_POST['username'];

$password = $_POST['password'];

$authenticate = $_POST['authenticate'];



$loginvars = "accountName=" . $username . "&password=" . $password;



define(WOW_LOGIN_URL, 'https://www.blizzard.com/login/login.xml?referer=http%3A%2F%2Fwww.wowarmory.com%2Fvault%2Fguild-bank-log.xml%3Fr%3DMy%2BRealm%26n%3DMy%2BGuild&loginType=com');



    $ch = curl_init(WOW_LOGIN_URL);

    curl_setopt($ch, CURLOPT_POST, 1);

    curl_setopt($ch, CURLOPT_POSTFIELDS, $loginvars);

    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);

    curl_setopt($ch, CURLOPT_HEADER, 0);

    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

    curl_setopt($ch, CURLOPT_COOKIEJAR, '../hidden/cookie.txt');

    $results = curl_exec($ch);

    curl_close($ch);





    if (strstr($results, "Invalid account name or password")) {

        echo "Invalid account name or password<br>";

    }

    elseif (strstr($results, "Account name required")) {

        echo "Invalid account name or password<br>";

    }

    elseif (strstr($results, "Password required")) {

        echo "Invalid account name or password<br>";

    }

    elseif (strstr($results, "Authenticator code required")) {

        echo "Invalid Authenticator code<br>";

    }

    elseif (strstr($results, "Login failed")) {

        echo "Invalid account name or password<br>";

    }

    elseif (strstr($results, "Authenticator Code")) {

        $loginauth = "authValue=" . $authenticate;

        

        $ch = curl_init(WOW_LOGIN_URL);

        curl_setopt($ch, CURLOPT_COOKIEFILE, '../hidden/cookie.txt');

        curl_setopt($ch, CURLOPT_POST, 1);

        curl_setopt($ch, CURLOPT_POSTFIELDS, $loginauth);

        curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1");

        curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);

        curl_setopt($ch, CURLOPT_HEADER, 0);

        curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

        curl_setopt($ch, CURLOPT_COOKIEJAR, '../hidden/cookie.txt');

        $results = curl_exec($ch);

        curl_close($ch);

    

        echo $results;

    }

    else {

        echo $results;

    }

}

else {

?>

<html>

<body>

<form action="banklog.php" id="loginForm" method="post">



Account Name<input id="username"name="username" type="text" /><br>



Password<input id="password" name="password" type="password" /><br>



Authenticator<input id="authenticate" name="authenticate" type="text" /><br>



<input type="submit" name="submit" value="Submit">



</form>

</body>

</html>

<?php

}

?>

My earlier mistake was in forgetting the CURLOPT_USERAGENT.

Now I am on the other side of this I can get it to work with my authenticator but I don't think it will work for those that don't have one as the CURLOPT_USERAGENT is not set in the first curl options. When I put it in there it I can't login with my authenticator.

Keep working on it! I'm sure you will solve it

And when it's working, we can add it to the class

PHP Code:


			
$guildLog = $armory->pull_guildLog($username, $password, $authenticate);

Auth would ofcourse be optional, since not everyone has it.

Yinx · 02-12-2009, 02:58 PM

Cheers for the OP and everyone's else contribution.

Got my roster working and caching, http://resistance.pewpew.nl/members.php

Tanax · 02-12-2009, 03:14 PM

How did you solve the caching?

Looks good!

Yinx · 02-12-2009, 03:25 PM

Instead of using some database, which sounded so complicated and making more connections etc.
I went for a file based caching with the use of ob_start() and ob_end_flush().

PHP Code:


			
<?php
if ($_GET['name']) {
    $cachefile = 'cache/'.$_GET['name'].'.cache'; // different cache file for each member profile
} else {
    $cachefile = 'cache/roster.cache'; // main members page cache file.
}
$cachetime = 24 * 60 * 60; // hold the cache for 1 day before re-fetching data from armory.
if (file_exists($cachefile) && (time() - $cachetime < filemtime($cachefile))) {
    include($cachefile); // if cache is not older than 1 day include the cache file.
    echo "<!-- Cached ".date('jS F Y H:i', filemtime($cachefile))." -->\n";
    exit; // stop continuing the script if cache file is included.
}
ob_start(); // start buffering/caching up.
require_once("header.php");
?>
Your usual HTML and or PHP code here, in other words your dynamic content such as this armory member stuff.
<?php
require_once("footer.php");

$cachefp = fopen($cachefile, 'w');
fwrite($cachefp, ob_get_contents()); //write the buffered/cached data to the cache file.
fclose($cachefp);
ob_end_flush();
?>

Simple but seems to be working :)

Tanax · 02-12-2009, 04:09 PM

Great!

Just a question though. Say the armory goes offline, then this will still work because it's cached for the next 24 hours, great! What if the armory is offline when it's supposed to recache? Then it will cache everything with nothing.. and it won't update it until 24 hours later o.O

Armory won't be offline for 24 hours hopefully, but still, it could be an issue. Any idea? Perhaps try sockets and see if armory is online? Or something..

Yinx · 02-12-2009, 07:23 PM

Won't be a issue.
Else for a quick fix, just add some extra copy backup old cache and do a filesize check etc :D

Tanax · 02-12-2009, 08:18 PM

How come that won't be an issue?
Yea, but that is spacerequiring, wouldn't it be better to run a check if the armory is online?

Yinx · 02-12-2009, 09:29 PM

Don't know how limited you are or how heavy your site is.
For me it is like 10-13KB per cache file.