Proxifying file_get_contents [URGENT!]

clookid · 09-25-2007, 01:39 AM

PHP Code:


			
<?php



//Enter Password:



$password = "passwordgoeshere";



//Enter Submission Information:



$url = "http://www.ihowd.com";

$description = "I%20Howd";

$note = "The%20only%20resource%20you'll%20ever%20need.%20I%20Howd%20is%20jam%20packed%20full%20of%20how%20to%20articles.<br />";

$tags = "how%20to,%20articles";



//Edit Usernames:



echo file_get_contents("https://clooscript01:$password@api.del.icio.us/v1/posts/add?url=$url&description=$description&extended=$note&tags=$tags&shared=yes");

echo "<a href='$url' alt='$description'>$description</a> has successfully been submitted to <a href='http://del.icio.us/clooscript01'>clooscript01</a>.<br />";

echo file_get_contents("https://clooscript02:$password@api.del.icio.us/v1/posts/add?url=$url&description=$description&extended=$note&tags=$tags&shared=yes");

echo "<a href='$url' alt='$description'>$description</a> has successfully been submitted to <a href='http://del.icio.us/clooscript02'>clooscript02</a>.<br />";

echo file_get_contents("https://clooscript03:$password@api.del.icio.us/v1/posts/add?url=$url&description=$description&extended=$note&tags=$tags&shared=yes");

echo "<a href='$url' alt='$description'>$description</a> has successfully been submitted to <a href='http://del.icio.us/clooscript03'>clooscript03</a>.<br />";

echo file_get_contents("https://clooscript04:$password@api.del.icio.us/v1/posts/add?url=$url&description=$description&extended=$note&tags=$tags&shared=yes");

echo "<a href='$url' alt='$description'>$description</a> has successfully been submitted to <a href='http://del.icio.us/clooscript04'>clooscript04</a>.<br />";

echo file_get_contents("https://clooscript05:$password@api.del.icio.us/v1/posts/add?url=$url&description=$description&extended=$note&tags=$tags&shared=yes");

echo "<a href='$url' alt='$description'>$description</a> has successfully been submitted to <a href='http://del.icio.us/clooscript05'>clooscript05</a>.<br />";



and so on...



?>

I am looking at using del.icio.us's API to make a mass del.icio.us submission script. Unfortunately, I am unsure of how to use a proxy server on top of the file_get_contents() function. If anyone were willing to help me, that would be great!

Wildhoney · 09-25-2007, 01:42 AM

I don't believe there's any way to add proxy support to the native file_get_contents(), but I've wrote you a function below, file_get_contents_proxy() that adds proxy support via the cURL library which will need to be enabled.

Note: Really tired so it may not even work :( ! Someone else will be able to help you from this if not.

PHP Code:


			
function file_get_contents_proxy($szURL, $szProxy, $iProxyPort = 8080)
{
    $pCurl = curl_init($szURL);
    
    curl_setopt(CURLOPT_PROXY, $szProxy);
    curl_setopt(CURLOPT_PROXYPORT, $iProxyPort);
    
    curl_setopt(CURLOPT_FOLLOWLOCATION, true);
    curl_setopt(CURLOPT_RETURNTRANSFER, true);
    
    return curl_exec($pCurl);
}

$szURL = 'http://www.google.com/';
$szProxy = 'myProxy.com';

file_get_contents_proxy($szURL, $szProxy);

clookid · 09-25-2007, 01:46 AM

PHP Code:


			
<?php

$servers = array("https://clooscript01:$password@api.del.icio.us/v1/posts/add?url=$url&description=$description&extended=$note&tags=$tags&shared=yes");
$proxies = array("192.17.239.250:3127");

$password = "********";

$title = "IP Secured";

$url = "http://www.ipsecured.info";
$description = "IP%20Secured";
$note = "IP%20Secured%20--%20you%20no%20longer%20need%20to%20worry%20about%20filters.";
$tags = "ip%20secured,%20proxy";

function file_get_contents_proxy($servers, $proxies)
{
$pCurl = curl_init($servers);
curl_setopt($pCurl, CURLOPT_PROXY, $proxies);
curl_setopt($pCurl, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($pCurl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($pCurl, CURLOPT_SSL_VERIFYPEER, false);
curl_exec($pCurl);
}

for($i=0; $i<count($servers); $i++) {
file_get_contents_proxy($servers[$i], $proxies[$i]);
}

echo "<a href='$url' alt='$title'>$title</a> has successfully been submitted to <a href='http://del.icio.us/clooscript01'>clooscript01.<br />";

?>

I have been working at the script for a while and the above works. I would like to know how to add a second url / proxy. :)

Wildhoney · 09-25-2007, 10:49 AM

Not sure what you're trying to do but how about something like this:

PHP Code:


			
<?php
    
    function file_get_contents_proxy($szURL, $szProxy, $iProxyPort = 8080)
    {
        $pCurl = curl_init($szURL);
        
        curl_setopt($pCurl, CURLOPT_PROXY, $szProxy);
        curl_setopt($pCurl, CURLOPT_PROXYPORT, 8080);
        
        curl_setopt($pCurl, CURLOPT_FOLLOWLOCATION, true);
        curl_setopt($pCurl, CURLOPT_RETURNTRANSFER, true);
        
        return curl_exec($pCurl);
    }
    
    
    $aServers = array    (
                            array    (
                                        'ServerAddr' => 'http://www.deli.icio.us/1/',
                                        'ProxyAddr' => '192.17.239.250',
                                        'ProxyPort' => 3127,
                                    ),
                                    
                            array    (
                                        'ServerAddr' => 'http://www.deli.icio.us//2/',
                                        'ProxyAddr' => '192.168.0.1:8080',
                                        'ProxyPort' => 8080,
                                    )
                        );
                        
    for($iIndex = 0; $iIndex < count($aServers); $iIndex++)
    {
        file_get_contents_proxy($aServers[$iIndex]['ServerAddr'], $aServers[$iIndex]['ProxyAddr'], $aServers[$iIndex]['ProxyPort']);
    }

?>

Salathe · 09-25-2007, 12:03 PM

Just to clarify, there's no need to use CURL in order to pass the request through a proxy but it's likely the fasted method.

I've coded up a quick and dirty class which might help. The file_get_contents_proxy method shows you how to use a proxy in conjunction with PHP's file_get_contents function (key point: it uses a context).

Also, it seems that a number of your proxies weren't allowing SSL communication so that's why I've used the same proxy in my sample class. But you can use any that you find to work.

Here's the class, sorry it's quite long and messy but *touch wood* it works for me.

PHP Code:


			
<?php

class DeliciousSpam
{

    private $m_aUsers = array();
    private $m_aSubmissions = array();
    
    /*
        Add a user to the queue of users to whom urls will be submitted.
    */
    public function addUser($username, $password, $proxy)
    {
        $this->m_aUsers[$username] = array(
            'username' => $username,
            'password' => $password,
            'proxy'    => $proxy);
    }
    
    /*
        Adds a site/url which will be submitted to del.icio.us
        The optional $options array should contain an associative
        array of fields which you want to send alongside the required
        $url and $description fields.
    */
    public function addSubmission($url, $description, $options = array())
    {
        $params = array_merge($options, array(
            'url'         => $url,
            'description' => $description));
        $this->m_aSubmissions[$url] = $params;
    }
    
    /*
        Submit our sites/submissions, cycling through each user 
    */
    public function submit()
    {
        $aResults = array();
        
        if (empty($this->m_aUsers) || empty($this->m_aSubmissions))
        {
            return $aResults;
        }
        
        foreach ($this->m_aSubmissions as $aRequest)
        {
            foreach ($this->m_aUsers as $aUser)
            {    
                $szResult = $this->_call($aRequest, $aUser);
                    
                if (false !== strpos($szResult, 'code="done"'))
                {
                    $szResult = 'SUCCESS';
                }
                else
                {
                    $szResult = 'FAILED';
                }
                
                $aResults[$aRequest['url']][$aUser['username']] = $szResult;
                
                // Del.icio.us does not like more than one request per second
                // so we will sleep for a while between each request.
                sleep(1);
            }
        }
        
        return $aResults;
    }
    
    /*
        Send an API call to del.icio.us
    */
    private function _call($aRequest, $aUser)
    {
        // Build full URL for the API request 
        $szURL = sprintf(
            'https://%s:%s@api.del.icio.us/v1/posts/add?%s',
            $aUser['username'],
            $aUser['password'],
            http_build_query($aRequest)
            );
    
        // Make the request (turn error_reporting off to skip failures)
        $iErrorLevel = error_reporting(0);
        $szResponse = isset($aUser['proxy']) 
                      ? $this->file_get_contents_proxy($szURL, $aUser['proxy'])
                      : file_get_contents($szURL, false);
        error_reporting($iErrorLevel);
        
        // Return the response string, or false on error 
        return $szResponse;
    }
    
    /*
        Function to fetch the contents of an URL 
        via a proxy server.
    */
    private function file_get_contents_proxy($szURL, $szProxy)
    {
        set_time_limit(12); // Push back the timeout for the PAGE
        
        // Define the context for file_get_contents 
        $aContext = array(
            'http' => array(
                'method'          => 'GET',
                'proxy'           => 'tcp://'.$szProxy,
                'request_fulluri' => true,
                'timeout'         => 10, // Timeout for fetching URL 
                'max_redirects'   => 1
                )
            );
        $cxContext = stream_context_create($aContext);
        $sFile = file_get_contents($szURL, false, $cxContext);
        return $sFile;
    }
}

And how to use it:

PHP Code:


			
// Create our mighty spam beast
$spam = new DeliciousSpam();

// Add a series of users and associated proxies
$spam->addUser('clookid10', 'sandb0x', '213.114.118.44:8080');
$spam->addUser('clookid11', 'sandb0x', '213.114.118.44:8080');

// Add a series of urls to spam/submit to del.icio.us.
// The substr(...) part is just to provide different URLs each time.
$spam->addSubmission(
    'http://www.ihowd.com?TFL&'.substr(md5(time()), 0, 5),
    'I Howd',
    array('extended' => "The only resource you'll ever need. Howd is jam packed full of how to articles.",
          'tags'     => 'how to, articles',
          'shared'   => 'yes'));
$spam->addSubmission(
    'http://www.talkphp.com/showthread.php?t=1222&rnd='.substr(md5(time()), 0, 5),
    'Proxifying file_get_contents [URGENT!] - TalkPHP',
    array('extended' => 'I am unsure of how to use a proxy server on top of the file_get_contents() function. If anyone were willing to help me, that would be great!',
          'tags'     => 'talkphp, clookid',
          'shared'   => 'yes'));


// Submit and show some basic output 
header('Content-Type: text/plain');
var_dump($spam->submit());

?>

clookid · 09-26-2007, 12:15 AM

@Wildhoney - A blank page is loaded, but none of my del.icio.us pages update.

@Salathe - I receive different error messages, depending on the version of PHP that is running.

PHP 4:

PHP Code:


			
Parse error: syntax error, unexpected T_STRING, expecting T_OLD_FUNCTION or T_FUNCTION or T_VAR or '}' in /home/ipsecure/public_html/test.php on line 6

PHP 5:

PHP Code:


			
array(2) {
  ["http://www.ihowd.com?TFL&05d4f"]=>
  array(2) {
    ["clooscript01"]=>
    string(6) "FAILED"
    ["clooscript02"]=>
    string(6) "FAILED"
  }
  ["http://www.talkphp.com/showthread.php?t=1222&rnd=05d4f"]=>
  array(2) {
    ["clooscript01"]=>
    string(6) "FAILED"
    ["clooscript02"]=>
    string(6) "FAILED"
  }
}

I am awfully sorry, and thank you for all of your attempts. Please keep it up. :)

Salathe · 09-26-2007, 09:08 AM

Change the proxies (I did notice lots of yours wouldn't accept the connection, or would time out) and/or the timeout values.

clookid · 09-26-2007, 07:49 PM

Okay, I have just tried another 20 from my proxy list, and it still hasn't worked.

I have also tried 10 from http://www.publicproxyservers.com/page1.html and they are producing the same results. Thank you. :)

Salathe · 09-26-2007, 08:04 PM

Can you query the del.icio.us API without using a proxy server?

clookid · 09-27-2007, 05:22 AM

The issue has been resolved. Thank you for all of your help. :)

soma56 · 06-10-2010, 12:56 PM

I can see that this thread is a few years old. However, it turned up as one of the only results for connecting a proxy to file_get_contents. I've been learning php for the last two weeks and I've been trying a few experiments. For simplicities sake here's a little program I put together that uses Yahoo, cycles through a set amount of Yahoo pages and searches each page for a word.

PHP Code:


			
<?php



//assign variable to query

$query = "The colors of Serps";



//assign integer to starting Yahoo search page

$page = 01; 

            

//assign variable to Yahoo Search page

$yahoo = "http://ca.search.yahoo.com/search?&b=$page&p=";



//define pattern to search for

$pattern= "/(red)(blue)/"; 

                            

//Get webpage contents

$resultspage = file_get_contents("http://ca.search.yahoo.com/search?&b=$page&p=".urlencode($query)); 



// create while loop to cycle through pages

    while (($page <= 100)){

    usleep(1000000);



// Search for your pattern in Serp 

if (!empty($resultspage)) {

  $res = preg_match_all($pattern, $resultspage, $matches);

      if ($res) {

        foreach(array_unique($matches[0]) as $pattern) {

              echo $pattern . "<br />". PHP_EOL;

            flush();

            ob_flush();

            usleep(50000);

            }

          } 

    $page = $page + 10;

}

    }



echo "<br />";

echo "PROGRAM END<br /><br />";

exit;



?>

I put this together myself and as I've only been learning PHP in the last two weeks I think I've come a long way. What happens is Yahoo eventually returns a '999' error and temporarily blocks your IP when you make too many requests in a short time and I can understand why. That being said the only logical solution would be to have the file_get_contents function go through a proxy.

I have a subscription to a page that I log into that gives me access to a simple page of proxies updated with new ones every second. The conveniently comes out in the following format:

proxy1:portA
proxy2:portB
proxy3:portC

I'll be reviewing the earlier responses in this thread to see if I can figure out how to do this.

Essentially I want the script to:

1> Auto log into my proxy page
2> Grab five random proxies (or top proxies)
3> Use them to make requests as per the script above
4> Loop back to "2" and get more proxies

I would think this would eliminate the block by Yahoo has the results are coming from multiple IPs. Am I on the right path?

Salathe · 06-10-2010, 07:32 PM

Hi soma56, welcome to TalkPHP.

Yes, it looks like you're thinking along the right lines if you do want to avoid them blocking of those pages. However, there might be other routes that you can use. Note that Yahoo! has a bunch of special services (e.g. BOSS) designed for being accessed by scripts, providing programmatic access to things like the search results that you're wanting; this would likely avoid the need for proxies entirely.

soma56 · 06-10-2010, 07:57 PM

Thank you. I'm taking a crash course on cURL right now. I've have a look a BOSS and follow up in a new thread.