code-diesel » curl

Parallel cURL execution in PHP

sameer — Wed, 04 Aug 2010 07:15:28 +0000

cURL is a Swiss army knife of web content processing. Programmers use it for a number of things everyday. One interesting feature the cURL library offers that many programmers are unaware of is the parallel execution of requests.
curl has two major modes of request execution: ‘easy’ mode and the ‘multi’ mode. Most people use the ‘easy’ mode – in this mode when we issue multiple requests, the second request will not start until the first one is complete. This is known as synchronous execution, and this is the one we normally use. This means that if we have 100 requests to process, each will be processed in a linear manner, which could take a lot of time. This is where ‘multi’ mode comes to the rescue. In this mode all requests can be handled in parallel or asynchronously. And it can be quite handy and time saving on many occasions. The ‘multi’ or parallel mode is handled by the following curl functions:

curl_multi_init();
curl_multi_add_handle();
curl_multi_select();
curl_multi_exec();
curl_multi_getcontent();
curl_multi_info_read();
curl_multi_remove_handle();

Using parallel curl libraries

But there is one caveat when using the multi mode; it can be a little confusing. And as always, working with asynchronous code is not always easy, so I prefer to use classes developed by other people. Between the various libraries I’ve found on the net, pete wardens is the one I like. Below is a rough outline of how it works. You first initialize the class and specify the number of requests it should handle in parallel (10 here) along with some curl options.

$parallel_curl = new ParallelCurl(10, $curl_options);

Next we call the startRequestmethod, specifying the url to request and a callback function, which will be run when the request is processed. We can also pass a optional third parameter which will be handed over to the callback function. If there are any post fields, you can pass them as the fourth parameter in the startRequestmethod.

$parallel_curl->startRequest($search_url, 'on_request_done', $search);

Thats it, this is all that is required to execute parallel curl requests. Below is a broad overview of how to use the class.


 
require_once('parallelcurl.php');
 
$url1 = 'http://www.example.com/');
$url2 = 'http://www.example.com/');
$url3 = 'http://www.example.com/');
 
// This function gets called back for each request that completes
function on_request_done($content, $url, $ch, $search) {
...
}
 
$max_requests = 10;
 
$curl_options = array(
    CURLOPT_SSL_VERIFYPEER => FALSE,
    CURLOPT_SSL_VERIFYHOST => FALSE
);
 
$parallel_curl = new ParallelCurl($max_requests, $curl_options);
 
// Start 3 parallel requests. All three will be started simultaneously.
$parallel_curl->startRequest($url1, 'on_request_done');
$parallel_curl->startRequest($url2, 'on_request_done');
$parallel_curl->startRequest($url3, 'on_request_done');
 
$parallel_curl->finishAllRequests();
 
?>

The complete test script using the class to run Google search queries in parallel is shown below.

 
require_once('parallelcurl.php');
 
define ('SEARCH_URL_PREFIX', 'http://ajax.googleapis.com/ajax/services/\
                            search/web?v=1.0&rsz=large&filter=0');
 
// This function gets called back for each request that completes
function on_request_done($content, $url, $ch, $search) {
    $httpcode = curl_getinfo($ch, CURLINFO_HTTP_CODE);    
    if ($httpcode !== 200) {
        print "Fetch error $httpcode for '$url'\n";
        return;
    }
 
    $responseobject = json_decode($content, true);
    if (empty($responseobject['responseData']['results'])) {
        print "No results found for '$search'\n";
        return;
    }
 
    print "********\n";
    print "$search:\n";
    print "********\n";
 
    $allresponseresults = $responseobject['responseData']['results'];
    foreach ($allresponseresults as $responseresult) {
        $title = $responseresult['title'];
        print "$title\n";
    }
}
 
// The terms to search for on Google
$terms_list = array(
    "Paul", "Lena",
    "Lee", "Marie",
    "Tom", "Ada",
    "Herman", "Josephine",
);
 
if (isset($argv[1])) {
    $max_requests = $argv[1];
} else {
    $max_requests = 10;
}
 
$curl_options = array(
    CURLOPT_SSL_VERIFYPEER => FALSE,
    CURLOPT_SSL_VERIFYHOST => FALSE,
    CURLOPT_USERAGENT, 'Parallel Curl test script',
);
 
$parallel_curl = new ParallelCurl($max_requests, $curl_options);
 
foreach ($terms_list as $terms) {
    $search = '"'.$terms.' is a"';
    $search_url = SEARCH_URL_PREFIX.'&q='.urlencode($terms);
    $parallel_curl->startRequest($search_url,'on_request_done',$search);
}
 
$parallel_curl->finishAllRequests();
 
?>

Note: I have found the php curl multi mode execution to be inconsistent on Windows. But it works great on Linux (I’m using Ubuntu).

You can download the parallelCurl class from gitHub.

Free Geolocation API tool

sameer — Mon, 13 Apr 2009 04:33:15 +0000

NOTE: http://www.ip2location.com have done away with the free api access from their site. So the following sample code will no longer work. They now provide a free sample database on their site and also a complete paid version.

iplocationtools.com offers a free geolocation API that lets you query with an ip address and get the location details such as city, country, zip, latitude, longitude etc. The site also offers a free MySQL database for the same if you would like to install it on your server. I’ve used CURL to wrap the API access. The complete function with a sample query and the response is shown below.


 
   /**
    * Geolocation API access
    *
    * @param    string  $ip         IP address to query
    * @param    string  $format     output format of response
    *
    * @return   string  XML, JSON or CSV string
    */
    function get_ip_location($ip, $format="xml") {
 
        /* Set allowed output formats */
        $formats_allowed = array("json", "xml", "raw");
 
        /* IP location query url */
        $query_url = "http://iplocationtools.com/ip_query.php?ip=";
 
        /* Male sure that the format is one of json, xml, raw.
           Or else default to xml */
        if(!in_array($format, $formats_allowed)) {
            $format = "xml";
        }
 
        $query_url = $query_url . "{$ip}&output={$format}";
 
        /* Init CURL and its options*/
        $ch = curl_init();
        curl_setopt($ch, CURLOPT_URL, $query_url);
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
        curl_setopt($ch, CURLOPT_TIMEOUT, 15);
 
        /* Execute CURL and get the response */
        return curl_exec($ch);
 
    }
 
 
    $location_data = get_ip_location("209.85.153.104");
    print_r($location_data);
 
?>

Query response:

>
	>209.85.153.104>
	>OK>
	>US>
	>United States>
	>06>
	>California>
	>Mountain View>
	>94043>
	>37.4192>
	>-122.057>
>

The function by default returns a XML response, but you can also get a response in json or csv format, as shown below.

/* Get response in 'json' format' */
$location_data = get_ip_location("209.85.153.104", "json");

{
"Ip" : "209.85.153.104",
"Status" : "OK",
"CountryCode" : "US",
"CountryName" : "United States",
"RegionCode" : "06",
"RegionName" : "California",
"City" : "Mountain View",
"ZipPostalCode" : "94043",
"Latitude" : "37.4192",
"Longitude" : "-122.057",
}

/* Get response in 'csv' format' */
$location_data = get_ip_location("209.85.153.104", "raw");

209.85.153.104,OK,US,United States,06,California,
Mountain View,94043,37.4192,-122.057

You can further parse the returned XML using SimpleXML.


 
    $location_data = get_ip_location("209.85.153.104");
 
    $data = '';
 
    try {
        $data = @new SimpleXMLElement($location_data);
    }
    catch(Exception $e) {
        echo "Error parsing XML.";
    }
 
    foreach($data as $key=>$value)
        echo $key . " : " . $value . "\n";
 
?>

6 essential cURL commands for daily use

sameer — Tue, 24 Mar 2009 08:35:30 +0000

cURL is a command line tool for doing all sorts of interesting and essential URL manipulations and data transfers. The original goal of the program was to transfer files programmatically via protocols such as http, ftp, gopher, sftp, ftps, scp, tftp, and many others, via a command line interface. The main benefit of using the command line interface is that you can use the program in your Windows batch file or Linux shell scripts to automate many URL related processes. In this post you will see some essential things you can do using cURL.

1. Reading URLs

Read a plain URL.

curl http://www.google.com

Read a secured URL.

curl https://www.secure-site.com

Get a web page and store it in a file. The following for example will store the index page retrieved to the file savedpage.html

curl -o savedpage.html http://www.example.com/

Get a HTTP Basic authenticated page

curl -u username:password http://www.example.com/

Sometimes a page may redirect to another resource. By default CURL will not follow page redirections. To make CURL follow redirections use the -L option.

curl -L http://www.example.com/

2. Reading URL’s with variable GET parameters

You can also download pages with a variable GET parameter. For e.g take the following url:

http://example.com/pages.php?pageNo=35

The variable here is the pageNo parameter. You can download all the pages by adding a regular expression like parameter in the CURL url as given below.

curl -o pages#1.html http://example.com/pages.php?pageNo=[1-12]

This will download all the pages from page no 1 to page no 12 and save it to a corresponding file.

3. Reading document information

Show document headers only

curl --head http://www.google.com/

You can also use it on any specific resource.

curl --head http://www.google.com/logo_plain.jpg

Dump document headers to a file

curl --dump-header headers.txt http://www.google.com/

4. CURL and FTP

Get a FTP directory listing

curl ftp://username:password@example.com

To get the listing of a different directory append the directory name to the URL.

curl ftp://username:password@example.com/directory/

Upload a file to a remote directory using FTP

curl -T uploadfilename -u username:password ftp://sitename.com/myfile

The ‘uploadfilename’ file will be copied to the remote site and named ‘myfile’. If the destination filename is eliminated the file will be copied with the original name. By default the file will be copied to the root directory. To copy to some other directory specify the directory after the site name;e.g.

curl -T uploadfilename -u username:password 
          ftp://sitename.com/directory/myfile

5. To POST to a page.

You can also process a POST request using CURL. The data will use the application/x-www-form-urlencoded encoding. Lets say you have the following POST form in your page:

You can use the following CURL command to POST the request.

curl -d "item=bottle&category=consumer&submit=ok" 
           www.example.com/process.php

6. Referer & User Agent

HTTP requests may include a ‘referer’ field, which is used to tell from which URL the client got to this particular page. Many programs/scripts check the referer field of requests to check the source of the request. You can simulate the referer field by the following command.

 curl -e http://some_referring_site.com  http://www.example.com/

All HTTP requests may set the User-Agent field. It names what user agent or client that is being used. Many web applications use this information to decide how to display web pages or use it to track browser usage. You can impersonate a particular browser by the following method:

curl -A "Mozilla/5.0 (compatible; MSIE 7.01; Windows NT 5.0)" 
        http://www.example.com

There are many more options you can use with curl, the ones given above are just some you may require on a regular basis.