6 essential cURL commands for daily use


cURL is a command line tool for doing all sorts of interesting and essential URL manipulations and data transfers. The original goal of the program was to transfer files programmatically via protocols such as http, ftp, gopher, sftp, ftps, scp, tftp, and many others, via a command line interface. The main benefit of using the command line interface is that you can use the program in your Windows batch file or Linux shell scripts to automate many URL related processes. In this post you will see some essential things you can do using cURL.

1. Reading URLs

Read a plain URL.

curl http://www.google.com

Read a secured URL.

curl https://www.secure-site.com

Get a web page and store it in a file. The following for example will store the index page retrieved to the file savedpage.html

curl -o savedpage.html http://www.example.com/

Get a HTTP Basic authenticated page

curl -u username:password http://www.example.com/

Sometimes a page may redirect to another resource. By default CURL will not follow page redirections. To make CURL follow redirections use the -L option.

curl -L http://www.example.com/

2. Reading URL’s with variable GET parameters

You can also download pages with a variable GET parameter. For e.g take the following url:

http://example.com/pages.php?pageNo=35

The variable here is the pageNo parameter. You can download all the pages by adding a regular expression like parameter in the CURL url as given below.

curl -o pages#1.html http://example.com/pages.php?pageNo=[1-12]

This will download all the pages from page no 1 to page no 12 and save it to a corresponding file.

3. Reading document information

Show document headers only

curl --head http://www.google.com/

You can also use it on any specific resource.

curl --head http://www.google.com/logo_plain.jpg

Dump document headers to a file

curl --dump-header headers.txt http://www.google.com/

4. CURL and FTP

Get a FTP directory listing

curl ftp://username:password@example.com

To get the listing of a different directory append the directory name to the URL.

curl ftp://username:password@example.com/directory/

Upload a file to a remote directory using FTP

curl -T uploadfilename -u username:password ftp://sitename.com/myfile

The ‘uploadfilename’ file will be copied to the remote site and named ‘myfile’. If the destination filename is eliminated the file will be copied with the original name. By default the file will be copied to the root directory. To copy to some other directory specify the directory after the site name;e.g.

curl -T uploadfilename -u username:password 
          ftp://sitename.com/directory/myfile

5. To POST to a page.

You can also process a POST request using CURL. The data will use the application/x-www-form-urlencoded encoding. Lets say you have the following POST form in your page:

<form method="POST" action="process.php">
          <input type=text name="item">
          <input type=text name="category">
          <input type=submit name="submit" value="ok">
</form>

You can use the following CURL command to POST the request.

curl -d "item=bottle&category=consumer&submit=ok" 
           www.example.com/process.php

6. Referer & User Agent

HTTP requests may include a ‘referer’ field, which is used to tell from which URL the client got to this particular page. Many programs/scripts check the referer field of requests to check the source of the request. You can simulate the referer field by the following command.

 curl -e http://some_referring_site.com  http://www.example.com/

All HTTP requests may set the User-Agent field. It names what user agent or client that is being used. Many web applications use this information to decide how to display web pages or use it to track browser usage. You can impersonate a particular browser by the following method:

curl -A "Mozilla/5.0 (compatible; MSIE 7.01; Windows NT 5.0)" 
        http://www.example.com

There are many more options you can use with curl, the ones given above are just some you may require on a regular basis.

This site is a digital habitat of Sameer Borate, a freelance web developer working in PHP, MySQL and WordPress. I also provide web scraping services, website design and development and integration of various Open Source API's. Contact me at metapix[at]gmail.com for any new project requirements and price quotes.

13 Responses

1

Football Blog

April 24th, 2009 at 11:47 am

Exactly what ive been looking for.. cheers dude..

2

Glen Bremner-Stokes

December 22nd, 2009 at 2:24 am

Nice. Do you perhaps know how to use curl to simulate a cross-domain http request ?

I am trying to set my tomcat server to return Access-Control-Allow-Origin: * in the headers. I found something that says it does it but I want to confirm that the header is set.

3

evm

January 13th, 2010 at 2:27 am

great post, thanks a mil!

4

Internet quotas

June 2nd, 2010 at 3:40 am

Thanks, I’m a totally nubie at Curl. Every other page I’ve looked at has too much details and confuses me. I’m now clear that some lines of code are there to simulate a real user on the destination website.

5

Zuber

November 15th, 2012 at 7:47 am

Hi,
Thanks for commands, after downloading cURL zip folder, in windows 7, where do I need to cd to in order to run/execute the commands?
Thanks

6

cURL Tutorial | Abhishek Singh Bailoo

January 17th, 2013 at 2:51 pm

7

HasBeenDrop

January 24th, 2013 at 5:33 am

wow. thanks sir. interesting app with console.
getting familiar with linux system.

8

Akash

January 27th, 2013 at 10:51 pm

Very nicely explained. very Impressive. thanks very much!! :)

9

HJAdjh

January 30th, 2013 at 4:09 am

Exactly waht I was looking for… thanks!!! cheers.

10

Sam

February 1st, 2013 at 8:34 pm

Thanks. I didn’t know you could set the destination filename when uploading via FTP.

11

dee

May 28th, 2014 at 1:36 am

nice

12

viagra

June 3rd, 2014 at 4:12 am

Hiya! Quick question that’s entirely off topic.

Do you know how to make your site mobile
friendly? My weblog looks weird when browsing from
my apple iphone. I’m trying to find a template or plugin that might be able to resolve this issue.
If you have any recommendations, please share. Thank you!

My web site … viagra

13

Lakshmi

June 17th, 2014 at 9:32 am

Is there a command in CURL to delete a file from FTP in windows ?

Your thoughts