Curl Linux Command Line Short Notes

Conquering the Command Line Unix and
Linux Commands for Developers Mark Bates

cURL
cURL is easily one of the most powerful tools in a developers toolkit, as well as one of the most
complex ones. It ships with a dizzying array of options and features, most of which you will not
need on a daily basis.
cURL is a tool for working with URLs. cURL lets us query a URL from the command line. It lets us
post form data, FTP to a server, and much, much more. One of the places that cURL comes in
handy most often is working with APIs. With cURL we can try out new APIs simply, with just the
command line, with no need for installing, or writing complex wrappers around the API.
In this chapter, we will look at how we can use cURL to work with remote URLs. Well also learn
about the flags that will give us the most bang for our buck in an average day.
1 Installation
Theres a very good chance that you already have cURL installed on your machine, but it is nice to
be up-to-date with the latest version.
If you are on a Mac, the best and easiest way to install cURL is using Homebrew.
$ brew install curl
If you are on a different platform, please see the install directions on the projects page for the
correct installation instructions for your platform.
2 The Basics
The simplest thing we can do with cURL is to make an HTTP request to a given server1 and print
its response out to the console.
$ curl quiet-waters-1228.herokuapp.com/hello
As you can see in Listing 1, cURL prints the response of the web page to the console.
Listing 1
Hello, World!
Thank you for cURLing me!
If we want to see more information about the response, we can make the same request using the -i
flag.
$ curl -i quiet-waters-1228.herokuapp.com/hello
Listing 2
HTTP/1.1 200 OK
Cache-Control: max-age=0, private, must-revalidate
Content-Type: text/html; charset=utf-8
Etag: "a0bb15ce430e40738d857e3e7dfe0de7"
Server: thin 1.6.1 codename Death Proof
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Request-Id: d605f89d-bffc-4983-8ca4-8ac2b77c7b8d
X-Runtime: 0.002568
X-Ua-Compatible: chrome=1
X-Xss-Protection: 1; mode=block
transfer-encoding: chunked
Connection: keep-alive
Hello, World!
Thank you for cURLing me!
Listing 2 demonstrates that the -i flag will return response information such as headers, server
type, content type, and more. This level of detail can prove to be extremely useful for debugging a
request.
3 Following Redirects (-L)

There are times when we are trying to download a resource from a server, but instead of the
resource we requested, the server returns a status code of 302. The 302 response may also include
a message that the resource has moved. See Listing 3 for just such an example.
$ curl -i quiet-waters-1228.herokuapp.com/redirectme
Listing 3
HTTP/1.1 302 Moved Temporarily
Cache-Control: no-cache
Content-Type: text/html; charset=utf-8
Location: http://quiet-waters-1228.herokuapp.com/hello
X-Request-Id: e9785070-173c-4e8a-bbf5-1686806cbd6b
X-Runtime: 0.007276
<html>
<body>
You are being
<a href="http://quiet-waters-1228.herokuapp.com/hello">redirected</a>.
</body>
</html>
How do we follow redirects without having to first parse the response and build a second request to
the correct resource? To do this, we can use the -L flag built into cURL.
$ curl -L quiet-waters-1228.herokuapp.com/redirectme
Using the -L flag we should now get the same response we did in Listing 1.
4 Downloading Files (-O)

Printing responses to a console can be useful, but one of cURLs most popular features is its ability
to easily download files.
If we were to run the following command against a file, the result wouldnt be very pleasing to your
eyes on the screen.
$ curl http://quiet-waters-1228.herokuapp.com/assets/image.jpg
Listing 4
??JFIF,,??tExifII*
????(1?2?i???
15:29:28??????"?'
NIKONE950,,Adobe Photoshop 7.02002:04:25
Listing 4 shows the first two lines from a lot of gibberish that represents a binary image file.
However if we add the -O flag to the request, the results will be saved into a file named
image.jpg.
$ curl -O http://quiet-waters-1228.herokuapp.com/assets/image.jpg
Listing 5 and Listing 6 show the file downloading, and then a confirmation of the file saved to disk.
Listing 5
% Total
100
413k
% Received % Xferd
100
413k
Average Speed
Time
Dload Upload
Total
306k
0 0:00:01
Time
Time Current
Spent
Left Speed
0:00:01 --:--:-- 306k
Listing 6: $ ls -la | grep image.jpg

-rw-r--r--@
1 markbates
staff
423159 Nov 18 10:46 image.jpg
As Listing 6 shows us, the file is downloaded from the server to the current working directory using
the same name as the file on the server. Section 4.1 will show us how we can change the name of
this file.
See Chapter 5 for more information on how to use grep.
4.1 Custom File Names (-o)

More often than not, we can use the name of the file from the server as the name of the file that we
download to disk. However, in this example case the name image.jpg leaves a lot to be desired.
We dont know what it is an image of.
We can solve this problem in several different ways. The first way is to use the mv command to
rename the file after weve downloaded it. Another way is to use the -o flag to give cURL a file
name to save the file to.

$ curl -o my_image.jpg http://quiet-waters-1228.herokuapp.com/assets/image.jpg
Listing 7: $ ls -la | grep my_image.jpg

-rw-r--r--@
1 markbates
staff
423159 Nov 18 10:58 my_image.jpg
Listing 7 shows that the file has now been downloaded using the custom file name that we specified
using the -o flag.
5 Changing the HTTP Request Method (-X)

So far we have just been using cURL to make GET requests; however, cURL supports all of the
HTTP request methods, including POST, PATCH, PUT, DELETE, etc.
To change the request method, we can use the -X flag.
Lets make a simple POST request to an echo server that will print out any parameters we send to
it.
$ curl -X POST quiet-waters-1228.herokuapp.com/echo
Listing 8 shows the result of our request.

Listing 8
<!DOCTYPE html>
<html>
<head>
<title>Unix and Linux Commands for Developers</title>
<link href="/assets/application.css" media="all" rel="stylesheet" />
<script src="/assets/application.js"></script>
</head>
<body>

url:

http://quiet-waters-1228.herokuapp.com/echo



method:

POST


</body>
</html>
Lets make the same request again, only this time using the PUT request method.
$ curl -X PUT quiet-waters-1228.herokuapp.com/echo
Listing 9 confirms that we did indeed change the request method from POST to PUT.
Listing 9
<!DOCTYPE html>
<html>
<head>
</head>
<body>






PUT


</body>
</html>
Depending on your request type (in our case this was HTTP), the -X flag changes the request
method appropriately. For example, with an FTP request the -X flag will allow you to change the
request type from LIST to another request method.
5.1 Sending Parameters

So far, we havent sent any parameters to the server. There are a few different ways to do this.
The first, and most obvious method, is to append query string parameters to the URL2 itself.
$ curl -X POST "quiet-waters-1228.herokuapp.com/echo?fname=Mark&lname=Bates"
Listing 10 shows that the query string parameters are parsed correctly by the echo server.
Listing 10
<!DOCTYPE html>
<html>
<head>
</head>
<body>


http://quiet-waters-1228.herokuapp.com/echo?fname=Mark&lname=Bates



parameters:

{"fname"=>"Mark", "lname"=>"Bates"}




POST


</body>
</html>
Passing query string parameters will work with any request type, not just POST. This is the
simplest, and easiest, way to pass data to the server in a cURL request, but not the only way.
5.2 Passing Data in a Request Body (-d)

When making non-GET requests, such as a POST, it is quite common to want to pass data to the
server via the body of the request. After all, query string parameters cant represent a complex form
very well.
cURL provides the -d flag to us pass data as part of the request body.
Instead of passing the data via query string parameters, lets use the -d flag instead.
$ curl -X POST -d "fname=Mark&lname=Bates" quiet-waters-1228.herokuapp.com/echo
As we can see in Listing 11, the echo server reads the body and converts it into parameters, just as if
we had used query string parameters.
Listing 11
<!DOCTYPE html>
<html>
<head>
</head>
<body>





parameters:

{"fname"=>"Mark", "lname"=>"Bates"}




POST



body:

fname=Mark&lname=Bates


</body>
</html>
We can also see in Listing 11 that the echo server now displays a body attribute to us. This proves
that the data is in fact coming as a request body, and not just basic parameters.
We can make this more interesting by passing JSON via the -d flag.
$ curl -X POST -d "{\"name\":\"Mark\"}" quiet-waters-1228.herokuapp.com/echo
Listing 12
<!DOCTYPE html>
<html>
<head>
</head>
<body>






POST




{"name"=>"Mark"}


</body>
</html>
Listing 12 shows the JSON has been received and parsed correctly.
5.3 Using a File for a Request Body (-d)

Being able to send the request body via the command line is very useful, but most of the time well
want to read in a file and send that as the body of the request. An example of this would be sending
larger amounts of JSON.
Lets say we have the file from Listing 13, how would we post that to a server?
Listing 13: form_data.json

{
"lname": "Bates",
"fname": "Mark",
"site": "http://www.markbates.com",
"twitter": "http://twitter.com/markbates"
}
The -d flag already gives us all that we need to get the job done. By passing the -d flag a file name
prefixed with @, it will read in that file as the request body.
$ curl -X POST -d @form_data.json quiet-waters-1228.herokuapp.com/echo
Listing 14
<!DOCTYPE html>
<html>
<head>
</head>
<body>






POST




{"lname"=>"Bates", "fname"=>"Mark",
"site"=>"http://www.markbates.com", "twitter"=>"http://twitter.com/markbates"}


</body>
</html>
With this simple way of using files for request bodies, it can be very easy to write a script using
cURL to upload files to a remote server.
5.4 Form Parameters (-F)

Another way to pass parameters using cURL is the -F flag. Using this flag we can send parameters
that will be interpreted by the remote server as if they had been posted from an HTML form.
$ curl -X POST -F user[fname]=Mark -F user[lname]=Bates -F foo=bar \
quiet-waters-1228.herokuapp.com/echo -H "Accept: application/json"
Listing 15
"url": "http://quiet-waters-1228.herokuapp.com/echo",
"parameters": {"user":{"fname":"Mark","lname":"Bates"},"foo":"bar"},
"method": "POST",
"body": "------------------------------d6e7f7029fa5\r\nContent-Disposition:
form-data; name=\"user[fname]\"\r\n\r\nMark\r\n-----------------------------d6e7f7029fa5\r\nContent-Disposition: form-data; name=\"user[lname]\"\r\n\r\n
Bates\r\n------------------------------d6e7f7029fa5\r\nContent-Disposition:
form-data; name=\"foo\"\r\n\r\nbar\r\n-----------------------------d6e7f7029fa5--\r\n"
}
In Listing 15 we can see that the parameters were interpreted by the server as form-data. This
can be used to mimic HTML forms, and can be particularly useful to post both form parameters as
well as one, or more, files.
File Uploads with Form Parameters
Uploading a file using the -F flag is almost identical to using the -d flag that we saw in Section 5.
The only difference is that the file must be given a parameter name. This can be see in Listing 16.
Listing 16
$ curl -X POST -F user[fname]=Mark -F user[lname]=Bates -F foo=bar \
-F user[photo]=@path/to/image.jpg quiet-waters-1228.herokuapp.com/echo
-H "Accept: application/json"
6 Setting Headers (-H)

So far, all of the responses we have received from the server have been in HTML, but the server is
also capable of sending us back data formatted in either JSON or the dreaded XML.
In order to tell the server which type of content we want returned, we must set the ContentType header using the -H flag.
$ curl -X POST -d @form_data.json quiet-waters-1228.herokuapp.com/echo \
-H "Accept: application/json"
Listing 17 shows that by using the -H flag and giving it

"Accept: application/json"
the server responds appropriately with JSON instead of HTML.

Listing 17
{
"body" : {
"site" : "http://www.markbates.com",
"twitter" : "http://twitter.com/markbates",
"lname" : "Bates",
"fname" : "Mark"
},
"url" : "http://quiet-waters-1228.herokuapp.com/echo",
"method" : "POST"
Setting the content type of a request isnt the only header we can specify using the -H flag. We can
set any headers that the responding server will accept. cURL allows multiple -H flags on a request,
making it easy to send several headers at once.
$ curl -X POST -d @form_data.json quiet-waters-1228.herokuapp.com/echo \
-H "Accept: application/json" -H "X-Auth: 1234567890"
As we see in Listing 18 the new header, "X-Auth", has been received by the server.
Listing 18
{
"body" : {
"site" : "http://www.markbates.com",
"twitter" : "http://twitter.com/markbates",
"lname" : "Bates",
"fname" : "Mark"
},
"url" : "http://quiet-waters-1228.herokuapp.com/echo",
"x-auth" : "1234567890",
"method" : "POST"
7 Basic HTTP Auth (-u)

A lot of the internet exists behind a login form these days, so its important to understand how we
can use cURL to access resources that may be protected by such an authentication requirement.
Lets see what happens when we try to access a resource that requires authentication.
$ curl -i -X POST quiet-waters-1228.herokuapp.com/login
Listing 19 shows that we received a status code of 401 and a message that it cant find a user.
Listing 19
HTTP/1.1 401 Unauthorized
Cache-Control: no-cache
Content-Type: application/json; charset=utf-8
X-Request-Id: da6bbeae-1b77-4f2a-bf75-300b981e108d
X-Runtime: 0.009238
{"error":"Unknown User"}
The -u flag will let us pass a username and a password to the server to solve this problem.
$ curl -X POST -u "user1:password1" quiet-waters-1228.herokuapp.com/login
As seen in Listing 20 the login was successful and we now use the information for user1.
Listing 20
"password" : "password1",
"name" : "User 1",
"id" : 1,
"username" : "user1"
With the -u flag we can now successfully authenticate ourselves against any page that uses HTTP
auth.
8 Saving and Sending Cookies/Headers (-D, -b)

In Section 7 we learned how to use the -u flag to authenticate against basic HTTP authentication.
But how do we handle a page that is behind a login, yet doesnt respond to HTTP auth?
Lets attempt to request the endpoint in question and see what the result is.
$ curl -i quiet-waters-1228.herokuapp.com/whoami
Not surprisingly we get a 401, just like in Listing 19, so lets try and use the -u flag and see if we
can gain access to the page that way.
$ curl -i -u "user1:password1" quiet-waters-1228.herokuapp.com/whoami
Again, we are greeted with a 401. The server is trying to use a session cookie to authenticate us,
and we dont have one. To solve this problem, we first need to hit the login endpoint using the -u
flag, save the session cookie that is returned, and then pass that session cookie back again on
subsequent requests.
The -D flag will tell cURL to dump headers and cookies into a specified file in the current
directory.
$ curl -X POST -D headers -u "user1:password1" \
quiet-waters-1228.herokuapp.com/login
The more verbose dump-header flag is a mnemonic synonym for -D.

$ curl -X POST --dump-header headers -u "user1:password1" \
Listing 21 shows the contents of the file that got created as a result of our request.
Listing 21: headers
HTTP/1.1 200 OK
Cache-Control: max-age=0, private, must-revalidate
Content-Type: application/json; charset=utf-8
Etag: "768e41b20e1e385d06f9b2da6f7e0f08"
Set-Cookie: _curl_test_app_rails_session=TWpwUitjellkSE1QOHpyTEN0Q1JkazFhVVhvMi9
4RnZvZ2JEdExjR2hrb05FbGpVUCtHZ2xraTF5a01qWFc4OWhyUER6cG53dlZQZUhHdnJxTHBKSW1ITHQ
0NTdUOXdFVU5nNE05VStadUo2dS80eEhkVzJzNUdXZVg5SC9OLy8tLVFRKzc3cHB2UWlKM3FOckM2S3h
jZHc9PQ%3D%3D--3ab5702185dca000cff025fd3139348147171c34; path=/; HttpOnly
X-Request-Id: a5efa267-3f9a-40ff-9a8d-49bfa8754c62
X-Runtime: 0.050191
With this headers file now saved, we can pass it around to subsequent requests and the server
will have access to the session cookie that it needs, along with all of the original headers that were
part of the response.
In order to pass the headers file back to the server with our request, we need to use the -b flag.
$ curl -b headers quiet-waters-1228.herokuapp.com/whoami
By passing the -b flag and giving it the name of the file that we had previously saved, Listing 22
shows that we have successfully accessed the end point that was previously unavailable to us.
Listing 22
{
"name" : "User 1",
"id" : 1,
8.1 Using the Cookie Jar (-c)

When using the dump-header flag we not only capture the cookies from a response, but also all
of the headers from that response. This may or may not be what you want.
curl has a built-in cookie jar meant for storing just the cookies and sending them back to the
server.
To store cookies in the cookie jar we use the -c flag and give it the name of a file we wish to store
the cookies in.
$ curl -X POST -c cookies.txt -u "user1:password1" \
Listing 23
{"id":1,"name":"User 1","password":"password1","username":"user1"}
As we can see in Listing 23 we get a successful response, just as we did when we used the dumpheader flag.
If we use the cat command (Chapter 9 Section 9.2) we can look at the contents of the
cookies.txt file.
$ cat cookies.txt
Listing 24
# Netscape HTTP Cookie File
# http://curl.haxx.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.
#HttpOnly_quiet-waters-1228.herokuapp.com FALSE / FALSE 0 _curl_test_app_rails_

session cm53d2RJN1VncVpnK3psVUpuRHB2L2d3emIyTENIMGhBVjlPOHozM0lJYmk2WmwrSGZEbEV
UbXpnSkdGaktvZjVFeThranhuSi9HZm9YTHFSYUthUzhoVUc0T3J3MEltempsNC96UVFweXMranlsem
RPMzNYSmxzK2pJME5DMEItLWVkSDZZN0pjdU5HTmt5WHRxTW1vSUE9PQ%3D%3D--06137b60e0ff66f
3a6cad05e4fe83a0094a989be
When we look at the cookies.txt file in Listing 24 we see that this time we are only storing the
cookies from the response, and not the headers as well.
Using the -b flag, as we did earlier, we can send the contents of the cookies.txt file to the
server with the request.
$ curl -b cookies.txt quiet-waters-1228.herokuapp.com/whoami
Listing 25
{
"name" : "User 1",
"id" : 1,
Listing 25 shows that we get a successful response from the server when passing cookies.txt
to it.
9 Conclusion
As you can see, cURL is an incredibly powerful tool. We have used it to download files, upload
files, access authenticated pages, and more. This is a very small amount of the features that are
available in cURL.
The MAN page for cURL is very long, and for a very good reason. The list of options that can be
passed is dizzying. Hopefully this chapter has given you the building blocks for using cURL on a
daily basis.
1. I have set up a small demo application at quiet-waters-1228.herokuapp.com to help better
illustrate the examples in this chapter.
2. Because we are using an & in the URL we need to make sure to escape the URL so that the
command doesnt interpret it incorrectly.

Curl Linux Command Line Short Notes

Transféré par

Informations du document

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Curl Linux Command Line Short Notes

Transféré par

Droits d'auteur :

Formats disponibles

Conquering the Command Line Unix and

Linux Commands for Developers Mark Bates

3 Following Redirects (-L)

4 Downloading Files (-O)

NIKONE950,,Adobe Photoshop 7.02002:04:25

Listing 6: $ ls -la | grep image.jpg

423159 Nov 18 10:46 image.jpg

4.1 Custom File Names (-o)

name to save the file to.

Listing 7: $ ls -la | grep my_image.jpg

423159 Nov 18 10:58 my_image.jpg

5 Changing the HTTP Request Method (-X)

Listing 8 shows the result of our request.

5.1 Sending Parameters

5.2 Passing Data in a Request Body (-d)

5.3 Using a File for a Request Body (-d)

Listing 13: form_data.json

5.4 Form Parameters (-F)

6 Setting Headers (-H)

Listing 17 shows that by using the -H flag and giving it

the server responds appropriately with JSON instead of HTML.

7 Basic HTTP Auth (-u)

8 Saving and Sending Cookies/Headers (-D, -b)

The more verbose dump-header flag is a mnemonic synonym for -D.

8.1 Using the Cookie Jar (-c)

#HttpOnly_quiet-waters-1228.herokuapp.com FALSE / FALSE 0 _curl_test_app_rails_

Vous aimerez peut-être aussi