Reverse cache proxy using squid proxy

If you came here looking for a reverse cache proxy, then you probably already heard
of squid proxy. As it name is, this is entirely designed as a proxy, however this does
not mean it cant be used as a reverse proxy or frontend proxy. Reverse proxy is a
type of proxy that is installed in front of a client for one server or multiple servers.
All resources that are being downloaded from the backend servers are being seen
as being downloaded from the reverse cache proxy. his is mostly used to reduce
the load on the backend servers by serving cached content directly from the reverse
proxy cache server.
How do you setup a reverse cache proxy?
In most reverse cache proxy applications, be it nginx, varnish, squid or any other
software, the basic setup is not exactly that hard to understand. !owever optimi"ing
the reverse proxy cache for your needs is a di#erent matter. A basic squid reverse
proxy installation would look similar to$
Apache
%
&
'
(
)
*
http+port ,- accel defaultsite.main/domain/name.com no/vhost
cache+peer %&0.-.-.% parent ,- - no/query originserver
name.1ourAccel2ame!ere
acl your+site+acl dstdomain main/domain/name.com
http+access allow your+site+acl
cache+peer+access 1ourAccel2ame!ere allow your+site+acl
cache+peer+access 1ourAccel2ame!ere deny all
3reak down of the above now. 4irs line we are saying to listen on port 80, in squid
acceleration mode and our main site is main/domain.com. 5host is not allowed, we
only server one domain in this setup.
2ext is the cache+peer, this is really important, we are saying the backend server is
on server ip %&0.-.-.% on port ,-, I67 port is set to - as we dont use multiple
backend server in this setup and we disable I67 requests by using no-query.
originserver is used to make the reverse proxy act as the real8origin server, this is
used mostly as an accelerator when it needs to act as a web server. he last one is
the name of your cache peer, this is useful when you use multiple cache+peer
option and you want to allow speci9c requests to either cache peer.
hen we have the A6: requsts, squid uses access control lists ;A6:s< for allowing
access to speci9c sections, in this case, we creat the access list speci9c to requests
that have their destination set as our main domain main/domain/name.com. 3y
default http+access denies all requests to pass to our squid reverse cache proxy
that have not been speci9cally allowed, so we use http_access allow
your_site_acl to allow access to our A6: that we created. he same for
cache_peer_access, we speci9cally allow access to our A6:.
hen everything is done, you =ust need to make sure your con9guration is set
correctly and you can restart squid.
Apache
%
&
squid /k parse
8etc8init.d8squid
restart
Reverse cache proxy virtual host based?
2ow what if we have a virtual host con9guration and we want to use the reverse
proxy cache in this situation. >ell the basic setup can be used almost the same, but
with a few ad=ustment. Also if you know the website domains, you can use the
above settings, but change the 9rst line to$
Apache
%
http+port ,- accel
vhost
?lse, if you you wish to allow all access to your squid cache proxy use the bellow
setting$
Apache
%
&
'
http+port ,- accel vhost
cache+peer %&0.-.-.% parent ,- - no/query
originserver
http+access allow all
7lease do keep in mind, the last line is 2@ really added, you should already have
the line somewhere bellow in your squid con9guration 9le$
Apache
%
http+access
deny all
hat needs to be changed to allow all access. hen everything will work as
expected, virtual host settings have been set. 1ou then make sure setup is properly
set and restart your squid cache reverse proxy.
Apache
%
&
squid /k parse
8etc8init.d8squid
restart
1. hat is !quid?
Aquid is used by hundreds of
Internet 7roviders world/wide to
provide their users with the best
possible web access. Aquid
optimi"es the data Bow between client and server to improve performance and
caches frequently/used content to save bandwidth. Aquid can also route content
requests to servers in a wide variety of ways to build cache server hierarchies which
optimi"e network throughput.
Aquid is a proxy server and web cache daemon. It has a wide variety of uses, from
speeding up a web server by caching repeated requestsC to caching web, D2A and
other computer network lookups for a group of people sharing network resourcesC to
aiding security by 9ltering traEc. Although primarily used for !7 and 47, Aquid
includes limited support for several other protocols including :A, AA:, Internet
Fopher and !7A. Aquid was originally designed to run on Gnix/like systems.
Released under the F2G Feneral 7ublic :icense, Aquid is free software. As an
example, Aquid is used by the >ikimedia 4oundation on >ikipedia.
It is good to read the 4AH section of Aquid when some unexpected problem happens
;or even better, before it happensI<. 6lick here or here ;older version but useful< to
read the 4AH section.
". hat is the Reverse #roxy $httpd-accelerator% &ode?
Aquid working in the Reverse 7roxy ;httpd/accelerator< mode caches incoming
requests for outgoing data ;i.e., that which you publish to the world<. It takes load
away from your !7 server and internal network. Aquid pulls the !7 data from
the JrealJ !7 server ;which means that only the accelerator needs to know where
the real server is<. he outside world sees no di#erence ;apart from an increase in
speed<.
he Aquid redirector can make one accelerator act as a single front/end for multiple
servers. If you need to move parts of your 9lesystem from one server to another, or
if separately administered !7 servers should logically appear under a single GR:
hierarchy, the accelerator makes the right thing happen. Keasurement of the Aquid
cache and its !arvest counterpart suggest an order of magnitude performance
improvement over 6?R2 or other widely available caching software. his order of
magnitude performance improvement on hits suggests that the cache can serve as
an httpd accelerator, a cache con9gured to act as a siteLs primary httpd server ;on
port '%&,<, forwarding references that miss to the siteLs real httpd ;on port ,-<. he
cache serves references to cachable ob=ects, such as !K: pages and FI4s, and the
true httpd ;on port ,-< serves references to non/cachable ob=ects, such as queries
and cgi/bin programs. If a siteLs usage characteristics tend toward cachable ob=ects,
this con9guration can dramatically reduce the siteLs web workload.
'. (nstallation
>e will consider two di#erent arquitectures.
In the 9rst case we have a 7roxy Aerver, where we will install Aquid listening at port
'%&, ;default port<, with two 2I6s. >e assume the 9rst 2I6 is connected to the
router of the IA7 provider. 4or instance you may have a Router called J:iveboxJ if
your IA7 provider is called J@rangeJ. :et us assume that the 7roxy Aerver has an
static I7 as %M&.%*,.%.%- and the D2A name is cosmos.linuxmaniac.net ;you will
have to adapt from now on this tutorial to your own parameters<. In the 8etc8hosts
9le of the 7roxy Aerver de9ne %M&.%*,.%.%- as cosmos.linuxmaniac.net. he second
2I6 is directly connected to a second computer working as a >eb Aerver ;a server
as Apache listening at port ,- is assumed to be installed and working in this
computer<.
1ou have to change the con9guration of your Router. In the router you have to
forward the port ,- to the port '%&, of the computer %M&.%*,.%.%- with D2A
hostname cosmos.linuxmaniac.net. If you have a :ivebox Router$
he D2A name is assigned in ?quipment /N Display, and clicking on the 7roxy
server, in Device Aettings, you can write the correct JD2A hostnameJ.
he static I7 is assigned in :ivebox /N Advanced 6on9guration /N D!67, in the
section JAtatic I7 addressJ.
o forward the ports, go to :ivebox /N Advanced 6on9guration /N 2A87A, and
choose !7 as Application8Aervice, ?xternal 7ort$ ,-, Internal 7ort$'%&,, 7rotocol$
67, Device$ cosmos.linuxmaniac.net.
he >eb Aerver will be assumed to have an internal D2A hostname ;con9gured in
8etc8hosts in the >eb Aerver< Jhp hp.homeJ with static I7$ %M&.%*,.%%.%& ;adapt that
data to your own setup<.
In the internal network, it is convenient in the case of a small :A2, to assign the
D2A hostname Jcosmos.linuxmaniac.netJ to the I7 %M&.%*,.%.%- to all computers
having access to the 7roxy Aerver, by means of the 8etc8hosts 9le, as explained in
section %).
In the case where you have a single computer running Aquid and Apache things are
not essentially di#erent. In this case you have a Jvirtual second computerJ working
with the I7 %&0.-.-.% ;which actually is the internal I7 address of every computer<.
1ou can check that by typing this I7 in your favorite browser and Apache will serve
your web. his means that %M&.%*,.%.%- will be the I7 of the 7roxy Aerver and
%&0.-.-.% will be the I7 of your web server. Oeep this in mind from now on.
@f course we install Aquid' in the proxy server using DebianLs repositories ;version
'.&< in Aynaptic.
). *on+guration
).1 !etting !quid in reverse proxy &ode
he con9guration 9le is 8etc8squid'8squid.conf. @pen it with your favorite editor so
we can make the necessary changes. he 9rst thing we do is to tell Aquid to operate
in reverse proxy mode, and setup a default host name which will be useful when
connecting to the reverse proxy by I7 address or an alias.
Around line %-0,, under the section 2?>@RO @7I@2A, add the following$
P Aquid normally listens to port '%&,
http+port '%&, accel defaultsite.cosmos.linuxmaniac.net vhost
where cosmos.linuxmaniac.net is the D2A hostname of the proxy server where we
are installing Aquid and '%&, is the port Aquid is listening by default.
)." ,e+ning the eb !erver
2ext we tell Aquid where were to 9nd and how to connect to the >eb Aerver. In
Aquid, this is referred to as a Jpeer cacheJ. @nly our >eb Aerver contains all the
information we need, therefore Aquid only needs to connect to it as a parent JpeerJ
to get that JcachedJ data.
Around line %,&* under the section @7I@2A >!I6! A44?6 !? 2?IF!3@R
A?:?6I@2 A:F@RI!K go ahead and add the following$
P 6hoosing >eb Aerver %M&.%*,.%%.%& listening at port ,- and name hp
cache+peer %M&.%*,.%%.%& parent ,- - no/query originserver name.hp
>e assume the server has I7 %M&.%*,.%%.%& and ,- is the port where the >eb
Aerver is listeningC the name is arbitrary and it is used for con9guration purposes
only.
If you use one single computer, write this instead$
cache+peer %&0.-.-.% parent ,- - no/query originserver name.hp
).' #er&ission-s .ssign&ent
2ow we create some A6:Ls ;Access 6ontrol :ist< giving users permission to access
our webserver through a speci9c host name.
Around line *(*, under the A6: section, add in$
acl hp+users dstdomain cosmos.linuxmaniac.net
Replace Jcosmos.linuxmaniac.netJ with the D2A hostname you will use to access
your Reverse 7roxy Aerver over the Internet. he A6: name Jhp+usersJ can be what
ever you want. >e will use this name to apply the A6: to permissions that we will
create.
If your site has several D2A hostnames, you can add them on the same line space
separated$
acl hp+users dstdomain cosmos.linuxmaniac.net cosmos.linuxlover.net
universe.linuxlover.net
2ow lets setup general !7 access permissions for all users on the proxy. Around
line 0,(, under the A6: section, add in$
PP I2A?R 1@GR @>2 RG:?;A< !?R? @ A::@> A66?AA 4R@K 1@GR 6:I?2A P
http+access allow hp+users
Kake sure you replace Jhp+usersJ with whatever A6: name you used.
4inally, allow the 7roxy users access to the >eb Aerver and deny everything else.
Around line %,*M or under the Jcache+peer+accessJ section, add in$
cache+peer+access hp allow hp+users
cache+peer+access hp deny all
@nce again replace JhpJ with the peer cache name you used and replace Jhp+usersJ
with the A6: name you used. Aave the con9guration 9le and restart Aquid using the
command line$
sudo 8etc8init.d8squid' restart
2ow, cross your 9ngers.
/. 0e&ory *ache 1ptions
Fo to the section K?K@R1 6A6!? @7I@2A, about line %M)-. he parameter
cache+mem does 2@ specify the maximum si"e of the process. It only speci9es
how much memory to use for caching JhotJ ;very popular< replies. AquidLs actual
memory usage is depends very strongly on your incoming request load. Aquid uses
memory for other things as well. he default si"e of cache+mem is &)* K3. 7robably
you may want to increase this amount depending on your system. In my case$
cache+mem )-- K3
he parameter maximum+ob=ect+si"e+in+memory is quite self/explanatory. @b=ects
greater than this si"e will not be attempted to kept in the memory cache. his
should be set high enough to keep ob=ects accessed frequently in memory to
improve performance whilst low enough to keep larger ob=ects from hoarding
cache+mem. he default si"e is )%&O3. 1ou may change this parameter depending
on your system. In my case$
maximum+ob=ect+si"e+in+memory )---- O3
It is good to monitor the memory used by Aquid and to ad=ust the parameters
depending on the system behaviour.
2. ,eny access to certain +les
:et us assume that our web server contains .php 9les. 3ut we wish to access to
them only from the internal network directly ;you may forward port ,- in the 7roxy
server to port ,- of the web server in the case you are using two computers<. his
means we wish to deny external access to those 9les through Aquid. hen you need
to de9ne an A6: name as we did before around line *(* under the A6: section$
P A6: name to deny access to .php 9les acl php+ext urlpath+regex /i Q.phpR
where php+ext is an arbitrary name for the access list. In the section http+access,
about line 0*M we add$
P Deny access
http+access deny php+ext
to deny the access to those 9les. It is possible to con9gure more so9sticated or
speci9c A6:s. 6heck the comments on the A6: section in the con9guration 9le to
9nd out.
3. 4ogging in .pache $(n the web server5 o6 course%
If Aquid is succesfully con9gured and running, you may check ApacheLs :ogs and
you will see that the I7s of your clients are lost. All requests seems to come from the
7roxy Aerver. o correct this, at the end of the 9le 8etc8Apache&.conf, where logging
is con9gured, follow the instructions about changing Sh for STU/4orwarded/forVi
and save the 9le. his allows correct logging of clients in the access log, but not in
the error log.
o get the right error logs, install the libapache&/mod/rpaf package. hen load the
module rpaf$
sudo a&enmod rpaf
?dit rpaf.conf and change the proxy server I7 ;%&0.-.-.% by default< ;only if the
7roxy Aerver and the >eb Aerver are in di#erent computers, otherwise do not
change the I7<. Restart Apache$
8etc8init.d8apache& restart
and check the changes in the logs.
8. Hu&an readable logs 6or !quid
Default log format for Aquid is de9ned at the end of the section :@F4I:? @7I@2A,
about line &%,($
P his de9nes the default log format JsquidJ logformat squid Sts.S-'tu S*tr SNa
SAs8S-'N!s SWst Srm Sru Sun SAh8SWA Smt
he meaning of every part is explained in the comments of the con9guration 9le.
6heck also in here for a more elaborated and clear explanation. I like having the
access time in a human readable form and a bit more information about the Referer
and Gser/Agent, so I use the following customi"ed logformat$
P his de9nes the log format JsquidJ
logformat squid STSd S!$SK$SAVtl S*tr SNa SAs8S-'N!s SWst Srm Sru JS
TRefererVNhJ JSTGser/AgentVNhJ Smt
In the following section of the con9guration 9le, the location of the log 9le is de9ned
by means of the variable access+log$
P Default$
access+log 8var8log8squid'8access.log squid
If you wish to change the log 9le, this is the place to do it.
7. !ending !quid logs to a re&ote co&puter
>e may want to receive logs of the 7roxy server into another computer of our
internal network to monitor the activity. his requires two main steps, 9rst we must
tell to the 7roxy Aerver that it must send the logs, then we must tell to the remote
computer to listen to the 7roxy Aerver and write the logs in a certain 9le. Are you
readyX
7.1 *on+guration o6 the #roxy !erver
4irst of all, we need Aquid to send the logs to rsyslogd. rsyslogd is a basic program
running in your system taking care of the system logsC the r in rsyslogd means
remote. rsyslogd is the program that will send the logs to the remote computer. >e
open the 9le 8etc8squid'8squid.conf and we go the section where access+log is
de9ned. 3y default you should have$
PDefault$
access+log 8var8log8squid'8access.log squid
>hich means that Aquid is sending the logs directly to the 9le
8var8log8squid'8access.log with the log format squid, which is de9ned in the previous
section of the con9guration 9le. 2ow, we add the following line in order to get Aquid
to send the logs to rsyslogd with the log format squid$
access+log syslog$local&.info squid
1ou need to restart Aquid so the changes take e#ect$
sudo 8etc8init.d8squid' restart
2ext, we edit the con9guration 9le of rsyslogd$ 8etc8rsyslog.conf. In the 9rst section
K@DG:?A the 9rst two lines must be active$
PPPPPPPPPPPPPPPPP
PPPP K@DG:?A PPPP
PPPPPPPPPPPPPPPPP
RKod:oad imuxsock P provides support for local system logging
RKod:oad imklog P provides kernel logging support ;previously done by rklogd<
In the section RG:?A, add the following lines, before any other rules$
PPPPPPPPPPPPPPP
PPPP RG:?A PPPP
PPPPPPPPPPPPPPP
P
P :ogging for my beloved Aquid'
P
local&.Y 8var8log8squid'.log
local&.Y ZZ%M&.%*,.%.%&$)%(
local&.Y ZZ%M&.%*,.%.&&$)%(
[ \
As you can see we are adding four lines. he 9rst line is telling rsyslogd to write the
logs into a 9le 8var8log8squid'.log. >e do this to check that rsyslogd is receiving the
logs from Aquid. :ater on we can delete this line. It is good to create the 9le
8var8log8squid'.log in advance using touch$
sudo touch 8var8log8squid'.log
because rsyslogd is not able to create 9les. he second and third lines means that
rsyslogd sends the logs to the remote computers with I7 %M&.%*,.%.%& and I7
%M&.%*,.%.&& using the standard port )%(. ZZ means we use 67 protocol to send
the logs ;we could use GD7 instead with one single Z<. @f course you change the
I7s depending on the I7s of your remote computers. he last line makes sure that
squid logs are not written in other system log 9les. 4inally we need to restart
rsyslogd$
sudo service rsyslogd restart
he 7roxy Aerver is done. 2ow we must con9gure the remote computer.
7." *on+guration o6 the Re&ote *o&puter
>e edit the con9guration 9le of rsyslogd$ 8etc8rsyslog.conf. In the 9rst section
K@DG:?A must look like the following one$
PPPPPPPPPPPPPPPPP
PPPP K@DG:?A PPPP
PPPPPPPPPPPPPPPPP
RKod:oad imuxsock P provides support for local system logging
RKod:oad imklog P provides kernel logging support
RKod:oad immark P provides //KARO// message capability
P provides GD7 syslog reception
PRKod:oad imudp
PRGD7AerverRun )%(
P provides 67 syslog reception
RKod:oad imtcp
RInput67AerverRun )%(
2otice that we expect the input logs in port )%( using 67 protocol. In the section
RG:?A, add the following lines, before any other rules$
PPPPPPPPPPPPPPP
PPPP RG:?A PPPP
PPPPPPPPPPPPPPP
if Rapp/name .. L;squid<L then 8var8log8squid'.log
[ \
Pif Rfromhost/ip startswith L%M&.%*,.%.L then 8var8log8squid'.log
P[ \
he 9rst line is quite self/explanatory, it sends Aquid logs to the 9le
8var8log8squid'.log ;which again must be created using touch<. he second line
makes sure that Aquid logs do not appear in other system logs. he commented
lines provide an alternative way to get Aquid logs into a 9le. 1ou can play with it if
you want to.
4inally we need to restart rsyslogd$
sudo service rsyslogd restart
2ow check the logs and make sure the remote computer receives Aquid activity
reports.