Thursday, November 11, 2010

How to Graph Your Network using cacti







At this point, you probably realize that graphing is Cacti's greatest strength. Cacti has many powerful features that provide complex graphing and data acquisition, some which have a slight learning curve. Do not let that stop you however, because graphing your network is incredibly simple.
The next two sections will outline the two basic steps which are typically required to create graphs for most devices.

Creating a Device

The first step to creating graphs for your network is adding a device for each network device that you want to create graphs for. A device specifies important details such as the network hostname, SNMP parameters, and host type.
To manage devices within Cacti, click on the Devices menu item. Clicking Add will bring up a new device form. The first two fields, Description and Hostname are the only two fields that require your input beyond the defaults. If your host type is defined under the host template dropdown, be sure to select it here. You can always choose "Generic SNMP-enabled Host" if you are just graphing traffic or "None" if you are unsure. It is important to remember that the host template you choose will not lock you into any particular configuration, it will just provide more intelligent defaults for that type of host.
Figure 7-1. Adding a New Device
The field definitions are as follows
Table 7-1. Device Field Definitions
FieldDescription
DescriptionThis description will show up in the first column of the device list. You may refer to it e.g. in graph titles
HostnameEither an IP address or a hostname. The hostname will be resolved using the standard host resolving mechanisms, e.g. Dynamic Name Services (DNS)
Host TemplateA Host Template is a container for a list of graph templates that will be related to this host.
NotesNew with cacti 0.8.7. Add notes to a host to add arbitrary text.
Disable HostExclude this host from being polled. This is of particular value, if a device is no longer available, but should be kept e.g. as a reference.
Availability/Reachability Options
Downed Device DetectionNONE: Deactivate downed host detection
PING and SNMP: perform both tests
SNMP: verify SNMP check on OID .1 and .1.3
ICMP: perform a ping test, see below
Ping MethodAvailable only for "PING and SNMP" or "PING"
ICMP: perform ICMP tests. Requires permissions
UDP: perform a UDP test
TCP: perform a TCP test
Ping PortAvailable only for UDP/TCP PING test types.
Please define the port to be tested here. Make sure, that no firewall intercepts the tests
Ping Timeout ValueAfter this time, the test fails. Measured in units of milliseconds
Ping Retry CountThe number of times Cacti will attempt to ping a host before failing.
SNMP Options
SNMP VersionVersion 1: Use SNMP Version 1. Be aware, that 64bit counters are not supported in this SNMP version
Version 2: Referred to as SNMP V2c in most SNMP documentations
Version 3: SNMP V3, supporting authentication and encryption
SNMP CommunitySNMP read community for this device.
SNMP PortEnter the UDP port number to use for SNMP (default is 161).
SNMP TimeoutThe maximum number of milliseconds Cacti will wait for an SNMP response (does not work with php-snmp support).
Maximum OID's Per Get RequestThis is a performance feature. Specifies the number of OID's that can be obtained in a single SNMP Get request.
NOTE: This feature only works when using Spine
NOTE: Some devices do not support values > 1
Security Options for SNMP V3
SNMP UsernameThe username of an SNMP V3 createUser statement or equivalent
SNMP PasswordThe authpassphrase of an SNMP V3 createUser statement or equivalent
SNMP Auth ProtocolThe authentication type of an SNMP V3 createUser statement or equivalent. Select either MD5 or SHA. This entry defaults to MD5.
SNMP Privacy PassphraseThe privacy passphrase of an SNMP V3 createUser statement or equivalent.
SNMP Privacy ProtocolThe privacy protocol of an SNMP V3 createUser statement or equivalent. Select either DES or AES. This entry defaults to DES.
SNMP ContextWhen using the View-Based Access Control Model (VACM), it is possible to specify an SNMP Context when mapping a community name to a security name with a com2secdirective, with the group directive and the access directive. This allows for defining special access models. If using such a parameter with your target's SNMP configuration, specify the context name to be used to access that target here.
After saving your new device, you should be redirected back to the same edit form with some additional information. If you configured SNMP for this host by providing a valid community string, you should see various statistics listed at the top of the page. If you see "SNMP error" instead, this indicates an SNMP problem between Cacti and your device.
Towards the bottom of the page there will be two addition boxes, Associated Data Queries, and Associated Graph Templates. If you selected a host template on the previous page, there will probably be a few items in each box. If there is nothing listed in either box, you will need to associate at least one data query or graph template with your new device or you will not be able to create graphs in the next step. If no available graph template or data query applies to your device, you can check the Cacti templates repository or create your own if nothing currently exists.


A Word About SNMP

The SNMP version that you choose can have a great effect on how SNMP works for you in Cacti. Version 1 should be used for everything unless you have reason to choose otherwise. If you plan on utilizing (and your device supports) high-speed (64-bit) counters, you must select version 2. Starting with Cacti 0.8.7, version 3 is fully implemented.
The way in which Cacti retrieves SNMP information from a host has an effect on which SNMP-related options are supported. Currently there are three types of SNMP retrieval methods in Cacti and are outlined below.
Table 7-2. SNMP Retrieval Types
TypeDescriptionSupported OptionsPlaces Used
External SNMPCalls the net-snmp snmpwalk and snmpget binaries that are installed on your system.All SNMP optionsWeb interface and PHP poller (poller.php)
Internal SNMP (php-snmp)Uses PHP's SNMP functions which are linked against net-snmp or ucd-snmp at compile time.Version 1 Only (Community and Port)Web interface and PHP poller (poller.php)
Spine SNMPLinks directly against net-snmp or ucd-snmp and calls the API directly.All SNMP optionsC-Based Poller (Spine)


SNMP V3 Options Explained

SNMP supports authentication and encryption features when using SNMP protocol version 3 known as View-Based Access Control Model (VACM). This requires, that the target device in question supports and is configured for SNMP V3 use. In general, configuration of V3 options is target type dependant. The following is cited from man snmpd.conf concerning user definitions

[   SNMPv3 Users
       createUser [-e ENGINEID] username (MD5|SHA) authpassphrase [DES|AES] [privpassphrase]

              MD5 and SHA are the authentication types to use.  DES and AES are the privacy
              protocols  to  use. If  the  privacy  passphrase  is not specified, it is assumed
              to be the same as the authentication passphrase.  Note that the users created will
              be useless unless they are also added  to  the  VACM access control tables
              described above.

              SHA  authentication  and DES/AES privacy require OpenSSL to be installed and the
              agent to be built with OpenSSL support.  MD5 authentication may be used without
              OpenSSL.

              Warning: the minimum pass phrase length is 8 characters.]

VACM directives are explained from man snmpd.conf as follows

[   VACM Configuration
       The full flexibility of the VACM is available using four configuration directives -
       com2sec, group,  view and access.  These provide direct configuration of the underlying
       VACM tables.

       com2sec  [-Cn CONTEXT] SECNAME SOURCE COMMUNITY
              map  an  SNMPv1 or SNMPv2c community string to a security name - either from a
              particular range of source addresses, or globally ("default").  A restricted
              source can either be a specific  hostname (or  address),  or  a  subnet -
              represented as IP/MASK (e.g. 10.10.10.0/255.255.255.0), or IP/BITS
              (e.g. 10.10.10.0/24), or the IPv6 equivalents.

              The same community string can be specified in several separate directives
              (presumably with different  source  tokens), and the first source/community
              combination that matches the incoming request will be selected.  Various
              source/community combinations can also map to the same security name.

              If a CONTEXT is specified (using -Cn), the community string will be mapped
              to a security  name  in the named SNMPv3 context. Otherwise the default
              context ("") will be used.

       group GROUP {v1|v2c|usm} SECNAME
              maps  a  security name (in the specified security model) into a named group.
              Several group directives can specify the same group name, allowing a single
              access setting to apply to several  users and/or community strings.

              Note  that  groups must be set up for the two community-based models separately -
              a single com2sec (or equivalent) directive will typically be accompanied by two
              group directives.

       view VNAME TYPE OID [MASK]
              defines a named "view" - a subset of the overall OID tree. This is most commonly
              a single subtree, but  several view directives can be given with the same view
              name, to build up a more complex collection of OIDs.  TYPE is either included
              or excluded, which can again define a more complex view (e.g by excluding certain
              sensitive objects from an otherwise accessible subtree).

              MASK is a list of hex octets (separated by "." or ":")
              with the set bits indicating which subidentifiers in the view OID to match against.
              This can be used to define a view covering a particular row  (or  rows) in a table.
              If not specified, this defaults to matching the OID exactly (all bits set), thus
              defining a simple OID subtree.

       access GROUP CONTEXT {any|v1|v2c|usm} LEVEL PREFX READ WRITE NOTIFY
              maps from a group of users/communities (with a particular security model
              and  minimum  security level, and in a specific context) to one of three views,
              depending on the request being processed.

              LEVEL is one of noauth, auth, or priv.  PREFX specifies how CONTEXT should be
              matched against  the context  of  the  incoming  request, either exact or prefix.
              READ, WRITE and NOTIFY specifies the view to be used for GET*, SET and
              TRAP/INFORM requests (althought the NOTIFY view is not currently used).  For
              v1 or v2c access, LEVEL will need to be noauth.]

As an example, following definitions in the snmpd configuration create a set of definitions for use with SNMP V3
# sample configuration for SNMP V3

# create an SNMP V3 user with an authpassphrase and a privacy passphrase
##         username   authProto  authpassphrase  privProto  privpassphrase
##         --------   ---------  --------------  ---------  --------------
createUser gandalf    MD5        myauthpass      DES        myprivpass


# Second, map the security name into a group name:
##    groupName    securityModel  securityName
##    ---------    -------------  ------------
group groupv3      usm            gandalf


# Third, create a view for us to let the group have rights to:
##          incl/excl  subtree   mask
##          ---------  -------   ----
view    all included   .iso      80


# Fourth, create the access for that group without context
##                context sec.model sec.level prefix read   write  notif
##                ------- --------- --------- ------ ----   -----  -----
access groupv3    ""      any       auth      exact  all    all    all
When adding this stuff to your snmp configuration, please remember to restart the agent. Verify this setting using
shell>snmpwalk -v 3 -a MD5 -A myauthpass -x DES \
-X myprivpass -u gandalf -l authpriv localhost interface
IF-MIB::ifNumber.0 = INTEGER: 3
IF-MIB::ifIndex.1 = INTEGER: 1
IF-MIB::ifIndex.2 = INTEGER: 2
IF-MIB::ifIndex.3 = INTEGER: 3
IF-MIB::ifDescr.1 = STRING: lo
IF-MIB::ifDescr.2 = STRING: irda0
IF-MIB::ifDescr.3 = STRING: eth0
IF-MIB::ifType.1 = INTEGER: softwareLoopback(24)
IF-MIB::ifType.2 = INTEGER: other(1)
IF-MIB::ifType.3 = INTEGER: ethernetCsmacd(6)
...

Creating the Graphs

Now that you have created some devices, it is time to create graphs for these devices. To do this, select the New Graphs menu option under the Create heading. If you're still at the device edit screen, selectCreate Graphs for this Host to see a screen similar to the image pictured below.
Figure 7-2. Creating New Graphs
The dropdown menu that contains each device should be used to select the host that you want to create new graphs for. The basic concept to this page is simple, place a check in each row that you want to create a graph for and click Create.
If you are creating graphs from inside a "Data Query" box, there are a few additional things to keep in mind. First is that you may encounter the situation as pictured above with the "SNMP - Interface Statistics" data query. If this occurs you may want to consult the section on debugging data queries to see why your data query is not returning any results. Also, you may see a "Select a graph type" dropdown box under some data query boxes. Changing the value of this dropdown box affects which type of graph Cacti will make after clicking the Create button. Cacti only displays this dropdown box when there is more than one type to choose from, so it may not be displayed in all cases.
Once you have selected the graphs that you want to create, simply click the Create button at the bottom of the page. You will be taken to a new page that allows you to specify additional information about the graphs you are about to create. You only see the fields here that are not part of each template, otherwise the value automatically comes from the template. When all of the values on this page look correct, click the Create button one last time to actually create your graphs.
If you would like to edit or delete your graphs after they have been created, use the Graph Management item on the menu. Likewise, the Data Source menu item allows you to manage your data sources in Cacti.
(from http://www.cacti.net)
















Upgrading Cacti

  1. Backup the old Cacti database.
    shell> mysqldump -l --add-drop-table cacti > mysql.cacti
    Note: You will probably have to specify the -u and -p flags for the MySQL username and password. This user must have permission to read from Cacti's database or you will end up with an empty backup.
  2. Backup the old Cacti directory.
    shell> mv cacti cacti_old
  3. Extract the distribution tarball.
    shell> tar xzvf cacti-version.tar.gz
  4. Rename the new Cacti directory to match the old one.
    shell> mv cacti-version cacti
  5. Edit include/config.php and specify the MySQL user, password and database for your Cacti configuration.
    $database_type = "mysql";
    $database_default = "cacti";
    $database_hostname = "localhost";
    $database_username = "cactiuser";
    $database_password = "cacti";
  6. Copy the *.rrd files from the old Cacti directory.
    shell> cp cacti_old/rra/* cacti/rra/
  7. Copy any relevant custom scripts from the old Cacti directory. Some script are updated between versions. Therefore, make sure you only over write if the scripts either don't exist or are newer than the distribution's.
    shell> cp -u cacti_old/scripts/* cacti/scripts/
  8. Copy any relevant custom resource XML files from the old Cacti directory. Some resource XML files are updated between versions. Therefore, make sure you only over write if the XML files either don't exist or are newer than the distribution's.
    shell> cp -u -R cacti_old/resource/* cacti/resource/
  9. Set the appropriate permissions on Cacti's directories for graph/log generation. You should execute these commands from inside Cacti's directory to change the permissions.
    shell> chown -R cactiuser rra/ log/
    (Enter a valid username for cactiuser, this user will also be used in the next step for data gathering.)
  10. Point your web browser to:
    http://your-server/cacti/
    Follow the on-screen instructions so your database can be updated to the new version.

Install and Configure Cacti monitoring tool in Ubuntu 9.10 (Karmic) Server

Cacti is a complete network graphing solution designed to harness the power of RRDTool’s data storage and graphing functionality. Cacti provides a fast poller, advanced graph templating, multiple data acquisition methods, and user management features out of the box. All of this is wrapped in an intuitive, easy to use interface that makes sense for LAN-sized installations up to complex networks with hundreds of devices.
Cacti Features
If you want to know more about cacti check here
Install cacti in Ubuntu 9.10 (Karmic) Server
Preparing Your system
First you need to make sure you have install Ubuntu 9.10 LAMP server and then you need to install the following packages
sudo apt-get install php5 php5-gd php5-mysql
Install accti using the following command
sudo apt-get install cacti-spine
This will start installing cacti and prompt for some questions.
Configuring libphp-adodb option select ok Enter to Continue.
Select your webserver in this example we are using apache2 select ok Enter to Continue
Configure database for cacti select yes Enter to Continue
Enter the root password for your mysql server select ok Enter to Continue
Enter the password (cacti) for the cacti database,select ok Enter to Continue
Re-enter your password
This will complete the cacti installation.
Configuring Cacti
Now you need to point your browser http://serverip/cacti press enter.You should see similar to the following screen Click Next to Continue.
You need to select the type of installation as New Installation and click next to Continue.
Now it will check all the required paths are correct or not you can see this in the following screen here click
on Finish
Now you should see the Cacti Login screen as follows
Here you need to enter username and password as admin/admin and click login
First time it will prompt you to change cacti admin user password for security reasons and click save
You should see the following screen like below once you logged.
In the above screen Under Configuration Click settings on your right hand side click on paths.Make sure in the “Spine Poller File Path”, you have
“/usr/sbin/spine”
Now you need to click on Poller Select “spine” in the “Poller Type”,click on Save.
Now you need to wait for 15 min to see your server graphs starts appearing in cacti graphs.
(from: http://www.ubuntugeek.com/install-and-configure-cacti-monitoring-tool-in-ubuntu-9-10-karmic-server.html)

Tuesday, November 2, 2010

Lubang Besar Menganga di Jerman

VIVAnews - Penduduk suatu kota kecil di Jerman Senin pagi, 1 November 2010, dikejutkan oleh suara bergemuruh yang terjadi di depan rumah mereka. Saat keluar, mereka kaget bukan kepalang mendapati sebuah lubang besar menganga di tengah-tengah kompleks pemukiman.

Menurut kantor berita Associated Press, lubang itu menyerupai kawah berdiameter 30 meter dan memiliki kedalaman hingga 20 meter.  Peristiwa ini diduga terjadi dalam semalam di suatu kompleks di Kota Schmalkalden, negara bagian Thuringia.

Lubang sebesar itu membuat hilang sebagian jalan, sebuah mobil yang terperosok serta sebuah pintu garasi warga. Beruntung, tidak ada yang terluka dalam peristiwa itu. Sementara sebuah mobil lainnya tergantung di tepi lubang.

Polisi berdatangan setelah warga melaporkan suara bergemuruh yang bising pada pukul 3 dini hari. Takut terjadi lagi pembentukan lubang pada pinggiran, polisi mengevakuasi enam rumah yang terdiri dari 25 orang.

Petugas dari badan geologi kota tersebut, Lutz Katschmann, belum dapat memastikan apa penyebab terbentuknya lubang tersebut. Namun kemungkinan besar, ujarnya, lubang terbentuk akibat formasi bebatuan yang terdiri dari batu kapur, batuan garam dan kalsium sulfat yang terkikis oleh air sehingga menciptakan sebuah rongga besar yang berujung kepada longsor.

“Kami, para ahli geologi, percaya bahwa ini adalah lubang yang terjadi secara alami. Namun kami tidak tahu pasti apa yang menyebabkannya,” ujarKatschmann seperti dikutip AP.

Katschmann mengatakan bahwa 20 lubang yang lebih kecil terbentuk di Thuringia setiap tahunnya. Dia mengatakan bahwa satu-satunya jalan untuk mangatasi hal ini adalah dengan menambal lubang menggunakan kerikil.

Lubang besar seperti itu bukan hanya terjadi di Jerman. Penduduk di ibukota Guatemala Juni lalu kaget dengan adanya sebuah lubang berdiameter 20 meter dan berkedalaman hampir 30 meter, yang menganga di sebuah perempatan jalan.

Lokasi lubang baru itu terletak dua kilometer dari lubang serupa yang terbentuk tiga tahun lalu. Lubang yang tercipta pada 2007 menewaskan tiga orang dan menelan beberapa rumah di kawasan yang sama.

Lubang besar juga pernah melanda Kota Quebec, Kanada dan Milwaukee, Amerika Serikat.

Sunday, October 24, 2010

Squid Tutorial From visolve



Table Of Contents



Introduction

Squid is a high-performance proxy caching server for web clients, supporting FTP, gopher, and HTTP data objects. Unlike traditional caching software, Squid handles all requests in a single, non-blocking, I/O-driven process.

Squid keeps meta data and especially hot objects cached in RAM, caches DNS lookups, supports non-blocking DNS lookups, and implements negative caching of failed requests.It supports SSL, extensive access controls, and full request logging. By using the lightweight Internet Cache Protocol, Squid caches can be arranged in a hierarchy or mesh for additional bandwidth savings.

Squid consists of a main server program squid, a Domain Name System lookup program dnsserver, some optional programs for rewriting requests and performing authentication, and some management and client tools. When squid starts up, it spawns a configurable number of dnsserver processes, each of which can perform a single, blocking Domain Name System (DNS) lookup. This reduces the amount of time the cache waits for DNS lookups.

This web caching software works on a variety of platforms including Linux, FreeBSD, and Windows. Squid is created by Duane Wessels.



Operating Systems Supported by Squid

  • Linux
  • FreeBSD
  • NetBSD
  • OpenBSD
  • BSDI
  • Mac OS/X
  • OSF/Digital Unix/Tru64
  • IRIX
  • SunOS/Solaris
  • NeXTStep
  • SCO Unix
  • AIX
  • HP-UX
  • OS/2
  • Cygwin


Installation Squid
Downloading Squid

Squid can be download as a squid source archive file in a gzipped tar ball form (eg.squid-*-src.tar.gz) available at http://www.squid-cache.org/ or from ftp://www.squid-cache.org/pub

squid can also be downloaded as an binary from http://www.squid-cache.org/binaries.html

Installing Squid from Source

1.Extract the source
tar xzf squid-*-src.tar.gz


2.Change the current directory to squid-*
cd squid-*


3.Compile and Installing squid
./configure
make
make install


Note:
This will by default, get installed in "/usr/local/squid".
To get more help for the compile time options available in squid.
./configure .help


Creating Squid Swap Directories

The Squid swap directories could be created by the following command

#/usr/local/squid/sbin/squid -z

Start, Stop & Restarting Squid

Start Squid #/usr/local/squid/sbin/squid

Stop Squid
Stopping squid . #/usr/local/squid/sbin/squid -k shutdown

Restart Squid
Stopping squid . #/usr/local/squid/sbin/squid -k shutdown
Starting squid - #/usr/local/squid/sbin/squid

Options Available
-k reconfigure|rotate|shutdown|interrupt|kill|debug|check|parse
Parse configuration file, then send signal to
running copy (except -k parse) and exit.


Running Squid as Daemon

For running squid as a daemon or a background process, it could be started as

#/usr/local/squid/sbin/squid -N

Starting Squid in Debugging Mode

Squid can be started in debugging mode by running squid as given below.

#/usr/local/squid/sbin/squid -Ncd1

which gives a debugging output.
If the test is perfect then it would print .Ready to serve requests..

Check Squid Status

To check whether squid is running the following command could be used.

#/usr/local/squid/sbin/squid -k check


Configuration

Basic Configuration

Squid Listening to a Particular Port

The option http_port specifies the port number where squid will listen for HTTP client requests. If this option is set to port 80, the client will have the illusion of being connected to the actual web server. Squid by default listen to the port 3128

Different modes of Squid Configuration

Squid could be configured in three different modes as Direct proxy, Reverse proxy and Transparent proxy.

Direct Proxy Cache

Direct proxy cache is used to cache static web pages (html and images) to a squid machine. When the page is requested second time, the browser returns the data from the proxy instead of the origin web server. The browser is explicitly configured to direct all HTTP requests to the proxy cache, rather than the target web server. The cache then either satisfies the request itself or passes on the request to the target server.

Configuring as Direct Proxy

By default, squid is configured in proxy mode. In order to cache web traffic and to use the squid system as a proxy, you have to configure your browser, which needs at least two pieces of information:
Set the proxy server's host name
Set the port that the proxy server is accepting requests on

Transparent Cache

Transparent cache achieves the same goal as a standard proxy cache, but operates transparently to the browser. The browser does not need to be explicitly configured to access the cache. Instead, the transparent cache intercepts network traffic, filters HTTP traffic (on port 80) and handles the request if the object is in the cache. If the object is not in the cache, the packets are forwarded to the origin web server.


Configuring as Transparent Proxy

Using squid transparently is a two part process, requiring first that squid be configured properly to accept non-proxy requests (performed in the squid module) and second that web traffic gets redirected to the squid port (achieved in three ways namely policy based routing, Using smart switching or by setting squid Box as a gateway).

Getting transparent caching to work requires the following steps

For some operating systems, have to configure and build a version of Squid which can recognize the hijacked connections and discern the destination addresses. For Linux this seems to work automatically. For BSD-based systems, you probably have to configure squid with the --enable-ipf-transparent option, and you have to configure squid as

httpd_accel_host virtual
httpd_accel_port 80
httpd_accel_with_proxy on
httpd_accel_uses_host_header on


You have to configure your cache host to accept the redirected packets - any IP address, on port 80 - and deliver them to your cache application. This is typically done with IP filtering/forwarding features built into the kernel. On linux they call this ipfilter (kernel 2.4.x), ipchains (2.2.x) or ipfwadm (2.0.x). On FreeBSD and other BSD systems they call it ip filter or ipnat; on many systems, it may require rebuilding the kernel or adding a new loadable kernel module.


Reverse Proxy Cache

A reverse proxy cache differs from direct and transparent caches, in that it reduces load on the origin web server, rather than reducing upstream network bandwidth on the client side. Reverse Proxy Caches offload client requests for static content from the web server, preventing unforeseen traffic surges from overloading the origin server. The proxy server sits between the Internet and the Web site and handles all traffic before it can reach the Web server. A reverse proxy server intercepts requests to the Web server and instead responds to the request out of a store of cached pages. This method improves the performance by reducing the amount of pages actually created "fresh" by the Web server.


Configuring as Reverse Proxy

To set Squid up to run as an accelerator then you probably want to listen on port 80. And finally you have to define the machine you are accelerating for. This is done in squid module,
http_port 80
httpd_accel_host visolve.com
httpd_accel_port 81
httpd_accel_single_host on
httpd_accel_with_proxy on

If you are using Squid as an accelerator for a virtual host system, then instead of a 'hostname' here you have to use the word virtual as:

http_port 80
httpd_accel_host virtual
httpd_accel_port 81
httpd_accel_with_proxy on

Different method of Intercepting HTTP Traffic

The methods could found in detail in the following link.

http://www.visolve.com/squid/whitepapers/trans_caching.php


WCCP configuration

Does Squid supports wccp?

Yes, Squid supports WCCP. Routers that support WCCP can be configured to direct traffic to one or more web caches using an efficient load balancing mechanism. WCCP also provides for automatic bypassing of an unavailable cache in the event of a failure

Configuring Squid for WCCP Support

Patches to be applied for linux kernel.

The linux kernel in the squid machine should be patched with ip_wccp as ip_gre is some what broken. Recompile the kernel enabling ip_gre and ip_wccp.

Now install the squid from source and configure it in the squid.conf to point to the WCCP router.

Squid Machine configuration.
The following iptables rule to be made so as to redirect all the http traffic to squid port 3128.
iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-ports 3128

Cache Inside the Routers network

If the cache is inside the routers network the packets coming from caches should be prevented from being redirected back to the caches again. So the the following firewall rule has to be prepended in the router machine.

iptables -t mangle -A PREROUTING 1 -p tcp --dport 80 -s -j ACCEPT


SNMP Configuration

Enabling SNMP support to Squid

To use SNMP with squid, it must be enabled with the configure script, and rebuilt. To enable SNMP in squid go to squid src directory and follow the steps given below :

./configure --enable-snmp [ ... other configure options ]

make all
make install

And edit following tags in squid.conf file :

acl aclname snmp_community public
snmp_access aclname

Once you configure squid and SNMP server, Start SNMP and squid.


Why should i go for SNMP?

SNMP in squid is useful in longer term overview of how proxy is doing. It can also be used as a problem solver. For example: how is it going with your file descriptor usage? Or how much does your LRU vary along a day? These informations can not be monitored normally.

Monitoring Squid

There are a number of tools used to monitor Squid via SNMP, among which MRTG is mostly used. The Multi Router Traffic Grapher (MRTG) is a tool to monitor squid information which generates a real-time status (graphical representation), in dynamic view by sampling data every five minutes (may vary according to your need). MRTG shows activity - in the last 24 hours and also in a weekly, monthly and yearly graph.

Parameters Monitored

Squid runtime information like CPU usage, Memory usage, Cache Hit, Miss etc., can be monitored using SNMP.

Delay Pools Configuration


Limiting Bandwidth

Delay Classes are generally used in places where bandwidth is expensive. They let you slow down access to specific sites (so that other downloads can happen at a reasonable rate), and they allow you to stop a small number of users from using all your bandwidth (at the expense of those just trying to use the Internet for work).
To ensure that some bandwidth is available for work-related downloads, you can use delay-pools. By classifying downloads into segments, and then allocating these segments a certain amount of bandwidth (in kilobytes per second), your link can remain uncongested for useful traffic.
To use delay-pools you need to have compiled Squid with the appropriate source code: you will have to have used the --enable-delay-pools option when running the configure program

An acl-operator (delay_access) is used to split requests into pools. Since we are using acls, you can split up requests by source address, destination url or more.


Configuring Squid with Delay Pools

To enable delay pools option,
Compile squid with --enable-delay-pools
Example
acl tech src 192.168.0.1-192.168.0.20/32
acl no_hotmail url_regex -i hotmail
acl all src 0.0.0.0/0.0.0.0
delay_pools 1 #Number of delay_pool 1
delay_class 1 1 #pool 1 is a delay_class 1
delay_parameters 1 100/100
delay_access 1 allow no_hotmail !tech

In the above example, hotmail users are limited to the speed specified in the delay_class. IP's in the ACL tech are allowed in the normal bandwidth. You can see the usage of bandwidth through cachemgr.cgi.



Caching

Can squid cache FTP contents?

Squid is a http proxy with ftp support, not a real ftp proxy. It can download from ftp, it can also upload to some ftp, but it can't delete/change name of files on remote ftp servers.When we block ports 20 and 21, we won't be able to delete/change name of files on remote ftp servers.It speaks FTP on the server-side, but not on the client-side

Can squid Cache dynamic pages?

Squid will not be able to cache pages that dynamically generate the scripts. It will cache only the static pages.

Deleing Objects from Cache

Deletion of object from is possible by using .purging. method.

Squid does not allow you to purge objects unless it is configured with access controls in squid.conf. First you must edit the following tag in squid.conf as

acl PURGE method PURGE
acl localhost src 127.0.0.1
http_access allow PURGE localhost
http_access deny PURGE

The above allows purge requests which come from the local host and denies all other purge requests.

/usr/local/squid/bin/client -m PURGE


Specifing Cache Size

Cache size could be specified by

Using cache_dir directive in squid.conf,

cache_dir ufs /usr/local/squid/cache 100 16 256


Here ufs is the squid filesystem, /usr/local/squid/cache is the default cache directory, 100 is the cache size in MB . The cache size could be specified here
and 16 and 256 are the number of sublevel directories in cache directory.

Squid Swap Formats

The squid swap formats systems available are

ufs,aufs,diskd and coss




Authentication

Configuring Squid for authenticating users

Squid allows you to configure user authentication by using auth_param directive.This is used to define parameters for the various authentication schemes supported by Squid.

Proxy authentication in transparent mode

Authentication can't be used in a transparently intercepting proxy as the client then thinks it is talking to an origin server and not the proxy. This is a limitation of bending the TCP/IP protocol to transparently intercepting port 80, not a limitation in Squid.

Authentication schemes available for squid

The Squid source code comes with a few authentication processes for Basic authentication. These include

LDAP: Uses the Lightweight Directory Access Protocol
NCSA: Uses an NCSA-style username and password file.
MSNT: Uses a Windows NT authentication domain.
PAM: Uses the Linux Pluggable Authentication Modules scheme.
SMB: Uses a SMB server like Windows NT or Samba.
getpwam: Uses the old-fashioned Unix password file.
sasl: Uses SALS libraries.
winbind: Uses Samba authenticate in a Windows NT domain

In addition Squid also supports the NTLM and Digest authentication schemes which both provide more secure authentication methods where the password is not exchanged in plain text.


Configuring squid for LDAP authentication

Compiling squid with ldap support.
./configure --enable-basic-auth-helpers="LDAP"

In squid.conf file edit the following

For Example
auth_param basic program /usr/local/squid/libexec/squid_ldap_auth -b dc=visolve,dc=com -f uid=%s -h visolve.com
acl password proxy_auth REQUIRED
http_access allow password
http_access deny all


Check Squid working with LDAP auth

To check whether the Squid machine communicates with the LDAP server Use the below command in command line

Example:
# /usr/local/squid/libexec/squid_ldap_auth -b dc=visolve,dc=com -f uid=%s visolve.com

This waits for the input.You have to give uid space passwd. If it was able to connect to LDAP server it will return "ok".


LDAP group authentication

Compiling squid with ldap support.
./configure --enable-basic-auth-helpers="LDAP" --enable-external-acl-helpers=ldap_group

In the confiuration file (squid.conf)

external_acl_type group_auth %LOGIN /usr/local/squid/libexec/squid_ldap_group -b "dc=visolve,dc=com" -f " (&(objectclass=groupOfUniqueNames)(cn=%a)(uniqueMember=uid=%v,cn=accounts,dc=visolve,dc=com))" -h visolve.com

acl gsrc external group_auth accounts
http_access allow gsrc


configuring Squid for NCSA

NCSA Authentication

This is the easiest to implement and probably the preferred choice for many environments. This type of authentication uses an Apache style htpasswd file, which is checked whenever anyone logs in. This is the best supported option, and a web based password changing program is provided to make it easy for our users to maintain their own passwords

To turn on NCSA authentication, edit some directives in squid.conf

authenticate_program /usr/local/squid/bin/ncsa_auth /usr/local/squid/etc/passwd

This tells Squid where to find the authenticator. Next we have to create an ACL.

Acl configuration for ncsa_auth :

acl auth_users proxy_auth REQUIRED
http_access allow auth_users
http_access deny all

Configuring Squid for SMB

SMB Auth Module :

smb_auth is a proxy authentication module. With smb_auth we can authenticate proxy users against an SMB server like Windows NT or Samba.

Adding smb_auth in Squid.conf :

Squid Configuration :

To turn on SMB authentication, edit some directives in squid.conf.

authenticate_program /usr/local/squid/bin/smb_auth -W domain -S /share/path/to/proxyauth

This tells Squid where to find the authenticator. Next we have to create an ACL .

Acl configuration for smb_auth :

acl domainusers proxy_auth REQUIRED
http_access allow domainusers
http_access deny all


Configuring squid for MSNT

MSNT Auth Module :

MSNT is a Squid web proxy authentication module. It allows a Unix web proxy to authenticate users with their Windows NT domain credentials.

Adding msnt_auth in Squid.conf :

Squid Configuration :

To turn on MSNT authentication, edit some directives in squid.conf

auth_param basic program /usr/local/squid/libexec/msnt_auth
auth_param basic children 5
auth_param basic realm Squid proxy-caching web server
auth_param basic credentialsttl 2 hours

This tells Squid where to find the authenticator. Next we have to create an ACL

Acl configuration for msnt_auth :

acl auth_users proxy_auth REQUIRED
http_access allow auth_users
http_access deny all

Configure squid for PAM

PAM Auth Module :

This program authenticates users against a PAM configured authentication service "squid". This allows us to authenticate Squid users to any authentication source for which we have a PAM module.

Adding pam_auth in Squid.conf

Squid Configuration

To turn on PAM authentication, edit some directives in squid.conf.

authenticate_program /usr/local/squid/bin/pam_auth

This tells Squid where to find the authenticator. Next we have to create an ACL .

Acl configuration for pam_auth :

acl auth_users proxy_auth REQUIRED
http_access allow auth_users
http_access deny all


Configure squid for NTLM

NTLM authentication is a challenge-response authentication type. NTLM is a bit different and does not obey the standard rules of HTTP connection management. The authentication is a three step (5 ways) handshake per TCP connection, not per request.

1a. Client sends unauthenticated request to the proxy / server.

1b. Proxy / server responds with "Authentication required" of type NTLM.

2a. The client responds with a request for NTLM negotiation

2b. The server responds with a NTLM challenge

3a. The client responds with a NTLM response

3b. if successful the connection is authenticated for this request and onwards. No further authentication exchanges takes place on THIS TCP connection.

Adding ntlm_auth and passwd file in Squid.conf

Squid Configuration:

To turn on NTLM authentication, edit some directives in squid.conf.

auth_param ntlm program /usr/local/squid/libexec/ntlm_auth (domainname)/(pdc name)
auth_param ntlm children 5
auth_param ntlm max_challenge_reuses 0
auth_param ntlm max_challenge_lifetime 2 minutes

This tells Squid where to find the authenticator. Next we have to create an ACL.

Acl configuration for ntlm_auth :

acl auth_users proxy_auth REQUIRED
http_access allow auth_users
http_access deny all




Filtering

Filtering a website

Filtering of websites could be made with ACL (Access Control List). Here is an example of denying a group of ip addresses to a specific domain.

acl block_ips src
acl block_domain dstdomain

http_access deny block_ips block_domain
http_access allow all


Denying a user from accessing particular site

Denying a user from accessing particular site coule be done by ACLs.
It is possible by using 'dstdomain' acl type.

For example..

acl sites dstdomain .gap.com .realplayer.com .yahoo.com

http_access deny sites


Filter a particular port

Filtering a particular port could be done in ACL as follows

acl block_port port 3456
http_access deny block_port
http_access allow all


Denying or allowing users

Denying access to websites for a particular timing could be done as follows.

To restrict the client from a source IP to access a particular domain during 9am-5pm on Monday,

acl names src
acl site dstdomain
acl acltime time M 9:00-17:00

http_access deny names site acltime
http_access allow all


What all squid cant filter?

Squid cannot filters virus and web pages based on content.

Filtering a Particular MAC address

To use ARP (MAC) access controls, you first need to compile in the optional code. Do this with the --enable-arp-acl configure option.

Example:

acl M1 arp 01:02:03:04:05:06
acl M2 arp 11:12:13:14:15:16
http_access allow M1
http_access allow M2
http_access deny all




Performance

Monitoring Squid Performance

Squid performance is monitored by using cache manager and SNMP.
Cache Manager:
This provides access to certain information needed by the cache administrator. A companion program, cachemgr.cgi can be used to make this information available via a Web browser. Cache manager requests to Squid are made with a special URL of the form

cache_object://hostname/operation

The cache manager provides essentially ``read-only'' access to information. It does not provide a method for configuring Squid while it is running.

SNMP:

SNMP could be used for monitoring squid runtime information like CPU usage, Memory usage, Cache Hit, Miss etc. The Multi Router Traffic Grapher (MRTG) is a tool to monitor squid information which generates a real-time status (graphical representation), in dynamic view by sampling data every five minutes.

Improving Squid Performance

Squid performance could be improved by gathering the performance data for the particular environment and tuning the Hardware and Kernel parameters for the peak performance.

Does the cache directory filesystem impact the performance?

The Cache directory has the default option ufs. When it is made with the following

cache_dir aufs

The aufs storage scheme improves the Squid.s disk I/O response time by using a number of thread processes for disk I/O operations .The aufs code requires a pthreads library. This is the standard threads interface defined by POSIX. To use aufs squid must be compiled with storeio option.

Note:

If disk caching is not used, it should be disabled by setting to 'null /tmp'.
This eliminates the need for meta-data cache index memory space used by squid.





Log files

Log files produced by squid

The list of log files produced by squid are

squid.out, cache.log, useragent.log, store.log, hierarchy.log, access.log.

Monitoring User Access

The access information gets stored in the access.log file.

Rotating Log

Larger log files could be handled by rotating the same.This could be done with the following command

squid -k rotate

To specify the number of logfile rotations to make when you type 'squid -k rotate' configure it in the squid.conf file in logfile_rotate directive.

Scheduling of this procedure could be done by Cron entry which rotates logs at midnight.

0 0 * * * /usr/local/squid/bin/squid -k rotate


Can squid supports logs of size greater than 2GB?

Squid by default doesnt supports logs of size greater than 2 GB.To make the squid supports files of size greater than 2GB compile the squid with the option(--with-large-files)

Disbaling Squid Log File

Disabling log files could be done
To disable access.log:
cache_access_log none
To disable store.log:
cache_store_log none
To disable cache.log:
cache_log /dev/null




Tools

Cache Manger (cachemgr.cgi)

The cache manager (cachemgr.cgi) is a CGI utility for displaying statistics about the squid process as it runs. The cache manager is a convenient way to manage the cache and view statistics without logging into the server.


Tools For Configuring Squid

There are many tools available to configure squid like webmin and so on.

You can get these tools from

http://www.squid-cache.org/related-software.html


Log Analysers

Calamaris

It is a commonly used tool to analyze Squid's access.log. It Supports many features like generating Status reports of incoming UDP-Requests and incoming TCP-Requests for total as well as on per host basis; Reports about requested second-level-domains and Top_Level_domains are generated; And also Reports about requested Content Types, file_extension and on Protocols are generated using calamaris. It generates ASCII or html reports. For a full list of features, please visit the Calamaris home page.

Weblog

WebLog is a group of Python modules containing several class definitions that are useful for parsing and manipulating common Web and Web proxy logfile formats.

The Webalizer

The Webalizer is a fast, free web server log file analysis program It is written in C to be extremely fast and highly portable. The results are presented in both columnar and graphical format. Yearly, monthly, daily and hourly usage statistics are presented, along with the ability to display usage by site, URL, referrer, user agent , search string, entry/exit page, username and country. Processed data may also be exported into most database and spreadsheet programs that support tab delimited data formats. In addition, wu-ftpd xferlog formatted logs and squid proxy logs are supported.

SARG

Sarg is a Squid Analysis Report Generator that allow you to view "where" your users are going to on the Internet. Sarg generates reports in html, with many fields, like: users, IP Addresses, bytes, sites and times.


Tools to generate user web access report

Webmin is a web-based tool for generating web access reports. Using any browser that supports tables and forms (and Java for the File Manager module), you can setup user accounts, Apache, DNS, file sharing and so on.

Webmin consists of a simple web server, and a number of CGI programs which directly update system files like /etc/inetd.conf and /etc/passwd. The web server and all CGI programs are written in Perl version 5, and use no non-standard Perl modules.


Miscellaneous


Controlling Uploads

The uploads can be controlled by using acls(req_header).

acl upload_control req_header Content-Length [1-9][0-9][0-9][0-9][0-9]{3,}
http_access deny upload_control
http_access allow all

Controlling Downloads

The Downloads can be controlled by using the following directive.

reply_body_max_size bytes allow|deny acl


{from: http://www.visolve.com/squid/Squid_tutorial.php}