Tag Archives: squid

Patch for SquidAnalyzer 6.6 to use standard date format

SquidAnalyzer is a great tool to visualize statistics for the Squid web proxy. Unfortunately up until version 6.6 there is no way to configure the date format used to parse Squid logfiles.

By default Squid uses a Unix timestamp for its access log which is hard to read. If you change that date format to a more readable string, SquidAnalyzer does not work.

Here is a patch that makes SquidAnalyzer 6.6 recognize the following date format:
%{%Y-%m-%d %H:%M:%S}tl %6tr %>a %Ss/%03>Hs %<st %rm %ru %[un %Sh/%<a %mt

This is basically the same format as the native squid_localtime format, except the date is displayed human readable (year-month-day hour:minute:second).

The patch for version 6.6 must be applied to the file SquidAnalyzer.pm before installation:
SquidAnalyzer.pm.patch

--- /usr/local/src/squidanalyzer-6.6/SquidAnalyzer.pm   2017-07-23 10:56:28.379684965 +0200
+++ SquidAnalyzer.pm    2017-07-23 11:43:43.336149777 +0200
@@ -404,6 +404,8 @@
my $ip_regexp = qr/^([a-fA-F0-9\.\:]+)$/;
my $cidr_regex = qr/^[a-fA-F0-9\.\:]+\/\d+$/;

+# Patch: %{%Y-%m-%d %H:%M:%S}tl %6tr %>a %Ss/%03>Hs %<st %rm %ru %[un %Sh/%<a %mt
+my $de_format_regex1 = qr/^(\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2})\s+(\d+)\s+([^\s]+)\s+([^\s]+)\s+(\d+)\s+([^\s]+)\s+(.*)/;
# Native log format squid %ts.%03tu %6tr %>a %Ss/%03>Hs %<st %rm %ru %un %Sh/%<A %mt
my $native_format_regex1 = qr/^(\d+\.\d{3})\s+(\d+)\s+([^\s]+)\s+([^\s]+)\s+(\d+)\s+([^\s]+)\s+(.*)/;
my $native_format_regex2 = qr/^([^\s]+?)\s+([^\s]+)\s+([^\s]+\/[^\s]+)\s+([^\s]+)\s*/;
@@ -535,8 +537,19 @@

my $time = 0;
my $tz = ((0-$self->{TimeZone})*3600);
-       # Squid native format
-       if ( $line =~ $native_format_regex1 ) {
+        # Patch
+        if ( $line =~ $de_format_regex1 ) {
+                $time = $1;
+                $time =~ /(\d{4})-(\d{2})-(\d{2})\s+(\d{2}):(\d{2}):(\d{2})/;
+                if (!$self->{TimeZone}) {
+                        $time = timelocal_nocheck($6, $5, $4, $3, $2 - 1, $1 - 1900);
+                } else {
+                        $time = timegm_nocheck($6, $5, $4, $3, $2 - 1, $1 - 1900) + $tz;
+                }
+                $self->{is_squidguard_log} = 0;
+                $self->{is_ufdbguard_log} = 0;
+        # Squid native format
+        } elsif ( $line =~ $native_format_regex1 ) {
$time = $1;
$self->{is_squidguard_log} = 0;
$self->{is_ufdbguard_log} = 0;
@@ -596,6 +609,11 @@
$self->{is_ufdbguard_log} = 1;
$self->{is_squidguard_log} = 0;
last;
+                # Patch
+                } elsif ( $line =~ $de_format_regex1 ) {
+                        $self->{is_squidguard_log} = 0;
+                        $self->{is_ufdbguard_log} = 0;
+                        last;
# Squid native format
} elsif ( $line =~ $native_format_regex1 ) {
$self->{is_squidguard_log} = 0;
@@ -1237,7 +1255,23 @@
#logformat combined   %>a %[ui %[un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st "%{Referer}>h" "%{User-Agent}>h" %Ss:%Sh
# Parse log with format: time elapsed client code/status bytes method URL rfc931 peerstatus/peerhost mime_type
my $format = 'native';
-               if ( !$self->{is_squidguard_log} && !$self->{is_ufdbguard_log} && ($line =~ $native_format_regex1) ) {
+                # Patch
+                if ( !$self->{is_squidguard_log} && !$self->{is_ufdbguard_log} && ($line =~ $de_format_regex1) ) {
+                        $time = $1;
+                        #$time += $tz;
+                        $elapsed = abs($2);
+                        $client_ip = $3;
+                        $code = $4;
+                        $bytes = $5;
+                        $method = $6;
+                        $line = $7;
+                        $time =~ /(\d{4})-(\d{2})-(\d{2})\s+(\d{2}):(\d{2}):(\d{2})/;
+                        if (!$self->{TimeZone}) {
+                                $time = timelocal_nocheck($6, $5, $4, $3, $2 - 1, $1 - 1900);
+                        } else {
+                                $time = timegm_nocheck($6, $5, $4, $3, $2 - 1, $1 - 1900) + $tz;
+                        }
+                } elsif ( !$self->{is_squidguard_log} && !$self->{is_ufdbguard_log} && ($line =~ $native_format_regex1) ) {
$time = $1;
$time += $tz;
$elapsed = abs($2);

Share

Squid, c-icap, ClamAV: Bug in the service. Please report to the service author!!!!

If you see this error in your c-icap server logfile, it might just be that c-icap is running out of temporary disk space and that the clamav/virus scanner configuration for c-icap is wrong:

Service antivirus_module virus_scan.so
ServiceAlias  avscan virus_scan?allow204=on&sizelimit=off&mode=simple
virus_scan.MaxObjectSize  5M
TmpDir /tmp

The option "... sizelimit=off..." for the virus_scan service means that the configuration value for "MaxObjectSize" will be ingored. If you have too many parallel squid client connections open or large files to download, c-icap is running out of temporary disk space. It will then log the following error message without further explanation:

Bug in the service. Please report to the service author!!!!

The webbrowser download will be terminated with an error message (something like "internal server error").

To solve this problem, add more free space to the partition where TmpDIr resides, and change the virus_scan service option to "... sizelimit=on ...".

In the worst case, free disk space for the c-icap TmpDIr has to be:
MaxServers * ThreadsPerChild * virus_scan.MaxObjectSize

Share

Squid performance

If there are only low cache hit rates, you can disable disk caching completely. Comment all cache_dir entries in squid.conf:

# cache_dir ...

Check with "squidclient mgr:info":
...
0 on-disk objects

(Why running squid without disk caching? See one of my next posts.)

 

Enable SMP mode and set CPU affinity, e.g. if you have 2 CPU cores:

workers 2
cpu_affinity_map process_numbers=1,2 cores=1,2

Check with "ps aux | grep squid":
root 21107 0.0 0.5 344768 5196 ? Ss Apr16 0:00 /usr/sbin/squid3 -YC -f /etc/squid3/squid.conf
proxy 21110 0.0 2.3 384584 24060 ? S Apr16 0:04 (squid-coord-3) -YC -f /etc/squid3/squid.conf
proxy 21111 0.0 2.5 387348 26536 ? S Apr16 0:13 (squid-2) -YC -f /etc/squid3/squid.conf
proxy 21112 0.2 3.2 391764 33024 ? S Apr16 2:06 (squid-1) -YC -f /etc/squid3/squid.conf

Notice the new worker processes "squid-1"  and "squid-2", and the new io process "squid-coord-3".

 

Share