How to find out why squid proxy exited due to signal 9 with status 0
I
am getting an error that read as “The proxy server is refusing connections.” I am using Squid 3 with Ubuntu Linux 16.04 LTS server and how can I find out why my squid proxy server died unexpectedly?
Squid is a caching and open source proxy for the Web supporting HTTP, HTTPS, FTP, and more. It reduces bandwidth and improves response times by caching and reusing. Squid can die due to any number of reasons. Let us see some basic steps to find out what caused such errors on Ubuntu Linux server.
Basic symptoms
You or user might see an error as follows on browser screen:
Fig.01: The first error message indicating something is wrong with our squid proxy server
Squid proxy server attached to the network and provided the proxy server. So let us start digging into the problem.
Check list #1: Client or server problem
Go to another client browser or computer and try to access the internet via squid proxy. If this computer also can’t access squid server, then you know the problem is more likely on squid server itself. Next, you need to focus troubleshooting on squid server itself. Login using ssh:
$ ssh vivek@server1.cyberciti.biz
Check list #2: Is server port open?
Type the following command on Ubuntu Linux server to verify that the TCP port 3128 (default) is open:
$ netstat -tulpn
$ ss -tulpn
$ netstat -tulpn | grep 3128
If no output means squid is not running. Try to start it, run:
$ sudo systemctl start squid
Check list #3: Is squid proxy server running?
Use the ps command along with the grep command as follows:
$ ps aux | grep -i squid
OR use the pgrep command:
$ pgrep squid
If squid still not starting or running, check for other errors.
Check list 4: Check squid log file or status of squid server
Type the following command:
$ sudo systemctl status squid
Bingo, I found the error:
Feb 04 04:09:01 nixcraft-squid-do-0001 squid[1658]: Squid Parent: (squid-1) process 1661 exited due to signal 9 with status 0
Random killing of the process indicates out of memory (OOM) situation. It is an often undesired state of computer operation where no additional memory can be allocated for use by programs or the operating system.
WHAT DOES SIGNAL 9 MEANS?
The 9 number indicates that kill signal i.e. kill running process. Naturally, someone killed the process. However, why? To find out the answer to our question, run:
$ dmesg | grep -i mem
or
$ dmesg | grep -i out_of_mem
Sample outputs:
858444.872840] lowmem_reserve[]: 0 0 0 0 0
[858444.872867] 0 pages HighMem/MovableOnly
[858444.872948] Out of memory: Kill process 1661 (squid) score 67 or sacrifice child
[860244.967476] sessionclean cpuset=/ mems_allowed=0
[860244.967512] [] out_of_memory+0x219/0x460
[860244.967521] [] alloc_kmem_pages_node+0x4b/0xc0
[860244.967544] Mem-Info:
mapped:10224 shmem:3037 pagetables:1029 bounce:0
[860244.967552] Node 0 DMA free:2136kB min:88kB low:108kB high:132kB active_anon:1824kB inactive_anon:384kB active_file:3760kB inactive_file:1776kB unevictable:68kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:68kB dirty:4kB writeback:0kB mapped:2048kB shmem:400kB slab_reclaimable:4448kB slab_unreclaimable:620kB kernel_stack:224kB pagetables:164kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[860244.967558] lowmem_reserve[]: 0 454 454 454 454
[860244.967561] Node 0 DMA32 free:10596kB min:2680kB low:3348kB high:4020kB active_anon:62740kB inactive_anon:11196kB active_file:175676kB inactive_file:110160kB unevictable:3588kB isolated(anon):0kB isolated(file):0kB present:507896kB managed:484228kB mlocked:3588kB dirty:204kB writeback:0kB mapped:38848kB shmem:11748kB slab_reclaimable:75600kB slab_unreclaimable:16648kB kernel_stack:2288kB pagetables:3952kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
So now you know exact reasons why our squid died. To fix this problem, you might need to stop the unwanted process. Another option is to add additional ram. In my case, I had more than enough RAM. It turns out it was a kernel bug – “Out of memory” errors after upgrade to kernel version 4.4.0-59 on Ubuntu 16.04 LTS:
$ uname -mrs
Linux 4.4.0-59-generic x86_64
The ultimate solution is to apply kernel fix by typing the apt-get command or apt command as follows:
$ sudo apt-get update
$ sudo apt-get upgrade
force upgrade ##
$ sudo apt-get dist-upgrade
Reboot the Linux server, enter:
$ sudo reboot
$ uname -mrs
Linux 4.4.0-62-generic x86_64
And, there you have it solution to find and fix the problem. I strongly suggest that you subscribe to bug tracking mailing list or feed.
