|
Alec Matusis
matusis at yahoo.com
Tue Jan 22 20:57:16 EST 2008
Here is another potentially useful cue: 32427 is this orphan process. # strace -p 32427 Process 32427 attached - interrupt to quit futex(0x5b7820, FUTEX_WAIT, 0, NULL and it's just stuck there. > -----Original Message----- > From: Graham Dumpleton [mailto:graham.dumpleton at gmail.com] > Sent: Tuesday, January 22, 2008 5:49 PM > To: Alec Matusis > Cc: mod_python at modpython.org > Subject: Re: [mod_python] remnant 'orphan' apache subprocesses > > BTW, originally you said: > > """I have been investigating a memory leak that occurs on an apache > server > since we switched to worker MPM. > I found that the source of it are apache subprocesses that lose track > of > their parent and never exit:""" > > If the processes are truly zombie processes, then they shouldn't > actually consume any resources except for the entry in the process > table. Ie., they are an process accounting artifact. > > So, strictly speaking they shouldn't be causing any memory leaks, or > more correctly consuming memory which isn't released. > > Graham > > On 23/01/2008, Alec Matusis <matusis at yahoo.com> wrote: > > > > [Tue Jan 22 10:44:45 2008] [notice] child pid 8798 exit signal > > > Segmentation > > > > fault (11) > > > > ... > > > > Fatal Python error: Inconsistent interned string state. > > > > > > This is corruption of memory used by Python. > > > > > > What version of mod_python are you using? > > > > Apache/2.2.6 (Unix) mod_python/3.3.1 Python/2.4.1 > > > > But since there is only one such log entry, I'd expect at most 1 > zombie as a > > result of this? I have 6. > > > > > -----Original Message----- > > > From: Graham Dumpleton [mailto:graham.dumpleton at gmail.com] > > > Sent: Tuesday, January 22, 2008 5:15 PM > > > To: Alec Matusis > > > Cc: mod_python at modpython.org > > > Subject: Re: [mod_python] remnant 'orphan' apache subprocesses > > > > > > On 23/01/2008, Alec Matusis <matusis at yahoo.com> wrote: > > > > > What do you get if you use a program like lsof or ofiles to > work > > > out > > > > > what open resources the zombie process may still be holding on > to? > > > > > > > > There are 6 zombie sub processes now; executing lsof -p pid takes > > > ages, and > > > > it brings the load average up from 8.0 to 23+ on this machine- so > I > > > am > > > > afraid to wait long enough to get a result. > > > > #ps -ef | grep httpd > > > > root 16197 1 0 Jan21 ? 00:00:15 > > > > nobody 23095 16197 0 08:44 ? 00:00:06 > > > > nobody 29548 1 0 13:14 ? 00:00:00 > > > > nobody 3812 1 0 13:57 ? 00:00:00 > > > > nobody 4161 1 0 13:59 ? 00:00:00 > > > > nobody 20110 1 0 15:43 ? 00:00:00 > > > > nobody 25399 1 0 16:17 ? 00:00:00 > > > > nobody 28722 1 0 16:38 ? 00:00:00 > > > > nobody 28971 16197 5 16:40 ? 00:00:20 > > > > nobody 29189 16197 7 16:42 ? 00:00:21 > > > > nobody 29327 16197 7 16:42 ? 00:00:18 > > > > nobody 29453 16197 6 16:43 ? 00:00:13 > > > > nobody 29496 16197 10 16:43 ? 00:00:20 > > > > nobody 29539 16197 9 16:43 ? 00:00:19 > > > > nobody 29639 16197 11 16:44 ? 00:00:14 > > > > nobody 29713 16197 11 16:45 ? 00:00:12 > > > > nobody 29804 16197 5 16:45 ? 00:00:05 > > > > nobody 29857 16197 10 16:45 ? 00:00:09 > > > > nobody 29902 16197 10 16:45 ? 00:00:08 > > > > nobody 29945 16197 11 16:46 ? 00:00:07 > > > > nobody 29998 16197 11 16:46 ? 00:00:06 > > > > nobody 30058 16197 16 16:47 ? 00:00:01 > > > > > > > > note that those zombie sub processes seem to have had 00:00:00 > run > > > time, > > > > unlike normal sub processes. > > > > 3 entries in apache error logs this time: > > > > > > > > [Tue Jan 22 10:44:45 2008] [notice] child pid 8798 exit signal > > > Segmentation > > > > fault (11) > > > > ... > > > > Fatal Python error: Inconsistent interned string state. > > > > > > This is corruption of memory used by Python. > > > > > > What version of mod_python are you using? > > > > > > > [Tue Jan 22 14:03:16 2008] [notice] child pid 4008 exit signal > > > Aborted (6) > > > > > > > > is there a faster way to see what it is holding on to? > > > > > > I know you said the applications weren't spawning sub processes, > but > > > if your system has 'ptree' try that. In other words, see if those > > > daemon processes have children which still exist. This was in part > > > what I was hoping to see, ie., pipes to child processes. The other > > > thing I was looking for was stuck file accesses to NFS mounted > > > filesystems or something. Alternative to ptree is just to look at > > > parent child relationships in ps output. > > > > > > Graham > > > > > > > > -----Original Message----- > > > > > From: Graham Dumpleton [mailto:graham.dumpleton at gmail.com] > > > > > Sent: Monday, January 21, 2008 10:27 PM > > > > > To: Alec Matusis > > > > > Cc: mod_python at modpython.org |