|
Jim Gallacher
jpg at jgassociates.ca
Thu Jun 28 17:27:20 EDT 2007
I've confirmed that mod_python leaks on graceful restart, but I don't see the dependency on the auth* modules. Apache always seems to leak when mod_python is loaded, regardless of the other modules. My test setup is a little different: On Debian with apache 2.2.3, mpm-worker, mod_python 3.3.1 running within VMWare. My gut tells me that this may be related to http://issues.apache.org/jira/browse/MODPYTHON-109. This issue also reminds me of http://issues.apache.org/jira/browse/MODPYTHON-195. I think at this point we should log this in JIRA and move the discussion over to python-dev. The obvious (but suboptimal) temporary solution for Aseem is to avoid using graceful restart. Jim Aseem Mohanty wrote: > Graham Dumpleton wrote: >> Is the version of mod_python you are using one you compiled yourself, >> or one from Ubuntu packages? > > I have used both and it does not seem to matter. When compiling from > source, I did the following: > ./configure --prefix=/opt --with-apxs=/opt/share/apache2/bin/apxs > make install_dso > to use the apache build from source. Apache was build with the following > config: > ./configure --prefix=/opt --enable-modules=all --enable-mods-shared=all > --enable-so --with-included-apr > >> >> Is the version of Python you are using one you compiled yourself, or >> one from Ubuntu packages? > > Python-2.5.1 is the one from the ubuntu packages.05:22 PM $ python > Python 2.5.1 (r251:54863, May 2 2007, 16:56:35) > [GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >> >> What is the size of mod_python.so? Does it use a shared Python library >> or static? Use 'ldd' on mod_python.so to see dependencies. > > mod_python from ubuntu packages: > v 3.2.10 > 05:24 PM $ ll mod_python.so > -rw-r--r-- 1 root root 99K 2007-01-16 04:44 mod_python.so > 05:24 PM $ ldd mod_python.so > linux-gate.so.1 => (0xffffe000) > libpython2.5.so.1.0 => /usr/lib/libpython2.5.so.1.0 (0xb7e4f000) > libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0xb7e38000) > libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb7e33000) > libutil.so.1 => /lib/tls/i686/cmov/libutil.so.1 (0xb7e2f000) > libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb7e08000) > libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7cc7000) > /lib/ld-linux.so.2 (0x80000000) > > mod_python built from source: > v 3.3.1 > 05:25 PM $ ll mod_python.so > -rwxr-xr-x 1 aseem aseem 343K 2007-06-27 09:50 mod_python.so* > aseem @ narada: /opt/share/apache2/modules > 05:25 PM $ ldd mod_python.so > linux-gate.so.1 => (0xffffe000) > libpython2.5.so.1.0 => /usr/lib/libpython2.5.so.1.0 (0xb7e30000) > libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0xb7e19000) > libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb7e14000) > libutil.so.1 => /lib/tls/i686/cmov/libutil.so.1 (0xb7e10000) > libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb7de9000) > libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7ca8000) > /lib/ld-linux.so.2 (0x80000000) > > In both cases they do seem to use shared python libs. > > Thanks. > AM > >> >> Graham >> >> On 28/06/07, Aseem Mohanty <aseem at metaweb.com> wrote: >>> Yes, hence the exotic phraseology. Results for the various >>> combinations are below: >>> >>> 1. mod_python only, no mod_auth* modules: leaks >>> 2. mod_auth_digest only, no other mod_auth*, no mod_python: does not >>> leak >>> 3. mod_authz_host only, no other mod_auth*, no mod_python: does not >>> leak >>> 4. mod_python + mod_authz_host, no mod_auth_digest: leaks >>> 5. mod_python + mod_auth_digest, no mod_authz_host: leaks < >>> (1) or (4) >>> 6. mod_python + mod_auth_digest + mod_authz_host: does not >>> leak >>> >>> NOTE: 'does not leak' implies ~50-100K leakage per graceful restart >>> (5) leaks ~500-600K per graceful restart >>> zero load on server >>> >>> As far as I can tell there is definitely some sinister interaction >>> going on >>> between these three modules. Unfortunately not being conversant with >>> apache >>> module development, I cannot really say what. >>> >>> Thanks. >>> AM >>> >>> >>> Jim Gallacher wrote: >>> > Aseem Mohanty wrote: >>> >> This may sound quite exotic >>> > >>> > >>> > Why yes, it does? Interaction between these modules resulting in a >>> > memory leak sure sounds weird. Did you try it with *only* mod_python >>> > loaded? (ie, without mod_authz_host) >>> > >>> > Jim >>> > >>> >> but here it is: >>> >> >>> >> In a vanilla apache2.2.x configuration if I only load mod_authz_host >>> >> and none of >>> >> the other mod_auth* modules, every graceful restart causes ~1Mb leak >>> >> in the >>> >> parent process. This is with mod_python loaded. Without mod_python >>> the >>> >> memory profile remains unchanged. >>> >> >>> >> Steps to reproduce: >>> >> Gentoo: >>> >> 1. In /etc/apache2/httpd.conf comment out all "LoadModule auth*" >>> >> directives except >>> >> LoadModule authz_host_module modules/mod_authz_host.so >>> >> >>> >> Ubuntu Feisty (repos): >>> >> 1. In /etc/apache2/mods-enabled rename auth*.load files except >>> >> authz_host.load >>> >> to auth*.load.not >>> >> >>> >> Ubuntu Feisty (httpd built from source): >>> >> 1. In conf/httpd.conf comment out all "LoadModule auth*" directives >>> >> except >>> >> LoadModule authz_host_module modules/mod_authz_host.so >>> >> >>> >> 2. Start apache >>> >> 3. Track apache parent process: top -d1 -p $(pgrep -o apache/httpd) >>> >> 4. Hit apache instances with graceful restarts: >>> >> for i in $(seq 150); do echo $i; kill -USR1 $(pgrep -o apache/httpd); >>> >> sleep 2; done >>> >> >>> >> In this case the parent will leak ~1Mb per restart. If however in >>> each >>> >> of the >>> >> configurations mod_auth_digest.so is also loaded the leak is only >>> >> about 150-200K >>> >> or so per graceful restart. This is on a server with 0 load. In my >>> >> case it >>> >> caused apache instances that were restarted every 55 minutes to grow >>> >> up to 300Mb >>> >> in a week. >>> >> >>> >> Tested on: >>> >> 1. Ubuntu Feisty (built from source httpd-2.2.4, mod_python-3.3.1): >>> >> ./configure --prefix=/opt --enable-modules=all >>> >> --enable-mods-shared=all --enable-so --with-included-apr >>> >> Server version: Apache/2.2.4 (Unix) >>> >> Server built: Jun 27 2007 09:34:16 >>> >> Server's Module Magic Number: 20051115:4 >>> >> Server loaded: APR 1.2.8, APR-Util 1.2.8 >>> >> Compiled using: APR 1.2.8, APR-Util 1.2.8 >>> >> Architecture: 32-bit >>> >> Server MPM: Prefork >>> >> threaded: no >>> >> forked: yes (variable process count) >>> >> Server compiled with.... >>> >> -D APACHE_MPM_DIR="server/mpm/prefork" >>> >> -D APR_HAS_SENDFILE >>> >> -D APR_HAS_MMAP >>> >> -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) >>> >> -D APR_USE_SYSVSEM_SERIALIZE |