|
Graham Dumpleton
graham.dumpleton at gmail.com
Wed Nov 7 00:49:25 EST 2007
On 07/11/2007, Alec Matusis <matusis at yahoo.com> wrote: > After I wrote this, I realized that the problem was in > > from sys import modules > def _import( module_path ): > runme = modules[ module_path ] > > sys.modules was trying to use already loaded modules. > I got rid of it, and forced it to actually load modules every time, which > got rid of errors. > I wonder if forcing to load modules comes with a CPU penalty compared to > prefork however. Can't tell. You still haven't showed how the modules are actually being imported. Why aren't you just using apache.import_module() from mod_python? See: http://www.modpython.org/live/current/doc-html/pyapi-apmeth.html Graham > > -----Original Message----- > > From: mod_python-bounces at modpython.org [mailto:mod_python- > > bounces at modpython.org] On Behalf Of Alec Matusis > > Sent: Tuesday, November 06, 2007 6:43 PM > > To: 'Graham Dumpleton' > > Cc: mod_python at modpython.org > > Subject: RE: [mod_python] mod_python or apache scalability? > > > > > So, what is 'modules' and '_import' and how are they used? > > > > 'modules' is 'from sys import modules', so it's sys.modules... > > > > '_import' is used in the publisher like so: > > > > publisher.py: > > > > from common._import import _import > > > > def handler(req): > > ... > > dyn_path = req.uri[1:] #to get rid of '/' prefix > > module_path = string.replace(dyn_path,'/','.') > > runme = _import( module_path ) > > .... > > > > > > common/_import.py: > > > > from sys import modules > > def _import( module_path ): > > runme = modules[ module_path ] > > #get the last part > > class_name = module_path.split('.')[-1] > > try: > > runme = getattr(runme, class_name ) <--- INTERMITTENT ERROR > > here > > except AttributeError: > > raise Exception,'Module does not contain class '+class_name+', > > runme='+repr(runme)+' module_path='+repr(module_path) > > > > > How do you > > > ensure that two concurrently executing request handlers trying to > > > import the same module don't interfere with each other? > > > > We do not insure that- the code was written for prefork MPM. > > What is the best way to insure that? > > > > > > > -----Original Message----- > > > From: Graham Dumpleton [mailto:graham.dumpleton at gmail.com] > > > Sent: Tuesday, November 06, 2007 3:53 AM > > > To: Alec Matusis > > > Cc: Martijn Moeling; mod_python at modpython.org > > > Subject: Re: [mod_python] mod_python or apache scalability? > > > > > > On 06/11/2007, Alec Matusis <matusis at yahoo.com> wrote: > > > > I found that this that > > > > > > > > IOError: image file is truncated (14432 bytes not processed) > > > > > > > > has to do with using input filter > > > > > > > > PythonInputFilter flashfilter FLASHFILTER > > > > SetInputFilter FLASHFILTER > > > > > > > > It somehow behaves differently in apache 2.2.6/mod_python 3.3.1 > > > versus > > > > 2.0.54/ 3.1.4. For now, I removed this input filter (we can get > > away > > > without > > > > it), and it seems to work. > > > > > > > > However, we have a bigger problem: we now have intermittent import > > > errors > > > > (one error per about 50-80 requests). > > > > > > > > This code executed by publisher: > > > > > > > > import modules > > > > def _import( module_path ): > > > > runme = modules[ module_path ] > > > > #get the last part > > > > class_name = module_path.split('.')[-1] > > > > try: > > > > runme = getattr(runme, class_name ) <--- INTERMITTENT ERROR > > > here > > > > except AttributeError: > > > > raise Exception,'Module does not contain class > > > '+class_name+', > > > > runme='+repr(runme)+' module_path='+repr(module_path) > > > > > > > > produces this error: > > > > > > > > Exception: Module does not contain class getprofile, runme=<module > > > > 'getprofile' from '/path/scripts/getprofile.py'> > > > module_path='getprofile' > > > > > > > > This used to always work with prefork. Is there anything not thread > > > safe > > > > about importing modules? > > > > > > The underlying Python and mod_python import mechanisms themselves are > > > never usually the problem. It is thread unsafe practices regarding > > > initialisation in the code around the imports. > > > > > > So, if you have built your own form of module importer or higher > > level > > > layer for one and you haven't properly thread protected it, it is > > most > > > likely the cause. > > > > > > So, what is 'modules' and '_import' and how are they used? How do you > > > ensure that two concurrently executing request handlers trying to > > > import the same module don't interfere with each other? > > > > > > Graham > > > > > > > > -----Original Message----- > > > > > From: mod_python-bounces at modpython.org [mailto:mod_python- > > > > > bounces at modpython.org] On Behalf Of Graham Dumpleton > > > > > Sent: Monday, November 05, 2007 1:29 PM > > > > > To: Martijn Moeling > > > > > Cc: mod_python at modpython.org; Alec Matusis > > > > > Subject: Re: [mod_python] mod_python or apache scalability? > > > > > > > > > > The LimitRequestBody directive support in mod_python is broken > > and > > > > > most likely would have resulted in a 500 error occurring during > > > > > mod_python.publisher processing of the form even before the user > > > code > > > > > was run. > > > > > > > > > > My guess at the reason is that they have enabled in the new > > Apache > > > > > server compression on request content. This may not work with > > > > > mod_python because how it reads request content is broken, with > > it > > > > > only reading up until original Content-Length in certain cases. > > If > > > > > there is a mutating input filter such as content decompression, > > you > > > > > will get truncated input data. > > > > > > > > > > https://issues.apache.org/jira/browse/MODPYTHON-240 > > > > > https://issues.apache.org/jira/browse/MODPYTHON-212 > > > > > > > > > > So, OP should ensure that directives to accept compressed request > > > > > content are disabled. > > > > > > > > > > Graham > > > > > > > > > > On 05/11/2007, Martijn Moeling <martijn at xs4us.nu> wrote: > > > > > > > > > > > > > > > > > > > > > > > > Hmm it looks to me that you have an upload limit in your apache > > > > > config. > > > > > > > > > > > > It might be the LimitRequestBody directive. > > > > > > > > > > > > Another thing might be that your processing is faster than the > > > data > > > > > comes > > > > > > in, but I doubt that > > > > > > Take a close look at you apache conf, and your apache error log > > > > > (right after > > > > > > an upload or with tail -f /var/log/httpd/error_log (or where > > ever > > > > > your > > > > > > logdir is)). > > > > > > > > > > > > Martijn > |