|
Graham Dumpleton
graham.dumpleton at gmail.com
Tue Nov 6 06:52:32 EST 2007
On 06/11/2007, Alec Matusis <matusis at yahoo.com> wrote: > I found that this that > > IOError: image file is truncated (14432 bytes not processed) > > has to do with using input filter > > PythonInputFilter flashfilter FLASHFILTER > SetInputFilter FLASHFILTER > > It somehow behaves differently in apache 2.2.6/mod_python 3.3.1 versus > 2.0.54/ 3.1.4. For now, I removed this input filter (we can get away without > it), and it seems to work. > > However, we have a bigger problem: we now have intermittent import errors > (one error per about 50-80 requests). > > This code executed by publisher: > > import modules > def _import( module_path ): > runme = modules[ module_path ] > #get the last part > class_name = module_path.split('.')[-1] > try: > runme = getattr(runme, class_name ) <--- INTERMITTENT ERROR here > except AttributeError: > raise Exception,'Module does not contain class '+class_name+', > runme='+repr(runme)+' module_path='+repr(module_path) > > produces this error: > > Exception: Module does not contain class getprofile, runme=<module > 'getprofile' from '/path/scripts/getprofile.py'> module_path='getprofile' > > This used to always work with prefork. Is there anything not thread safe > about importing modules? The underlying Python and mod_python import mechanisms themselves are never usually the problem. It is thread unsafe practices regarding initialisation in the code around the imports. So, if you have built your own form of module importer or higher level layer for one and you haven't properly thread protected it, it is most likely the cause. So, what is 'modules' and '_import' and how are they used? How do you ensure that two concurrently executing request handlers trying to import the same module don't interfere with each other? Graham > > -----Original Message----- > > From: mod_python-bounces at modpython.org [mailto:mod_python- > > bounces at modpython.org] On Behalf Of Graham Dumpleton > > Sent: Monday, November 05, 2007 1:29 PM > > To: Martijn Moeling > > Cc: mod_python at modpython.org; Alec Matusis > > Subject: Re: [mod_python] mod_python or apache scalability? > > > > The LimitRequestBody directive support in mod_python is broken and > > most likely would have resulted in a 500 error occurring during > > mod_python.publisher processing of the form even before the user code > > was run. > > > > My guess at the reason is that they have enabled in the new Apache > > server compression on request content. This may not work with > > mod_python because how it reads request content is broken, with it > > only reading up until original Content-Length in certain cases. If > > there is a mutating input filter such as content decompression, you > > will get truncated input data. > > > > https://issues.apache.org/jira/browse/MODPYTHON-240 > > https://issues.apache.org/jira/browse/MODPYTHON-212 > > > > So, OP should ensure that directives to accept compressed request > > content are disabled. > > > > Graham > > > > On 05/11/2007, Martijn Moeling <martijn at xs4us.nu> wrote: > > > > > > > > > > > > Hmm it looks to me that you have an upload limit in your apache > > config. > > > > > > It might be the LimitRequestBody directive. > > > > > > Another thing might be that your processing is faster than the data > > comes > > > in, but I doubt that > > > Take a close look at you apache conf, and your apache error log > > (right after > > > an upload or with tail -f /var/log/httpd/error_log (or where ever > > your > > > logdir is)). > > > > > > Martijn > > > > > > > > > ________________________________ > > > Van: Alec Matusis [mailto:matusis at matusis.com] > > > Verzonden: zo 04.11.2007 21:46 > > > Aan: Martijn Moeling > > > CC: mod_python at modpython.org > > > Onderwerp: RE: [mod_python] mod_python or apache scalability? > > > > > > > > > > > > > > > > I had similair problems however, they turned out to be MySQLdb > > > > related and not in a way you suspect > > > > > > Thanks! I will try separating MySQLdb connections. We do pass the > > connection > > > around currently. > > > > > > However, how would you explain this PIL error: > > > > > > File "/path/publisher/publisher.py", line 78, in handler > > > flow_instance.dispatch() > > > File "/path/scripts/updateprofile.py", line 92, in > > > dispatch > > > im.save(full, self.im_format) > > > File > > > "/usr/local/encap/Python-2.4.1/lib/python2.4/site- > > packages/PIL/Image.py", > > > line 1272, in save > > > > > > File > > > "/usr/local/encap/Python-2.4.1/lib/python2.4/site- > > packages/PIL/ImageFile.py" > > > , line 192, in load > > > IOError: image file is truncated (14432 bytes not processed) > > > > > > This only happened for large images, small ones uploaded fine. > > > The code is > > > > > > def dispatch(self): > > > pix = self.form['Filedata'] > > > try: > > > im = Image.open(pix.file) > > > except IOError: > > > self.req.status = apache.HTTP_NOT_ACCEPTABLE > > > im.save(full, self.im_format) > > > > > > The error is in the last line. It did not occur with prefork. > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > From: mod_python-bounces at modpython.org > > > [mailto:mod_python- > > > > bounces at modpython.org] On Behalf Of Martijn Moeling > > > > Sent: Sunday, November 04, 2007 5:16 AM > > > > To: Graham Dumpleton > > > > Cc: mod_python at modpython.org; Alec Matusis > > > > > > > Subject: Re: [mod_python] mod_python or apache scalability? > > > > > > > > hi, > > > > > > > > I use Pil without any problems, a I do with MySQLdb. > > > > > > > > I had similair problems however, they turned out to be MySQLdb > > > > related and not in a way you suspect > > > > > > > > Never ever use a global database connection variable. Create the > > > > database connection within your handler, > > > > end register a cleanup for closing it. I fixed a lot of trouble by > > > > altering my system by making all "global" stuff a method or > > property > > > > of the req object. > > > > > > > > The cleanup procedure cleans all objects which do anything to 3rd > > > > party external (net connected) stuff (like imap, MySQL) > > > > > > > > > > > > think of something like this: > > > > > > > > > > > > def handler_cleanup(req): > > > > req.db.cursor.close() > > > > req.db.close() > > > > > > > > Def handler (req): > |