[mod_python] requests are blocking when using req.sendfile

I haven't read this latest response, but have one guess al the same. I
will say first that working out what the problem has been has been
made more difficult by you not offering up any source code for your
handler to show exactly what you are doing.

Anyway, one viable explanation for the problem is that you are using
mod_python session objects. Session objects are not normally unlocked
until cleanup handler, which is after all response content has been
returned. Thus if sending a lot of data back, requests from the same
user will be locked out, as well as other users who are unfortunate
enough to have the session keys fall into the same session mutex lock
bucket.

If you are using sessions, just before you return from handler, do
something like:

  req.session.save() # if necessary
  req.session.unlock()

I only say 'req.session' here as that is recommended convention for
where you store it, you may well have it stored elsewhere.

Graham

2008/9/28 Matt Barnicle <mattb at wageslavery.org>:
>> I ask again what I asked before and which you didn't answer.
>>
>> How many concurrent requests for these files are you receiving and
> how
>> long does it take to download a file?
>
> sorry for not answering that earlier.  there aren't all that many.
> we get one or two requests for various pages about every 30 seconds
> during normal traffic, sometimes that number goes up.  we only get
> requests for downloads occassionally.  a much smaller percentage of
> the requests are for the downloads.  they are various sizes.  the
> one that is getting the most requests right now is an 18 MB zip
> file.  it takes around 30 seconds to 2 minutes on a DSL connection
> to complete the download.  you can see for yourself here:
>
> http://giveback.net/download/music/3
>
> i kept the site hidden earlier, because i've posted some code from
> of our infrastructure and thought i should keep who i'm working for
> hidden for that reason, but i think it may help toward getting the
> problem solved and i'm not so concerned anymore..  apologies if that
> has hindered the process in any way.
>
>> In respect of your original comment
>>
>>> it works..  but the problem is, requests start backing up whenever a
>>> download is being sent.  it looks like new requests are being sent
>>> to the apache process that is tied up sending the download file.  i
>>> don't understand why this is happening though.  i would think that
>>> while it's sending the file, no new requests would be queued in that
>>> process.
>>
>> If you are truly using prefork MPM, then new requests cant be sent to
>> the busy process as requests are effectively accepted by a process
>> only when it is ready to handle a request.
>>
>> Please verify which MPM is being used by running:
>>
>>   httpd -V
>
> # httpd -V
> Server version: Apache/2.2.3
> Server built:   Jan 15 2008 20:33:41
> Server's Module Magic Number: 20051115:3
> Server loaded:  APR 1.2.7, APR-Util 1.2.7
> Compiled using: APR 1.2.7, APR-Util 1.2.7
> Architecture:   64-bit
> Server MPM:     Prefork
>  threaded:     no
>  forked:     yes (variable process count)
> Server compiled with....
>  -D APACHE_MPM_DIR="server/mpm/prefork"
>  -D APR_HAS_SENDFILE
>  -D APR_HAS_MMAP
>  -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
>  -D APR_USE_SYSVSEM_SERIALIZE
>  -D APR_USE_PTHREAD_SERIALIZE
>  -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
>  -D APR_HAS_OTHER_CHILD
>  -D AP_HAVE_RELIABLE_PIPED_LOGS
>  -D DYNAMIC_MODULE_LIMIT=128
>
>> How many httpd processes are running at the time you are making
>> requests that appear to be blocked?
>
> it varies.  could be less than 10, could be between 10 and 40.  i've
> seen it happening with any number of processes, it doesn't seem to
> affect things.
>
>> What happens if you make requests at same time from a different web
>> browser or from another box, are they also stalled?
>
> it depends.  if i download in one browser, then surf around in
> another one, i may very well be able to click around in the other
> the entire time without any problems.  but i may also get the
> blocking issue.  and sometimes i will load the site fresh for the
> first time on some days, and it won't load at all for a few minutes,
> then it will.  and in these times, i will check /server-status and
> see a lot of requests in the WAIT state, and always when it's like
> this, there are one ore more downloads happening.
>
>> Are you using any AJAX stuff on client side for handling the
>> download?
>>
>> Graham
>
> no.  there is a lot of AJAX on the site, but none of it related to
> downloads.