[mod_python] Caching issues

2009/1/13 Tim Valenta <tonightslastsong at gmail.com>:
> Thank you for the links-- they prove useful.
>
> Capitalization shouldn't be an issue here... When I began learning
> Unix a good while back, I quickly cast off relying on the OS to
> 'figure it out'.  I've been keeping everything case sensitive.
>
> Backing up to something foundational: I seem to be able to use the
> import_module() method without problems now, but practically any use
> of standard python 'import'/'from' keywords fails.  The only success I
> get is when the desired module is in the same directory as the one
> trying to use the 'import' keyword.  Does this stem from the fact that
> import_module isn't look for packages, but rather specific files?  Is
> it for that reason that I can't use
>    from subDir import C
> or, the equally broken,
>    import subDir.C

Packages not supported with mod_python module importer, so these fall
through import_module and uses standard Python import mechanism.

> Jiggering around with adding more entries in importer.path to my
> subdirectories doesn't seem to have any effect.

You should be able to add multiple directories and so long as a single
Python module code file, should be found even when using 'import'.

Graham

> If the 'import'
> keyword really is bound to it's working directory only, I will likely
> in fact have to use import_module() is several places, rather than
> just at the head of the request (./A.py)
>
> Despite my general lack of success, I appreciate the comments you've
> made along the way.
>
> Tim
>
> On Mon, Jan 12, 2009 at 10:40 PM, Graham Dumpleton
> <graham.dumpleton at gmail.com> wrote:
>> 2009/1/13 Tim Valenta <tonightslastsong at gmail.com>:
>>>> You just need to ensure you set mod_python's own module search path,
>>>> ie. mod_python.importer.path
>>>
>>> And yet, as I've now removed my www directory from my PythonPath, I
>>> actually get limited functionality out of the mod_python imports.
>>> I've got my apache conf file globally applying the option:
>>>
>>>        PythonOption mod_python.importer.path "['~']"
>>> Also have tried using an absolute path, which is producing the same
>>> functionality:
>>>        PythonOption mod_python.importer.path "['V:/www/']"
>>>
>>> My main module ('A') is requested by Apache in my browser, which uses
>>> import_module to find and import file B.  File B is in the same
>>> directory as A, and they only way it was found was through my apache
>>> conf noted above.  This part works just fine, but two cases develop
>>> from here:
>>>
>>> 1) If I use module B to import a third module 'C', I can't seem to
>>> make it find module C if it's in a subdirectory.  For instance, doing
>>> either 'import subDirectory.moduleC' or doing
>>> 'apache.import_module("subDirectory/moduleC")' results in a failure to
>>> find the module.  Note that in this case of subdirectories, I'm
>>> intentionally avoiding absolute paths to reach my modules.  I only
>>> want to get absolute in my Apache conf.  On the flip side, if module C
>>> is in the same local directory as A and B, then mod_python finds
>>> module C just fine.
>>
>> Try:
>>
>>  apache.import_module("./subDirectory/moduleC.py")
>>
>>> 2 ) If I place C in the same directory as A and B, so that mod_python
>>> finds it, I tried using the natural Python 'import' keyword in B to
>>> import C.  According to what I understand, this should be fine for
>>> auto-reloading, since A imports B via import_module, and so then B can
>>> use either importing strategy to get C in the picture because
>>> mod_python uses import_module behind the scenes anyway.  In this
>>> setup, my caching issue remains, and changes to C are definitely not
>>> reflected in apache until a server restart.  Again note that it
>>> doesn't matter if I make B use the 'import' keyword or the
>>> apache.import_module() method.  Both result in failure to reload.
>>>
>>> I can't seem to diagnose what's even wrong, since the only reason
>>> module A, B, or C is being found in the first place is because of my
>>> importer.path variable is set in httpd.conf.  I've been using
>>> practically empty files to test this.  Each file contains no more than
>>> 5 lines to get the modules imported and then to write out a
>>> confirmation to the screen that it worked.
>>>
>>> Wisdom to offer?
>>
>> Yes, from memory the module importer isn't properly case insensitive
>> on Windows or MacOS X where file systems is case insensitive. Thus,
>> make sure your imports match exactly the case of the directories/files
>> in the file system.
>>
>> Also, use:
>>
>>  from mod_python import apache
>>  apache.log_error('__name__ = %s' % repr(__name__))
>>  apache.log_error('__file__ = %s' % repr(__file__))
>>
>> in modules so that information about imported files is logged as
>> imported. This will tell you which files are being imported.
>>
>> With PythonDebug On you should also see messages in Apache error logs
>> describing when they are being imported, initially and when changed.
>>
>> You can also write some code which extracts out of the mod_python
>> importer cache the details of everything that has been loaded and the
>> relationships. The function to call is:
>>
>>  from mod_python import apache
>>  data = apache.request_modules_graph()
>>
>> write that to a file and run:
>>
>>  http://www.graphviz.org/
>>  http://www.graphviz.org/Download_windows.php
>>
>> on it.
>>
>> Graham
>>
>>>> Now you do realise you don't need to convert 'import' to
>>>> import_module() everywhere?
>>>>
>>>> The 'import' directive when used inside a file which was already
>>>> managed by mod_python calls import_module() internally anyway.
>>>>
>>>> You just need to ensure you set mod_python's own module search path,
>>>> ie. mod_python.importer.path
>>>>
>>>> Graham
>>>>
>>>> 2009/1/12 Tim Valenta <tonightslastsong at gmail.com>:
>>>> > Okay, I think I'm on the same page as you now-- I've been playing
>>>> > around with it and it seems that when I leave my PythonPath to find my
>>>> > import modules, I definitely get no auto reloading goodness, while
>>>> > when I specify the root path to find them as files, rather than as
>>>> > modules (as explained in that documentation you linked to), things
>>>> > seem to work as expected.
>>>> >
>>>> > I wasn't getting any warnings in my apache log, and after putting
>>>> > together the details, I shouldn't be expecting any of that behavior
>>>> > with two of the same module in memory.  My issue seems to be simply in
>>>> > avoiding letting my PythonPath find my code.
>>>> >
>>>> > Thank you much,
>>>> > Tim
>>>> >
>>>> > On Sun, Jan 11, 2009 at 9:39 PM, Graham Dumpleton
>>>> > <graham.dumpleton at gmail.com> wrote:
>>>> >> 2009/1/12 Tim Valenta <tonightslastsong at gmail.com>:
>>>> >>> Okay, that makes sense.  My PythonPath system var definitely includes
>>>> >>> my www directory.
>>>> >>
>>>> >> That in itself may not be enough to cause problem. You should actually
>>>> >> see a warning in Apache error log complaining you have done this. That
>>>> >> is, overlapped sys.path with directories that mod_python module
>>>> >> importer uses.
>>>> >>
>>>> >> The real problem with doing this overlapping is that a module which
>>>> >> wasn't one managed by mod_python, ie., standard Python module, imports
>>>> >> something from the www directory, which also happens to be a handler
>>>> >> module, there will actually be too copies in memory and so can give
>>>> >> unexpected results.
>>>> >>
>>>> >> Graham
>>>> >>
>>>> >>> I had been using a forced path in my apache conf
>>>> >>> file to point to '~', for simple reference for the time being.
>>>> >>>
>>>> >>> I think my problem is simply that the modules I'm trying to import
>>>> >>> aren't 'candidates' for reloading.  Other than that, all other modules
>>>> >>> should already have been chained together with these
>>>> >>> apache.load_module calls.  My test case was pretty simple-- just two
>>>> >>> scripts, the first loading the second.
>>>> >>>
>>>> >>> Tim
>>>> >>>
>>>> >>> On Sun, Jan 11, 2009 at 8:25 PM, Graham Dumpleton
>>>> >>> <graham.dumpleton at gmail.com> wrote:
>>>> >>>> 2009/1/12 Tim Valenta <tonightslastsong at gmail.com>:
>>>> >>>>> Apologies for more trouble on the caching issue-- I've been adapting
>>>> >>>>> some of my scripts over to this apache.load_module function, but I
>>>> >>>>> don't think I'm seeing resolution to the issue.  If I've read the
>>>> >>>>> documentation properly, it seems that the PythonOption for
>>>> >>>>> auto-reloading is set to On by default, so to be clear, I haven't put
>>>> >>>>> a line for that in my httpd.conf file.  Are there any other tricks, or
>>>> >>>>> have I simply misread the documentation?  I still have to restart the
>>>> >>>>> Apache service in order to push the changes.
>>>> >>>>
>>>> >>>> Reloading does not work for any Python modules/packages installed in
>>>> >>>> standard Python module search path. Thus, if you have set PythonPath
>>>> >>>> to some directory where you store your modules, or if they are in
>>>> >>>> system site-packages directory, they are not candidates for reloading.
>>>> >>>>
>>>> >>>> Where are the modules you are modifying and expecting to be reloading
>>>> >>>> located? Is that location on sys.path, or have you explicitly setup
>>>> >>>> mod_python module importer path to tell it where they are?
>>>> >>>>
>>>> >>>> From within running application, print out __name__ from the module
>>>> >>>> you have loaded and are expecting to be reloadable, What is it? Does
>>>> >>>> it look like a normal module name or something magic?
>>>> >>>>
>>>> >>>> How was that module imported? There needs to be a chain from
>>>> >>>> mod_python handler for imports right down through to all modules you
>>>> >>>> want reloadable. If there aren't, for example, you are using
>>>> >>>> import_module() from a normal Python module which isn't a candidate
>>>> >>>> for reloading, you will not see the depth checking for reloading
>>>> >>>> occurring as you might expect.
>>>> >>>>
>>>> >>>> Graham
>>>> >>>>
>>>> >>>>> Tim
>>>> >>>>>
>>>> >>>>> On Sun, Jan 11, 2009 at 2:32 PM, Tim Valenta <tonightslastsong at gmail.com> wrote:
>>>> >>>>>> Despite the jab at Windows, the site will be deployed on a Unix
>>>> >>>>>> machine.  I just prefer Windows for ease of use, especially since my
>>>> >>>>>> wife has zero experience on a unix terminal or in the dearth of truly
>>>> >>>>>> professional software found on the Unix/Linux platform.  So yes, in
>>>> >>>>>> the end it will be on a "real" operating system,  But for now I'm
>>>> >>>>>> willingly sticking to a more user-friendly environment for the simple
>>>> >>>>>> folk.
>>>> >>>>>>
>>>> >>>>>> Given the characteristics of PHP that allow it to circumvent this
>>>> >>>>>> caching stuff, I still think a nice little friendly red-flag reminder
>>>> >>>>>> on the main mod_python tutorial should point out this limitation of an
>>>> >>>>>> Apache module like this, for those of us like me, who would have gone
>>>> >>>>>> many months more before having every accidentally found this manual
>>>> >>>>>> import method.
>>>> >>>>>>
>>>> >>>>>> On Sun, Jan 11, 2009 at 2:23 PM, Graham Dumpleton
>>>> >>>>>> <graham.dumpleton at gmail.com> wrote:
>>>> >>>>>>>
>>>> >>>>>>> 2009/1/12 Tim Valenta <tonightslastsong at gmail.com>:
>>>> >>>>>>> >> Still not short enough, so much so I stopped part way through.
>>>> >>>>>>> >
>>>> >>>>>>> > And for that I apologize.  I do appreciate the straight answer given,
>>>> >>>>>>> > though I would appreciate much more the addition of that fact in some
>>>> >>>>>>> > point-blank documentation that was easy to find.  I have scoured the
>>>> >>>>>>> > mod_python archives by search engine and came out utterly empty
>>>> >>>>>>> > handed.  I knew what the problem was, but I could find nothing to
>>>> >>>>>>> > remedy the issue.  I haven't encountered anything quite like this in
>>>> >>>>>>> > my adventures with PHP, perl, or other similar scripting languages
>>>> >>>>>>> > piloted by Apache.  It raises a valid concern to which I ideally
>>>> >>>>>>> > should have found answers very quickly.
>>>> >>>>>>> >
>>>> >>>>>>> > I hope this cures the subsequent issue given in the latter half of my message.
>>>> >>>>>>>
>>>> >>>>>>> PHP is specifically designed for web applications and it purposely
>>>> >>>>>>> throws away all code at the end of each request and thus is reloading
>>>> >>>>>>> it all on every request. Have a read of:
>>>> >>>>>>>
>>>> >>>>>>>  http://blog.ianbicking.org/2008/01/12/what-php-deployment-gets-right/
>>>> >>>>>>>
>>>> >>>>>>> I would be very surprised if mod_perl auto reloaded code as it is
>>>> >>>>>>> similar to Python in that you are applying a non web language to the
>>>> >>>>>>> web. Only way perl code would be reload is if you were running them as
>>>> >>>>>>> CGI scripts. If you run Python as CGI scripts you will get reloading
>>>> >>>>>>> as well, but as with anything CGI, much slower.
>>>> >>>>>>>
>>>> >>>>>>> BTW, maybe read:
>>>> >>>>>>>
>>>> >>>>>>>  http://blog.dscpl.com.au/2008/12/using-modwsgi-when-developing-django.html
>>>> >>>>>>>
>>>> >>>>>>> For WSGI hosted Python web application, this provides auto reloading
>>>> >>>>>>> on code changes. You do need to use a real operating system though,
>>>> >>>>>>> and not Windows.
>>>> >>>>>>>
>>>> >>>>>>> Graham
>>>> >>>>>>>
>>>> >>>>>>> > Tim
>>>> >>>>>>> >
>>>> >>>>>>> > On Sat, Jan 10, 2009 at 10:23 PM, Graham Dumpleton
>>>> >>>>>>> > <graham.dumpleton at gmail.com> wrote:
>>>> >>>>>>> >>
>>>> >>>>>>> >> 2009/1/11 Tim Valenta <tonightslastsong at gmail.com>:
>>>> >>>>>>> >> > Hello all-- I've been experiencing a caching issue from the
>>>> >>>>>>> >> > very beginning of my use of mod_python...  It's been at least 2 months now,
>>>> >>>>>>> >> > and I keep running into actual issues that prevent me from coding.
>>>> >>>>>>> >> > I have a knack for over-explaining, so I'll try to keep this concise yet
>>>> >>>>>>> >> > descriptive.
>>>> >>>>>>> >>
>>>> >>>>>>> >> Still not short enough, so much so I stopped part way through.
>>>> >>>>>>> >>
>>>> >>>>>>> >> The simple matter of it is that mod_python does not do deep checking
>>>> >>>>>>> >> of code for changes, nor does it automatically restart the process
>>>> >>>>>>> >> when code is changed. Thus the need to restart Apache when you make
>>>>