Publisher : iterable return values should be corretly published

• Key: MODPYTHON-15
• Type: Improvement
• Status: Reopened
• Priority: Minor
• Assignee: Nicolas Lehuen
• Reporter: Nicolas Lehuen
• Affects Version/s: 3.1.3

Description

Suppose this function in a published module :

def index(req)
    req.content_type = 'text/plain'
    yield '1\n'
    yield '2\n'
    yield '3\n'
    yield '4\n'

When published, this module should return a text content with '1\n2\n3\n4\n'.

This could also be useful with a file() object, since they are iterable ; this would provide another way to send a file, only slightly less performing than the send_file() method. Handy when you want to filter a file :

def filter(req,filename):
    f = open(filename,'r')
    for line in f:
        yield re.sub('foo','bar',line)

Comments

[Graham Dumpleton]

I would be worried here about cases where a generator/iterator of this form never actually had a stop condition. Imagine something which continually generates random numbers (if that makes sense). If mod_python were to just keep calling it to generate all data, it would eventually blow up. Seems to me to be safer for mod_python not to do anything with generators/iterators at all and make the user wrap them as appropriate to extract all the required data.

Note, I could be talking nonsense here as I have never consciously used generators so making some assumptions about them. :-)

[Graham Dumpleton]

The PEP on generators actually gives an example which if accessible would cause mod_python to blow up if mod_python kept calling it until all values were exhausted. Ie.,

   def fib():
            a, b = 0, 1
            while 1:
                yield b
                a, b = b, a+b

Thus, safer to make users wrap it in ordinary Python code and collect data themselves and then put it together as required.

[Nicolas Lehuen]

Nobody prevents anyone from putting an infinite loop in a standard publisher, either. This would cause mod_python to blow up :

dex index(req):
    while True:
        req.write('foobar\n')

So we don't need to worry about the possibility that some code written by a developer goes into an infinite loop. I really think that the advantages of iterator publication outweight this particular drawback.

[Nicolas Lehuen]

Fixed in subversion.

[Nicolas Lehuen]

Reopened to fix MODPYTHON-97.

It's a little bit more complicated than expected, because it breaks some RPC / JSON code (which rely on str() being called on tuples or lists)... Plus we should implement chunked encoding when returning content produced by iterators, since we can't set the Content-Length header upfront.