I wanted to do HTTP Pipelining using urllib2. But, first of all, what is pipelining ? HTTP pipelining is a technique in which multiple HTTP requests are written out to a single socket without waiting for the corresponding responses.

What is the benefit of pipelining ? Less network load, speedup processing ! I was searching a way to do it with urllib2... But solution are complicated, and not fit well to my needs. But way, why stay on urllib2 ? Use httplib !

Reusing the connection :

First, reusing the same connection

import httplib
server = httplib.HTTPConnection('yourserver.com')
server.request('GET', '/index.html')
print 'RESPONSE1:', server.getresponse().read()

server.request('GET', '/index2.html')
print 'RESPONSE2:', server.getresponse().read()

server.request('GET', '/index3.html')
print 'RESPONSE3:', server.getresponse().read()

Second, try pipelining !

import httplib
server = httplib.HTTPConnection('yourserver.com')
server.request('GET', '/index.html')
res1 = server.getresponse()
server.request('GET', '/index2.html')
res2 = server.getresponse()
server.request('GET', '/index3.html')
res3 = server.getresponse()

print 'RESPONSE1:', res1.read()
print 'RESPONSE2:', res2.read()
print 'RESPONSE3:', res3.read()

After a long time, c++, not c++, benchmarks, testing, bug-fixes, documentation fixes, website, releases notes and wiki... PyMT 0.4 is released !

Thanks to all of you, that's was fun :)

For PyMT, Sharath need BGR support into PyMT, while his graphic card don't support GL_BGR. Well, after adapting swap code from Pyglet sourcecode, he say: "< patali> it works but very slow"

The goal is to swap a string from 'bgrbgrbgr' to 'rgbrgbrgb'. Just let's do a rapid benchmark.

import sys
import re
import time

swap1_pattern = re.compile('(.)(.)(.)', re.DOTALL)
swap1_repl = r'\3\2\1'
def swap1(bytes):
    return swap1_pattern.sub(swap1_repl, bytes)

def swap2(bytes):
    out = ''
    for i in xrange(0, len(bytes), 3):
        b, g, r = bytes[i:i+3]
        out += r
        out += g
        out += b
    return out

def swap3(bytes):
    bytes = list(bytes)
    for i in xrange(0, len(bytes), 3):
        b, g, r = bytes[i:i+3]
        bytes[i:i+3] = r, g, b
    return ''.join(bytes)

def swap4(bytes):
    blues = bytes[0::3]
    greens = bytes[1::3]
    reds = bytes[2::3]
    return ''.join(''.join(x) for x in zip(reds, greens, blues))

def swap5(bytes):
    from array import array
    a = array('b', bytes)
    a[0::3], a[2::3] = a[2::3], a[0::3]
    return a.tostring()

def swap6(bytes):
    import numpy
    a = numpy.array(bytes, 'c')
    b = a[...,::-1]
    return b.tostring()

def swap7(bytes):
    a = list(bytes)
    a[0::3], a[2::3] = a[2::3], a[0::3]
    return ''.join(a)


def bench(func, bytes):
    sys.stderr.write('Bench %s: ' % str(func))

    start = time.time()
    for i in xrange(20):
        sys.stderr.write('.')
        ret = func(bytes)
        if ret[:3] != 'rgb' or ret[-3:] != 'rgb':
            sys.stderr.write('INVALID DATA, start=%s, end=%s' %
                            (ret[:3], ret[-3:]))
            break

    end = time.time() - start
    sys.stderr.write('| Finished in %.4f\n' % end)
    return ret


bytes = 'bgr' * 256 * 256
bench(swap1, bytes)
bench(swap2, bytes)
bench(swap3, bytes)
bench(swap4, bytes)
bench(swap5, bytes)
bench(swap6, bytes)
bench(swap7, bytes)

Swap1() is actually the way of pyglet. swap2/swap3 is from me, and others is found on some threads in pygame mailing list.

So, who win ?

21:09 tito@ashaka ~ $ python benchmark.py 
Bench <function swap1 at 0x7f1d154632a8>: ....................| Finished in 3.2713
Bench <function swap2 at 0x7f1d15463230>: ....................| Finished in 1.2860
Bench <function swap3 at 0x7f1d15470cf8>: ....................| Finished in 0.6535
Bench <function swap4 at 0x7f1d15470e60>: ....................| Finished in 0.6475
Bench <function swap5 at 0x7f1d15470ed8>: ....................| Finished in 0.0367
Bench <function swap6 at 0x7f1d15470f50>: ....................| Finished in 0.1959
Bench <function swap7 at 0x7f1d15473050>: ....................| Finished in 0.1482

The array solution is a lot faster than any other implementation ! And it's python standard :) Let's take this...

I've been some week without blogging, but lot of things happened ! Finally, i've been able to see Thomas in IRL at Google Mentor Summit in San Francisco. It was an awesome week-end, funny, creative and geeky ! Lot of ideas, and goal : making PyMT more stable and well designed : the roadmap of PyMT 0.4 is finally available. We've also done a little session too at summit.

Thomas is now 3 days to ITS 2009 in Banff Canada, for a conference about PyMT. The paper will be available for everyone soon :) And this week-end, i'm going to Edinburgh with Lucie to meet Mark, and see the awesome new multitouch room of Napier University. Sharath is also going next week to a FOSS in India to spread PyMT too :)

Theses days are really cool :)