Bottle is my favorite python web framework, it is lightweight and fast. Those are good qualities, especially when you need to write web services.
However it misses some nice features of other “bigger” framework. One of them is a nice way to handle and validate query parameters.
To make an example, lets suppose that your route function expect the GET parameters a, b and c. In bottle you will have to write something like this:

from bottle import route, request
 
@route('/test/')
def test():
    a = int(request.GET.get("a"))
    b = float(request.GET.get("b"))
    c = request.GET.get("c").upper()
    # logic go here
    return "ok"

As you can see we have tree line of boilerplate code that will reduce the focus from the real logic of the function. The situation will be further complicated if we want to have some of this parameter required and others optional with a fallback default value and if we want to handle the errors in a proper way.

While one of my web service requires to get some parameters from the query string, I decide to write down this decorator:

"""
Copyright (c) 2011, Giuseppe Tribulato.
License: MIT (see http://www.opensource.org/licenses/mit-license.php for details)
"""
from bottle import route, request
import functools
import inspect
 
def checkParams(**types):
    def decorate(f):
        farg, _, _, def_params = inspect.getargspec(f)
        if def_params is None: def_params = []
        farg = farg[:len(farg) - len(def_params)]
 
        param_info = [(par, ptype, par in farg) for par, ptype in types.iteritems()]
 
        @functools.wraps(f)
        def wrapper(*args, **kargs):
            getparam = request.GET.get
            for par, ptype, required in param_info:
                value = getparam(par)
                if not value: # None or empty str 
                    if required:
                        error = "%s() requires the parameter %s" % (wrapper.__name__, par)
                        raise TypeError(error)
                    continue
                try:
                    kargs[par] = ptype(value)
                except:
                    error = "Cannot convert parameter %s to %s" % (par, ptype.__name__)
                    raise ValueError(error)
 
            return f(*args, **kargs)
 
        return wrapper
    return decorate

To see how it works, lets rewrite the previous example using checkParams:

import string
 
@route('/test/')
@checkParams(a = int, b = bool, c = string.upper)
def test(a, b, c):
    return dict(a = a, b = b, c = c)

Inside checkParams we are declaring that we want to handle the GET parameters a, b and c with the provided conversion functions and we expect such parameters to be passed to our decorated function test.

We can test it, pointing the browser to our server (in my case on localhost:8080) with the following url:

http://localhost:8080/test/?a=10&b=1&c=test

this will output

{"a": 10, "c": "TEST", "b": true}

The parameters were parsed, converted and passed to our test function as expected. I define all the three parameters of the function test as required. Lets take a look to what happens removing one of the parameters from the query string:

http://localhost:8080/test/?a=10&b=1

if you set bottle.debug(True) this will output:

Internal Server Error

TypeError('test() requires the parameter c',)

If I want to define c as optional, I can give to it a default value:

@route('/test/')
@checkParams(a = int, b = bool, c = string.upper)
def test(a, b, c = "default"):
    return dict(a = a, b = b, c = c)

This time we get a valid response:

{"a": 10, "c": "default", "b": true}

Finally lets try what happens if we pass an invalid type to a:

http://localhost:8080/test/?a=no&b=1

this will output:

Internal Server Error

ValueError('Cannot convert parameter a to int',)

That’s it! I hope that this snippet can be useful to someone.

In my previous post (six months ago) I claim about the bad performance of Python 3.0. The first py3k version contained a new I/O library written totally in python that was the main cause of such problems.

Now Python 3.1 present a totally new I/O Library rewritten in c for maximum performance. As promised, I re-execute all my previus tests with the latest python version (3.1.1).

The next graph show the results, look at my previous post for details on the tests.

Different python version compared for performance

Text Read/Write

First you can see that the incredible bad text read issue is now solved. Python 3.1 is now much much faster, but it remain 2.42x slower than python 2.5. This is probably due to the text decoding phase. Text writing is now comparable to python 2.5.

Binary Read/Write

The rewritten I/O module is now able to read binary data faster than before. The same is not true for the writing performance that remain slow as Python 3.0.

Print

Another problem that I noticed in python 3.0 was the very bad performance on the new print function compared to the old print statement. In this test, Python 3.1 is now “only” 5 times slower than Python 2.5.

In general a good work was done to optimize py3k. I think that for general applications, py3k is now perfectly usable.

Many thanks to the python development team!

Python 3.0 (aka Python 3000) was released about a week ago.

In the what’s new the python team claim:

The net result of the 3.0 generalizations is that Python 3.0 runs the pystone benchmark around 10% slower than Python 2.5. Most likely the biggest cause is the removal of special-casing for small integers. There’s room for improvement, but it will happen after 3.0 is released!

In this week however, many people notice big performance differences in the new version versus the old 2.5/2.6 series related to I/O.

Since the read and write performance are very important for my work (data analysis), I did some tests in my laptop (MacBook Pro 2.4Ghz).

I installed from source code Python 2.5.2 and Python 3.0. Similar script has been compared by running them 5 times and taking average times.

Binary Read Test

Loading of a big file (156Mb) into memory in one step.

#!python2.5
import time
 
f = open("bigfile.txt","rb")
start = time.time()
f.read()
stop = time.time()
f.close()
print "%.3f sec" % (stop-start)

#!python3.0
import time
 
f = open("bigfile.txt","rb")
start = time.time()
f.read()
stop = time.time()
f.close()
print("%.3f sec" % (stop-start))

Text Read Test

Iterating over a big file (156Mb) line by line.

import time
 
f = open("bigfile.txt","r")
start = time.time()
for l in f:
	pass
stop = time.time()
f.close()
print "%.3f sec" % (stop-start)

import time
 
f = open("bigfile.txt","rt")
start = time.time()
for l in f:
	pass
stop = time.time()
f.close()
print("%.3f sec" % (stop-start))

Binary Write Test

This test measure writing time for binary data 100Mb.

import time
 
l = "A" * 1024*1024
 
f = open("writefile.txt","wb")
start = time.time()
for i in xrange(100):
	f.write(l)
stop = time.time()
f.close()
print "%.3f sec" % (stop-start)

import time
 
l = "A" * 1024*1024
l=l.encode("iso-8859-1")
f = open("writefile.txt","wb")
start = time.time()
for i in range(100):
	f.write(l)
stop = time.time()
f.close()
print("%.3f sec" % (stop-start))

Text Write Test

Write performance for text files.

import time
 
l = "*" * 1024 + "\n"
f = open("writefile.txt","w")
start = time.time()
for i in xrange(100000):
	f.write(l)
stop = time.time()
f.close()
print "%.3f sec" % (stop-start)

import time
 
l = "*" * 1024 + "\n"
f = open("writefile.txt","wt")
start = time.time()
for i in range(100000):
	f.write(l)
stop = time.time()
f.close()
print("%.3f sec" % (stop-start))

Print

From Python 3.0 “print” became a function. Previously it was a statement.  What does it means? From a syntax point of view, are necessary small changes (for example add brackets around parameters). But the real bad news are on the execution time. In this test I compare the execution of the statement/function without parameters.

import timeit
import sys
cmd='for i in xrange(1000000): print '
t=timeit.Timer(cmd).timeit(1)
 
sys.stderr.write("%.3f sec\n" % t)

import timeit
import sys
cmd='for i in range(1000000): print()'
t=timeit.Timer(cmd).timeit(1)
 
sys.stderr.write("%.3f\n" % t)

In the next python release (3.0.1), is expected some speed improvement on many of the previous tests. I will redo my test when this new version will be available.

In the latest release of the iPhone SDK 2.2 (9m2621) there is an issue concerning the localization procedure.

When the input string file is utf-8 and “Strings file Output Encoding” is set to binary, the resulting file isn’t Localizable.string but Localizable.string.XXXX where XXXX is a pid number.

To correct this issue you can patch the ruby script copystrings located in /Developer/Library/Xcode/Plug-ins/CoreBuildTasks.xcplugin/Contents/Resources/copystrings.

Simply change line 134 from:

system('plutil -convert binary1 -s -o "' + OPTIONS[:OutputDir] + '/' + 
File.basename(path) + '" -- "' + path + '"')

to

system('plutil -convert binary1 -s -o "' + OPTIONS[:OutputDir] + '/' + 
File.basename(path.gsub(".#{Process.pid}","")) + '" -- "' + path + '"')