Valentino Volonghi's Weblog

A web server in twisted - Part 2

2005-12-05T10:58:00.000+01:00

Yesterday I wrote this post about how to write a very simple web server for a predefined content. Now I want to change it and discuss the solution a bit more in detail. First of all let's make it a real webserver, serving files from a known directory instead of serving some dummy content and avoid calling the browser which is not really needed.


from twisted.internet import reactor
from twisted.web2 import server, static, channel, http
PORT = 8080
DIRECTORY = '/Users/dialtone/'
s = server.Site(static.File(DIRECTORY))
reactor.listenTCP(PORT, channel.HTTPFactory(s))
reactor.run()

This is a better solution now. But how is it any good?

As this post states, the HTTPListener solution of the .Net world is not really useful since it only supports Windows Server 2003 or Windows XP with SP2, not even SP1 or Windows 2000. The author then goes on explaining what will be done to rewrite a new webserver that will solve this problem and how hard it is to handle all the request parsing for HTTP.

Mr. Andrea Boschin is very right in his statements, HTTP is not an easy protocol to implement given all the details that it comes with, especially considering that its RFC is sometimes contraddictory with itself. Fortunately the real advantage of Twisted Matrix is that all these details are already solved for you by the library author. In this case we are using the second version (which is currently under development but stable and used in many project, I use it myself in some of my projects) of the HTTP 1.1 protocol implementation.

The Original Poster starts dealing with the first problems with the integration between the existing application and the new to-be-written web server, the solution proposed is threading, since he is running this webserver as a Windows service he is in the need of running a parallel thread to deal with HTTP requests. Being completely async allows you to already forget about this. You can easily run many different services in the same process (like an SMTP server, an HTTP server and an IMAP server) without the need of threading while still keeping a good request rate in the order of 900 requests fullfilled each second on an Opteron 2.4Ghz.

The solution I came up with is already capable of adding a complete SMTP server to its features, in fact:


from twisted.internet import reactor
from twisted.web2 import server, static, channel, http
from twisted.mail import mail, maildir

PORT = 8080
DIRECTORY = '/Users/dialtone/'
MAIL_PATH= '/Users/dialtone/mail/'
SMTP_PORT = 25
POP3_PORT = 110

s = server.Site(static.File(DIRECTORY))
reactor.listenTCP(PORT, channel.HTTPFactory(s))

mailservice = mail.MailService()
domain = maildir.MaildirDirdbmDomain(mailservice, os.path.abspath(MAIL_PATH))
domain.addUser('foo', 'bar')
mailservice.addDomain('localhost', domain)

smtp = mailservice.getSMTPFactory()
pop3 = mailservice.getPOP3Factory()

reactor.listenTCP(SMTP_PORT, smtp)
reactor.listenTCP(POP3_PORT, pop3)

reactor.run()

This is that service with an HTTP server plus an SMTP server and a POP3 server. (Note: due to lack of time I haven't been able to completely test this piece of code, but it should serve as an idea about how to achieve the goal).

What is all this post about then, besides writing something about twisted and how easily you can integrate different protocols without worrying too much about concurrency (since Twisted itself handles it for you)? This post was just made to show how the .Net platform is actually limiting its users not only in the features' 'space' (it's hard to write even simple services and it's even harder to integrate them) but also from the legacy point of view. This should serve as a good lesson about the strengths of Open Source against Proprietary Software too.

A Web server in Twisted

2005-12-04T01:51:00.000+01:00

Starting from this post that was triggered by this one I decided to do the same thing using Twisted Matrix.

Again the basic task is to write a webserver (the original poster was surprised in finding out how 'easy' it is to write a web server using .Net 2).

This is my solution:


from twisted.internet import reactor
from twisted.web2 import server, resource, channel, http
import webbrowser
PORT = 8080

class Page(resource.Resource):
    addSlash = True
    def render(self, ctx):
        return http.Response(stream="""
            <html><body><h1>This is the HTML body</h1></body></html>
        """)

s = server.Site(Page())
reactor.listenTCP(PORT, channel.HTTPFactory(s))
webbrowser.open("http://%s:%d" % ("localhost", PORT))
reactor.run()

Easy isn't it?

Random image rotators everywhere

2005-12-01T01:00:00.000+01:00

After I came across this I decided that I should write something about the same thing but done in Nevow.
First of all the task is to write a random image rotation implementation for jpegs, gifs and pngs. This is easily accomplished by doing the following 3 things:

1. Add this to the root Page of your web site. 'random/images/directory' is a directory in your HDD containing lots of images and other random data.

child_random_images = static.File('random/images/directory')

2. Add this to each of your Page objects that use the random image (or use a common base class to have this behaviour everywhere for free):

 
def render_random_image(self, ctx, data):
    files = [f for f in os.listdir('random/images/directory') 
               if os.path.splitext(f)[1] in ['.jpg', '.gif', '.png']]
    return ctx.tag(src='/random_images/'+random.choice(files))

3. Use the following tag wherever you want to have a randomly rotated image in your web app:

 <img nevow:render="random_image" alt="Random Image" />

Now you have a wonderfully randomly rotated image in your web browser and with 2 lines less than the Ruby On Rails counterpart without sacrifying readability too much.

2005-04-23T11:50:00.000+02:00

Today I came across this and I want to answer to Ryan since he deserves more than one answer.

I strongly agree with Ryan but I notice he never tried using web2 (probably because it is not released yet) in Twisted Matrix.

... have a sane mechanism for attaching different behavior to different verbs for a single resource?

In this near-to-be-released web server each renderable resource has a renderHTTP method that is used to dispatch the request to the right renderer method after looking at the request method. For example inside the resource you can have http_GET or http_POST or http_DELETE methods and so on.

... help you use response status codes properly? (e.g. Nearly all dynamic content returns either a 200 or 500).

The result of each method call through renderHTTP must be a Response object which has the following signature:

def __init__(self, code=None, headers=None, stream=None):

I don't think I need to tell what code is, but web2 contains a module that has all the HTTP response codes with a human-readable name (one of the main problems with response codes is that they are codes, and humans can't read them).

headers is a simple dictionary of Headers:Values or directly an Headers object instance which will take care of header serialization and unserialization in an RFC compliant way.

... make dealing with transfer encodings (gzip, compress, etc.) easy?

What's the stream argument in the signature above? It is a an object that allows you to handle input and output to and from the server (James Knight, the author, would say that it's an handy implementation of the Consumer/Producer model). There are many streams already available in Web2 like MemoryStream, FileStream (when sending a file to the clients), gzipstream, deflatestream and you can write your own if you like. The stream argument is the content you need to send if the stream doesn't support the Consumer logic or, alternatively, you can pass content to the stream using the write() method if it supports Consumer logic.

... make dealing with media types easy?

MediaTypes are handled correctly with the MimeType object togheter with the corresponding serializer (which is handled transparently by web2, but you can provide your own).

... provide facilities for implementing a smart caching strategy for dynamic content? (proper use of If-Modified-Since, Expires, Cache-Control, etc.)

It's not completely clear to me what this means. Caching dynamic content is very hard and the framework should never step-in IMHO. That is a completely user-driven feature but web2 allows you to put your headers in the response object, hence I believe this satisfies this point.

What are my conclusions? Just give web2 a try. Come in #twisted.web on freenode to talk about future web2 and Nevow development (Nevow gave a lot of code to web2 resource handling mechanisms, but James Knight, web2 author, changed the original code enough to allow all the flexibility of the new model I just described in this post).

As a Nevow developer I must say that I'm looking forward to the day when we move from twisted.web to twisted.web2 to enable all this new stuff, which I also need to handle from my software.

To close everything: web2 already supports wsgi.

2004-11-03T11:44:00.000+01:00

Some people are interested in my Gnus look, which you can find here.

Since it took me a lot of time to get it working I think I'll write down the steps I made:

Installed Emacs-cvs with GTK2 support (it's pretty easy if you don't have any problem), here is the cvs address :ext:anoncvs@savannah.gnu.org:/cvsroot/emacs

Installed t-gnus (which is the multi-language Gnus version), FreeBSD5 has a port named t-gnus that will be installed under /usr/local/share/emacs/21.3/site-lisp, you should move it from 21.3 to 21.3.50 to use it under emacs-cvs. Here you can find some versions like t-gnus-6_12_0-02 which I'm using

Then you are ready to use my .emacs with some stuff for python and Gnus. You can download it here.

Hope This Helps

2004-10-29T14:38:00.000+02:00

I'm a PDF!

Which File Extension are You?

I'm a proprietary format. But I'm cross-platform anyway. I can get along with this ehe

2004-10-20T20:54:00.000+02:00

About "Seven Cool Mono Apps"

I just read the last column on O'Reilly network here by Edd Dumbhill:

Because decoding audio itself isn't a simple process, there aren't any all-Mono ways of doing this yet. Instead, Muine uses the "p-invoke" facility of Mono to hook into the C-based multimedia library libxine. Calling C code from Mono doesn't require any supplemental C coding, unlike most other languages such as Perl or Python.

I would say that he is quite completely wrong. He maybe doesn't know anything about ctypes module, which is a rather complete and useful way to call functions inside compiled dynamic libraries.

You can find ctypes here with a little tutorial here. And now a little example, just to convince you about what I'm saying:

>>> from ctypes import *

>>> print cdll.msvcrt.time(None)

1048777320

>>> print hex(windll.kernel32.GetModuleHandleA(None))

0x1d000000

See you to my next post

2004-06-28T13:02:00.000+02:00

Today, while surfing The Net, I bumped into this blog post from Marco Russo, a C# programmer.

Basically he is saying that generics are such a great feature because with them you can write stuff like this:

List integers = new List();
for(int i=1; i<=10; i++) integers.Add(i);
int sum;
integers.ForEach(delegate(int i){ sum+=i; });

Now, after reading that thing 3 times I realized that all it was doing was to sum the first 10 integers which can be done in this way using python:

sum([x for x in xrange(11)])

Maybe times will come, when people will realize that such poor languages, like C# or java, are nowhere near highly dynamic ones like Python. And machine power is not giving them any reason to be learned.

2004-05-24T00:52:00.000+02:00

Hi all, long time since my last posting.

As you probably know, I'm not a C++ expert, even if I can almost read C++ code without many problems.

One thing that I didn't know is that encapsulation is not that strong as it is belived to be, there are 2 techniques that enable you to access all attributes inside a class and I'm going to show you:

Technique 1:

class D {
public:
int x;
};

C c;
(reinterpret_cast<D*>(&c))->x = 1;

Technique 2:

#define private public
#include <C_definition.h>
#undef private

C c;
c.x = 1;

This is just to say that Python is not the only one that plays some tricks instead of having real private attributes/methods. But again who really needs private methods?

2004-04-07T21:44:00.000+02:00

Today has been a busy day.

My parents went to Creta, in the mediterranean sea, togheter with my brother. So I'm alone at home for Easter, good news for me and my girlfriend. But I had to bring them to the airport which is over an hour away from home.

Then I had to answer some mails about my new rss aggregator. Alex Martelli proposed me to add it to the ActiveState cookbook, and maybe they'll add it to the next edition of the Python Cookbook.

Then again Jan Alonzo, one of the Straw developers, mailed me about my rss-aggregator, telling that it was interesting and asking if I've already written a patch for straw. Unfortunately I haven't since Straw's code is not really clear to me, and I'll need a lot of help from them in order to integrate it correctly.

And worst of all... In 20 days my University exams will start, I'll need to study a bit starting from tomorrow.

So... This is going to be a tough month.

2004-04-06T20:34:00.000+02:00

Lately I've been working on an rss aggregator made with python and Twisted. It's growing really fast, since twisted is marvellous.

Now the protocol supports a lot of things like:

Arbitrary number of multiple simultaneus connections

Request timeout

Max age of each feed

Memoizing of the feeds with a timestamp

Multi-client modular design

DOING: web interface, now I've only written a primitive one

I must say it. Twisted is a really impressive framework.

That's not the only thing I've been doing in the last days, as many of you know, on April 1st the GNOME team released their DE version 2.6, and today I've been busy searching for a good theme. Fortunately in the gentoo forums I've read of a project for making an official theme.

Here is how it looks:

Full desktop

Xmms Skin

Old desktop

My emacs GNUS screenshot

Cya next time

2004-03-18T17:21:00.000+01:00

This is just a test posting, made to be sure that bloGTK is working to help me writing my blog.

I think I'll start for real to write some good stuff on my blog. There are a lot of things to talk about and I would like to share them with you all.

Cya next time :)

2003-07-23T22:34:00.000+02:00

Hi, all. This is my first post. I think this is the time for some presentations. I'm Valentino Volonghi, 20 years old and Italian. Proud linux user and fond of python. I think that anyone who works on python is a kind of a genius: Guido Von Rossum, Tim Peters and Alex Martelli (there are more and more but I have one thing to say about this last one). Today I decided to start this blog because of one thing: I need to say to anyone that I'll probably be working togheter with Alex Martelli... He is the best programmer I've ever known, we talked over an hour and a half and it was the best time spent in my life.

I hope to post some photograps in the future since I think I am very lucky :).

Obviously this is not my only passion, I also like mangas and anime a lot and I have a fairly big collection of mangas.
For this time it's all, I'm going to write again in the next days.

Oh, I forgot to say one thing: really thanks to a friend of mine: Lawrence Oluyede. We both work for the same company and we both met Martelli.
Hi all.