A web server in twisted - Part 2

Yesterday I wrote this post about how to write a very simple web server for a predefined content. Now I want to change it and discuss the solution a bit more in detail. First of all let's make it a real webserver, serving files from a known directory instead of serving some dummy content and avoid calling the browser which is not really needed.

from twisted.internet import reactor
from twisted.web2 import server, static, channel, http
PORT = 8080
DIRECTORY = '/Users/dialtone/'
s = server.Site(static.File(DIRECTORY))
reactor.listenTCP(PORT, channel.HTTPFactory(s))
reactor.run()

This is a better solution now. But how is it any good?

As this post states, the HTTPListener solution of the .Net world is not really useful since it only supports Windows Server 2003 or Windows XP with SP2, not even SP1 or Windows 2000. The author then goes on explaining what will be done to rewrite a new webserver that will solve this problem and how hard it is to handle all the request parsing for HTTP.

Mr. Andrea Boschin is very right in his statements, HTTP is not an easy protocol to implement given all the details that it comes with, especially considering that its RFC is sometimes contraddictory with itself. Fortunately the real advantage of Twisted Matrix is that all these details are already solved for you by the library author. In this case we are using the second version (which is currently under development but stable and used in many project, I use it myself in some of my projects) of the HTTP 1.1 protocol implementation.

The Original Poster starts dealing with the first problems with the integration between the existing application and the new to-be-written web server, the solution proposed is threading, since he is running this webserver as a Windows service he is in the need of running a parallel thread to deal with HTTP requests. Being completely async allows you to already forget about this. You can easily run many different services in the same process (like an SMTP server, an HTTP server and an IMAP server) without the need of threading while still keeping a good request rate in the order of 900 requests fullfilled each second on an Opteron 2.4Ghz.

The solution I came up with is already capable of adding a complete SMTP server to its features, in fact:

from twisted.internet import reactor
from twisted.web2 import server, static, channel, http
from twisted.mail import mail, maildir

PORT = 8080
DIRECTORY = '/Users/dialtone/'
MAIL_PATH= '/Users/dialtone/mail/'
SMTP_PORT = 25
POP3_PORT = 110

s = server.Site(static.File(DIRECTORY))
reactor.listenTCP(PORT, channel.HTTPFactory(s))

mailservice = mail.MailService()
domain = maildir.MaildirDirdbmDomain(mailservice, os.path.abspath(MAIL_PATH))
domain.addUser('foo', 'bar')
mailservice.addDomain('localhost', domain)

smtp = mailservice.getSMTPFactory()
pop3 = mailservice.getPOP3Factory()

reactor.listenTCP(SMTP_PORT, smtp)
reactor.listenTCP(POP3_PORT, pop3)

reactor.run()

This is that service with an HTTP server plus an SMTP server and a POP3 server. (Note: due to lack of time I haven't been able to completely test this piece of code, but it should serve as an idea about how to achieve the goal).

What is all this post about then, besides writing something about twisted and how easily you can integrate different protocols without worrying too much about concurrency (since Twisted itself handles it for you)? This post was just made to show how the .Net platform is actually limiting its users not only in the features' 'space' (it's hard to write even simple services and it's even harder to integrate them) but also from the legacy point of view. This should serve as a good lesson about the strengths of Open Source against Proprietary Software too.

A Web server in Twisted

Starting from this post that was triggered by this one I decided to do the same thing using Twisted Matrix.

Again the basic task is to write a webserver (the original poster was surprised in finding out how 'easy' it is to write a web server using .Net 2).

This is my solution:

from twisted.internet import reactor
from twisted.web2 import server, resource, channel, http
import webbrowser
PORT = 8080

class Page(resource.Resource):
    addSlash = True
    def render(self, ctx):
        return http.Response(stream="""
            <html><body><h1>This is the HTML body</h1></body></html>
        """)

s = server.Site(Page())
reactor.listenTCP(PORT, channel.HTTPFactory(s))
webbrowser.open("http://%s:%d" % ("localhost", PORT))
reactor.run()

Easy isn't it?

Random image rotators everywhere

After I came across this I decided that I should write something about the same thing but done in Nevow.
First of all the task is to write a random image rotation implementation for jpegs, gifs and pngs. This is easily accomplished by doing the following 3 things:

1. Add this to the root Page of your web site. 'random/images/directory' is a directory in your HDD containing lots of images and other random data.

child_random_images = static.File('random/images/directory')

2. Add this to each of your Page objects that use the random image (or use a common base class to have this behaviour everywhere for free):

 
def render_random_image(self, ctx, data):
    files = [f for f in os.listdir('random/images/directory') 
               if os.path.splitext(f)[1] in ['.jpg', '.gif', '.png']]
    return ctx.tag(src='/random_images/'+random.choice(files))
    

3. Use the following tag wherever you want to have a randomly rotated image in your web app:

 <img nevow:render="random_image" alt="Random Image" /> 

Now you have a wonderfully randomly rotated image in your web browser and with 2 lines less than the Ruby On Rails counterpart without sacrifying readability too much.

Today I came across this and I want to answer to Ryan since he deserves more than one answer.

I strongly agree with Ryan but I notice he never tried using web2 (probably because it is not released yet) in Twisted Matrix.

... have a sane mechanism for attaching different behavior to different verbs for a single resource?

In this near-to-be-released web server each renderable resource has a renderHTTP method that is used to dispatch the request to the right renderer method after looking at the request method. For example inside the resource you can have http_GET or http_POST or http_DELETE methods and so on.

... help you use response status codes properly? (e.g. Nearly all dynamic content returns either a 200 or 500).

The result of each method call through renderHTTP must be a Response object which has the following signature:

def __init__(self, code=None, headers=None, stream=None):

I don't think I need to tell what code is, but web2 contains a module that has all the HTTP response codes with a human-readable name (one of the main problems with response codes is that they are codes, and humans can't read them).

headers is a simple dictionary of Headers:Values or directly an Headers object instance which will take care of header serialization and unserialization in an RFC compliant way.

... make dealing with transfer encodings (gzip, compress, etc.) easy?

What's the stream argument in the signature above? It is a an object that allows you to handle input and output to and from the server (James Knight, the author, would say that it's an handy implementation of the Consumer/Producer model). There are many streams already available in Web2 like MemoryStream, FileStream (when sending a file to the clients), gzipstream, deflatestream and you can write your own if you like. The stream argument is the content you need to send if the stream doesn't support the Consumer logic or, alternatively, you can pass content to the stream using the write() method if it supports Consumer logic.

... make dealing with media types easy?

MediaTypes are handled correctly with the MimeType object togheter with the corresponding serializer (which is handled transparently by web2, but you can provide your own).

... provide facilities for implementing a smart caching strategy for dynamic content? (proper use of If-Modified-Since, Expires, Cache-Control, etc.)

It's not completely clear to me what this means. Caching dynamic content is very hard and the framework should never step-in IMHO. That is a completely user-driven feature but web2 allows you to put your headers in the response object, hence I believe this satisfies this point.

What are my conclusions? Just give web2 a try. Come in #twisted.web on freenode to talk about future web2 and Nevow development (Nevow gave a lot of code to web2 resource handling mechanisms, but James Knight, web2 author, changed the original code enough to allow all the flexibility of the new model I just described in this post).

As a Nevow developer I must say that I'm looking forward to the day when we move from twisted.web to twisted.web2 to enable all this new stuff, which I also need to handle from my software.

To close everything: web2 already supports wsgi.

Some people are interested in my Gnus look, which you can find here.

Since it took me a lot of time to get it working I think I'll write down the steps I made:

• Installed Emacs-cvs with GTK2 support (it's pretty easy if you don't have any problem), here is the cvs address :ext:anoncvs@savannah.gnu.org:/cvsroot/emacs



• Installed t-gnus (which is the multi-language Gnus version), FreeBSD5 has a port named t-gnus that will be installed under /usr/local/share/emacs/21.3/site-lisp, you should move it from 21.3 to 21.3.50 to use it under emacs-cvs. Here you can find some versions like t-gnus-6_12_0-02 which I'm using



• Then you are ready to use my .emacs with some stuff for python and Gnus. You can download it here.

Hope This Helps

I'm a PDF!

Which File Extension are You?

I'm a proprietary format. But I'm cross-platform anyway. I can get along with this ehe

About "Seven Cool Mono Apps"

I just read the last column on O'Reilly network here by Edd Dumbhill:

Because decoding audio itself isn't a simple process, there aren't any all-Mono ways of doing this yet. Instead, Muine uses the "p-invoke" facility of Mono to hook into the C-based multimedia library libxine. Calling C code from Mono doesn't require any supplemental C coding, unlike most other languages such as Perl or Python.

I would say that he is quite completely wrong. He maybe doesn't know anything about ctypes module, which is a rather complete and useful way to call functions inside compiled dynamic libraries.

You can find ctypes here with a little tutorial here. And now a little example, just to convince you about what I'm saying:

>>> from ctypes import *

>>> print cdll.msvcrt.time(None)

1048777320

>>> print hex(windll.kernel32.GetModuleHandleA(None))

0x1d000000

See you to my next post