jmoiron.net

I've disabled comments because I have some neat captcha-less ideas for how to tell bots to fuck the hell off from my comments section but I don't feel like implementing them for django. The new system (mostly homegrown glue w/ selector, beaker, mako & werkzeug thrown in) has been in heavy development the past few weeks and commenting should be available by the summer. Sorry Mike; now you've no reason to come here.

For a while the bots were just hitting a few old posts about 5-10 times a day, so I'd kill the spam every once in a while, but then I started to get pretty disgusting stuff on brand new posts so now it's gone. It's one of the classic models of digital social interaction at work, I suppose.

Otherwise, I've been hard at work developing the new site. As part of creating the new site, I wanted to develop very specific things and then generalize them out quickly to create a sort of reusable toolkit. That toolkit (which is standalone, basically) is davenport, and it's been getting a lot of love recently. Some couch-specific modeling stuff and hopefully a davenport CRUD generator will be in the future, as I flesh out the backend of the Hot New Shit. I also have to get my act together and send cmlenz some patches for python-couchdb. Busy Busy Busy.

I've been doing a lot with CouchDB and WSGI the past few months, with positive and negative results. I'm finally getting the "hang" of Document Orited Databases (DODB? Rubbish acronym), and starting to understand what is possible and what isn't, how to do one-to-many and many-to-many relationships without having lots of queries. It struck me that although the per-query time in CouchDB is thusfar a lot slower than relational databases, the overall database time for individual webpage loads is far less, and there are far less queries and complex information joins going on.

There are a few reasons for this. The first is that Django is a general purpose framework, and it's ORM is similarly meant to be general purpose. The result is that getting the comment counts for 10 posts on the front page takes 10 queries. This type of a query will be possible in one view in CouchDB when Reduce is implemented, but the only way to do this without modifying your "data schema" is to get all of the comments and count them manually.

Because data is so fluid, it's no problem to add a "comments_n" field to each of your commented-on documents. This kind of application specific update-on-write hack is probably going to become very commonplace as CouchDB gains mindshare and the "best practices" are discovered, not because there are inherent limitations with the view system (or at least, there won't be, once it's done), but because the lack of a schema makes it extremely easy, and it's always more efficient to write the application this way than to calculate it over and over.

In light of all this tinkering, I've done a few things. I've started to take a closer look at Erlang especially of late, as I've started to use CouchDB trunk instead of 0.7.2. I've also done a lot of python speed & feature related hacking with both the more or less official couchdb-python library and with my own re-implementation using pycurl instead of httplib2 which I've called curlcon. Originally, I had devised an httplib-style replacement object that used curl as it's transport layer (Hence Curl Connection) to be plugged into python-couchdb, but it evolved into a sort of experiment to see how fast I could get the http part of CouchDB to run.

The code is now available in my mercurial repository under the "slipcover" repository. As I start to add testing to some of my "web framework" for the next version of jmoiron.net, "slipcover" will become more of a full project whose purpose (besides to run my website) is to be an example of an idiomatic CouchDB/Python web application.

Happy Anniversary to me!

6 years ago I started my blog on the now defunct IRIX server attila.stevens-tech.edu. The first post in this blog was made with a bash script that basically used cat & sed to style a quasi-structured text file that I'd shell into the server and create. The server did not offer any CGI services (for fear that the students would screw up and bring down the server, which also handled email), so it would be a year or so until I was able to move my website to PHP.

From there, I wrote two versions of jmoiron.net using the classic LAMP stack. The second one used PEAR:DB for database safety, a thin home-grown templating system based in PHP (both heavily inspired by jeremy mikola).

In May, 2003, I started to learn the Python programming language (This was the beginning of my summer vacation for the Junior->Senior year of University). My very first mention of the language in these hallowed pages was, well, completely retarded: I was "gonna write an interpreter, or a C compiler, or something." It's funny ha-ha.

By late 2004/early 2005, I had ditched the PHP beginnings and had written some custom mod_python stuff to run the site. I ditched mysql at the same time. By late 2005/early 2006, I started writing my own plain-text -> HTML markup language based on MoinMoin syntax. I had almost my whole database converted (automatically) to this script when, in mid 2006, I traded manual mod_py for django and my markup language to markdown.

I quickly ditched the first iteration of the django site and ended up with what you see today; It's been here for over a year and a half! It was by far the harshest transition, because I also made a switch from storing my posts as HTML to storing them as markdown (which I am, of course, now unhappy with); and back then the automagic html->markup filters left a bit to be desired. I am working on yet another iteration with yet another set of technologies; the next iteration (probably set to finish around April) won't even involved SQL at all, and will be the first one without a real solid framework since I moved to Django.

This site is kind of a technical experiment of mine; It's where I express myself both through code, through design, and through words. Hopefully sometime soon, through pictures too. I hope that it will continue to be something that I can hang out there to dissuade future employers from hiring me!

I've been thinking about WSGI and CouchDB recently, while on the subject of digital inflexibility. First, I want to clarify a few things about what I mean by flexibility with respect to an application, and how the current crop of frameworks approach this problem. If you want to follow this musing well, I highly suggest reading "What PHP Deployment Gets Right" by Ian Bicking; or just his entire blog, and most of the crosstalk on the web about REST, Web Services, and the evolution of the WWW.

How do modern frameworks (Rails, Django, Turbogears, or other "canonical" ones) deal with the problem of flexibility? They don't, for the most part. For one, flexibility is hard both programmatically and conceptually. Secondly, they replace flexibility with simplicity, which is almost always a tradeoff that results in quality. They achieve this simplicity by strictly dividing tasks and then conquering each task by building up a structure around how one is supposed to go about solving that task.

These are not negative qualities at all; one of the things that Rails has gotten right is that design ought to be opinionated. So your task when you go to develop in your now "classic" framework is to set up your REST API (your "Routes" or "urls.py") and your controllers, design your models, and set up your views, and you've got the whole MVC ready to go. The problem is rigidity, repetition, and BigDesignUpFront.

The solution is flexibility. Joel defends BigDesignUpFront, and when you are working with a team on some critical make-or-break software for your company, BDUF might be well worth it for it's benefits in fleshing out potential problems, helping with schedule and cost estimation, etc. But for prototyping, exploring technology, or exploring a problem space, BDUF is deadly. For "agile" or TDD, popular buzzwords that are worth far less than their hype but still provide useful insight for all developers, this is potentially damaging. Coupled with rigidity (in the form of SQL) and repetition (even in DRY espousing frameworks like Django) this is tough to overcome, especially when looking at migrating lots of data to a new application framework.

The first way to overcome flexibility I want to talk about is WSGI, whose design is inspired in part (or so I understand it) by Java Servelets. At it's core, WSGI is a specification for how web servers and python applications communicate; but more interesting (and far more necessary in the statically typed world of Java) it also defines specifically how various python applications are called by the web server. This means that other python applications, given that they abide by the specs, are free to call other WSGI applications themselves with impunity and expect them to work.

The way it's implemented, you need only define the __call__ method to receive 2 passed arguments and return an iterable in order to qualify as a WSGI application. These are incredibly weak requirements on applications, and make many middlewares truly plug and play. What's more, the effort was originally to define a standard that the existing plethora of Python frameworks could all use so that their component pieces would be interoperable with each other. WSGI is still pretty new, and opinionated frameworks like Django are probably not eager to ditch their middleware integration layers for pure WSGI interfaces anytime soon (although Django does work w/ WSGI, I think that's more of an interface between a web server and a Django application taken as a whole), but the proposition of using, say, Django's caching middleware, for any python web application written to conform to WSGI is really exciting.

This gives you flexibility in designing your own "framework" built of hand chosen component pieces. Pylons is essentially a framework built upon PythonPaste that facilitates you in choosing these WSGI middleware components, but I've found some of the areas (particularly the URI routing) to be a little less flexible than I'd like (and, sadly, the documentation is a far cry from Django's). Accepting the dogma of one framework or another does come at a practical advantage; you avoid writing the necessary glue between components. But as the glue itself is agonized over, standardized and simplified, it becomes just another component.

It also gives you another interesting flexibility: the ability to attach applications written completely differently (even in different frameworks) to different URIs at the same site, all of them using the same middleware. This blog works as a Django application; why change it? But my Gallery might be better implemented using other technologies (and I discuss this below); with everyone on board using WSGI, it'd be trivial to attach a different application to handle the '/gallery/' URI space but keep both applications using the same caching, gzip, and authentication middleware. This idea is extremely powerful, because it allows one to select the proper tool for the job and align with whatever tool chain most closely reflects the problem at hand.

What about flexibility at the genesis of the application? Web applications these days deal mostly with the storage and presentation of data. Certainly, the current crop of frameworks reinforce this idea; ditch Django's ORM or ActiveRecord and see what's left with respect to creation of a data driven website. This is where CouchDB, or what I perceive CouchDB to offer, enters the equation.

As a metaphor, lets look at programming languages and type binding as a method of describing and manipulating data. In a statically typed programming language, the structure of data is described explicitly and is enforced by the compiler. You go about defining what a widget is, and then create instances of widget. Methods that would manipulate widgets must receive a widget as their in put.

Where statically typed languages provide subtypes, super types, and other ways to make the definition of what qualifies as a widget more malleable, databases struggle at this. You describe data (in the form of tables, relations, etc) beforehand as before, with each field being a strict type and each table describing some strictly typed record. To alter these definitions, you have to define new tables to make additions to the previously defined record types, and modifying the type of their existing data is not possible.

If you want to act on all widgets, you must be cognizant of other widget like tables. Even if your new widget is exactly alike from the old one, grouping both is either manual or inane and always slow. So how do you do migration? You dump the database, add or massage the types of the new table columns you will be adding, and then re-import. Some frameworks provide tools around this process, but the necessity is fundamentally broken.

The document oriented approach CouchDB takes is much more like having a large, flat, "duck typed" table where you can store anything. You define views of your large data soup that pick out items based on specific characteristics of those items, not on their structure. Want all "things" published on some day? It isn't a problem; everything is a thing. A quick stab at structure is to add a type field that allows you to filter out "things" that match a type string. These things are guaranteed, upon delivery, only to have matched that type string and nothing more. This is a weak guarantee, but weak guarantees buy us flexibility.

In Python, often times functions are described as taking objects that allow certain actions on them; for instance, iterable. Requiring only that an object be iterable is a very weak requirement, far weaker than acting on "anything of type foo". In practice, many functions merely require that the objects they manipulate only contain certain methods or attributes, not necessarily that they satisfy some larger unused type structure. This is a trade off, to be sure, but it's a trade off towards both simplicity and flexibility.

As a concrete example of how this can be useful, lets take my ever languishing gallery application. The goal is to keep in the database my images as well as their EXIF tags such that I could easily perform a search like "Find me all images with this aperture" or "Find me all images taken with this camera." Because I have images taken from at least 3 different cameras (not to mention pictures my friends or family take that I might want to include), and camera makers all add their own types of tags in the "MakerNote" section, I can't have a single per-image "tags" table.

As it stands now, my proto-SQL database has 3 simplistic tables to handle this: a gallery_image consisting of id, title, description, etc; a gallery_image_tag, which is supposed to represent a single EXIF tag consisting of an id, title, desc, etc, and 'gallery_image_tags' which allows me to tie the two together so I can get "an image and all of it's exif tags" in one query. This is straightforward (albeit painfully unoptimized) using the Django ORM, but it's a horrible rigid design that sees me making potentially dozens of database updates for each uploaded image.

In CouchDB, I could simply designate my images as having a type field of "image", and then dump in the tags as key/value pairs. It is as trivial to create views of the type described above of this database as it is to create a view returning all images; while the 'all images' view would map documents based on their satisfaction of (type == image), the more complex views are just as simple (camera_model_name == ...).

Looking to the future, it is also far easier to modify the CouchDB database to allow for new features. Lets look at some potentially interesting features: an algorithm that gauges the color temperature of a photo to group "like" photographs together, such that you can view a "gallery" of dusk pictures, dark pictures, or black and white pictures algorithmically rather than by manual tags. Implementing facial recognition to determine whether or not a picture is a portrait. I could run these algorithms on my database images in batch mode and then simply update each document with their temperature score or their boolean portrait status without ever explicitly modifying any structure. As the temperature scores or portrait statuses are tabulated, they are added to each document and the "gallery" views incorporate them automatically.

Software developers have this kind of wish list view of the future, where writing a web gallery can quickly turn into pushing the forefronts of computer vision technology or spawn a perl to python compilation project. Sometimes these whims manifest themselves as something very interesting or inspiring, and wherever they aren't too critical they should be possible!

I have been programming in Python for quite some time now, and I've been doing it professionally for over 2 years. Despite this, I am not nearly as proficient at the language as I could be, probably because I am using it professionally and have to devote time I could be using learning the language to solving problems.

I find myself struggling sometimes to figure out what the "pythonic" way of doing something is. Whenever I realize a new solution to a design problem or just a regular coding problem, I mull over whether or not it fits the language I am writing in first. When I do this in Python, when I use what the language gives me rather than trying to force it to provide the solution I originally thought of, the results are almost always clearer and faster.

The python daemon I wrote at work in the last few months deals a lot with system state. The Object Oriented Paradigm really shines here, because there is much to share via inheritance, and by extension much to gain. The states I deal with are usually just kept in lists, with objects built around the lists mostly to provide the necessary knowledge on how to create them and occasionally to provide convenient transformations or functionality. There are many places in my program where I want to filter some of these state objects based on an arbitrary parameter; say I want all network interfaces with a last-measured latency under 100ms, or a list of writable data partitions with over 100MB free.

The way I was taught to do this, quite frankly, was terrible. If you have some special list, you are taught to write all sorts of special crap for every transformation you want to allow on that list, and that is that. If you have an object that represents a collection of something, like BagOfFruit, under this school of thought, you'd create some methods like BagOfFruit.filterByColor, or even better, the Bag superclass will have implemented a 'functional' style filter that takes a comparison function.

The "right" answer, in Python anyway, is a lot simpler. You have at your fingertips one of the most delightfully malleable built in generic collection objects in the world of programming languages. Want to filter some items from a list? There's a bunch of easy, short, and agile ways to do that in python code: just grab that list from the object and go. There's 4 ways I can think of, off the top of my head, to filter items from a list in python, and most of them look and read better than adding methods everywhere:

The oldfashioned way:

    mylist = []
    for item in oldlist:
        if item.foo < threshold: mylist.append(item)

The functional way:

    mylist = filter(lambda item: item.foo < threshold, oldlist)

The itertools way:

   mylist = list(itertools.ifilter(lambda item: item.foo < threshold, oldlist))

And the new way:

    mylist = [item for item in oldlist if item.foo < threshold]

The old way is the way you'd think to solve this problem if you were a programmer who did not know python. You think about what know you can do and what you need to return and you go about creating it. You need a list, so you make one, and then you add to it everything that meets your conditions. Even though this is a very manual way of doing this in python, it's still a very useful level of abstraction over C/C++: no manual iteration.

The functional way is so named because map, filter, reduce and it's ilk were created historically for programmers used to that paradigm. Functional programmers deal mostly in data transformations (that's what functions do, since there's no state: everything's a transform), and as such they already had "patterns" on how to deal with many of these "I have a list and I want to do something with it" problems. Unfortunately, if you don't know python, and you aren't from a functional programming background (which is probably true), you'd have to reach for the documentation on 'filter' to know what was really going on in this code.

itertools is a python module bundled with the distribution that provides the same functionality as a lot of standard python functions, but returns an iterator instead. Although it's not the case in this example, if you were not planning on going through every item in the original list, and if that list was very large, using ifilter to filter your original list might be a very large time savings!

Now, on to the "answer." The new way is the best for lots of reasons. It's short, uses the list literal syntax in the creation of mylist, contains only semantics about the creation of the list (no book keeping or comparison function creation), and is more flexible than the filter, since you can store permutations on 'item' in the resultant list trivially (you can do this by composing filter & map, but if you do it in the straightforward way, then have fun iterating over the whole list twice).

But, back to my original question, which way is fastest? I wrote up a quick little test of dubious scientific quality, and here were the results:

running oldway() 100000 times ... 2.91310501099
running functional() 100000 times ... 3.14215993881
running itertools way (iter_) 100000 times ... 3.90754389763
running newway() 100000 times ... 2.10518980026

Not only is the list comprehension cleaner semantically than the other 3, but it is a lot faster. What if you wanted to iterate over the filtered list and do some more complex operations on the filtered set? This is presumably where itertools would be the Right Way (tm), but it looks like it's almost twice as slow as the comprehension. Indeed, when I added code to the comprehension to save the list, iterate over the whole saved list (but do nothing), and then return the filtered list, it still ran 100000 in only 2.61785793304 seconds.

The real importance of this all isn't that list comprehensions are the fastest, it's that their semantic purity is not a performance tradeoff. They really are the most pythonic way to approach this particular problem, and they happen to be the fastest. I ran into a similar problem with some timestamp printing code in a small logging library I wrote:

    tz_adjust = (time.gmtime()[3] - time.localtime()[3]) * 3600
    
    def default():
        t = time.time() - tz_adjust
        ms = ("%.2F" % t).split(".")[1]
        return time.strftime('%H:%M:%S', time.gmtime(t)) + '.' + ms

Since this function would format the timestamp for every logged message, you had better believe that I tested the crap out of it to make sure it was the fastest I could manager. I was dismayed that the time module didn't give me anything better to work with than time.time as far as getting microseconds; as you can see from the code, not only do I have to convert the value (which is seconds since the epoch w/ 6 decimal digit microseconds) in order to get the time, but I had to pre-calculate the timezone since time.time() apparently always returns it's value in GMT/UTC.

Later on, I found out (writing something else where I needed to deal with dates) that the datetime module has a function/object called datetime.datetime.now(), which returns an object that has the current local time and microseconds in easily accessible attributes! I re-wrote my function as follows:

   def default2():
       dt = datetime.datetime.now()
       return "%02d:%02d:%02d.%s" % (dt.hour, dt.minute, dt.second, str(dt.microsecond)[:2])

It ran faster, it was less code, no more fooling around with timezones. What more could I ask? Correctness would be nice. This code has a bug in the way it displays hundredths of a second. While thinking it over, I remembered something I had noticed while goofing with the old code: in python, even if you think a numeric task is going to be complicated, it's almost always faster to stick with integer operations than to convert to a string. I only wanted 2 digits (hundredths), and the conversion would be as easy as dt.microsecond/10000, so I rewrote default2() to use integer division. Here are the times for default, default2, and default3 over 1000000 calls:

• default: 14.675798892974854
• default2: 10.18687105178833
• default3: 9.7911970615386963

My code now basically looked like this return "%02d:%02d:%02d.%02d" % (dt.hour, dt.minute, dt.second, dt.microsecond/10000); far cleaner than the original, actually correct unlike the second one, and the net speed increase was around 33%. Unfortunately, in both of these examples, the multitude of possibilities obscured the "right" solution; hopefully, with Python3k moving forward, and my python skills moving with it, some of the standard library can be merged so that a good mental coverage of it will be easier.

Just "finished" hacking together some code that started out as a small desire and has ended up an obsession of sorts. Initially, it was just going to be a smart, specialized file renamer. It has ballooned into a poorly written (but fairly solid) beast of a program with crc checking and unrar/zipfile support. When I say it's poorly written, I mean it's not beautiful like my xdccq module is, for instance. It isn't elegant in the least, does things in a way that is acknowledged as poor design decisions, and the main dispatch is a giant ugly conditional mess. But it works pretty well!

The script takes a ClrMamePro formatted rom release list and a local directory as arguments (with various options available), and can perform hash checks and smart renames of files to 'Official' names. You can find such formatted files at advanscene or pocketheaven. There is a special switch on the program that removes several hardware dumps from the pocketheaven list and outputs a 'clean' list which will match up more with the 'scene' numbering system.

There are quite a lot of programs like this available, but this one is command line, written in python, runs in linux, suited to logging, and written by me. You'll need the rarfile module written by Mario Kreen (thanks Cheeseshop!) if you want rar support (or if you want the program not to throw an exception when it doesn't find it). You'll also need cksum.py in the same directory because I am feeling quite lazy this time around. Even though there's nothing in there (even rarfile has a windows equivalent wich wraps UnRAR.dll or something) that is overly platform specific, because 1. this was a oneoff, 2. I run linux, and 3. I don't like Windows anyway, this won't run in Windows and I have ZERO impetus to change that.

Anyway, the help screen (options):

usage: gbaname.py [options] datfile [romdir]

options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -c, --crc             perform crc checking
  -r, --rename          perform rom renaming
  -p, --pretend         do not rename but display actions
  -m, --missing         show missing roms
  -g RANGE, --range=RANGE
                        perform only over range (#-#)
  -n, --nds-clean       remove MAX Media Launcher, reorder, etc
  -v, --verbose         ever present verbose mode

Features:

• looks up files by release number and correlates them to your release info dat
• performs actions over a range of release numbers (rather than all files)
• renames files according to the dat
• performs crc hash checks on files (.gba, .nds, .zip, and .rar supported)
  • if a file fails a hash check and renaming is active, (!) is recorded in the filename
• clean pocketheaven NDS lists (this feature is pretty dangerous; it will mangle non-PH lists)
• show a log of 'missing' roms (missing from the current list of known dumps)

If you still want it (it is pretty to look at at least), you can download it. There are some useful recipe's in there, even for what's basically an elaborate one-off... too useful not to throw up on the internet.

There are tons more features I can think of that I'm in no hurry to implement, including a feature that was initially planned which is doing fuzzy string comparisson (using normalized levenshtein distances) to figure out what file is what rather than release numbers (which, afterall, can be wrong). This would provide a great deal of safety to the whole thing (although if your archive's crc files match up it's pretty straightforward to modify this script and fix the names). I won't make any ridiculous patronizing remarks about how this is for your own backups etc and so forth, since most people who are looking for something like this do not own 2600 gba games and fret unduly over their private dump collection. I will add (interestingly) that, using my own linker (which dumps as well), I was able to dump a bad image of my broken Metal Slug Advanced cart and good (matching the 'release' crc!) images of Final Fantasy IV and Megaman Zero 2. So.. hey!

I mentioned before that I had splintered off my blog developing efforts into a project I named saudade. The first somewhat releasable ball of code is now finished and in operation at jmoiron.net. This is a (thusfar modest) blog engine utilizing Django and focusing on levereging existing technologies, best practices, and a slim codebase. Some features saudade has over my previous software:

Saudade Specific

• saudade is developed with deployability in mind
• auto-summary support utilizing the BeautifulSoup html/sgml parsing library
• improved rss feeds
• more reliance on generic views (reduction in code size and complexity)
• ditch custom flatpages for django.contrib.flatpages (reduction in code size)
• improved blog navigation
  • navigate by date or page/list view

STILL PLANNED:

• generic callendar widget for linking information
• todo list user interface enhancements
  • removed 'created' from editable fields

I need to work with the template system and get the template loader working so that I can have a drop in system before I release a tarball, but development is coming along well enough. I'm officially done with the todo list for now, even though there are some enhancements planned, and turning my full focus to the gallery application. First up is to work w/ ceache on his pyexif project to get the parsing and table creation working for at least Canon cameras. After we have an idea of how that will work, things can proceed!

This one's a quickie. I promised to make available some changes I made to Gene Cash's EXIF.py library. I'm only providing them as a diff patch because ceache is working on improving the performance of the library extensively by using PIL's exif parsing code with EXIF.py's MakerNote deciphering code.

Image.open()

parse_file()

He's hard at work banging out the changes and hopefully once he's done long awaited work on the gallery can start anew. There is already quite a lot going on in terms of development, but I still have to document a lot of changes and finalize some things. I'd say the todo app will be ready to launch in another week or two.

It started out as an insertcredit forum post, but the promise of extreme difficulty coupled with my complete amnesia about completing Super Mario Bros. 2 made me very interested. To throw some fun into the mix, and as a little academic exersize, I wrote the mario challenge! website for me and some friends to track our progress and win fabulous prizes.

The two linear mario games are very straightforward, and vary wildly in difficulty, which makes them really good for this kind of challenge. Their systems and controls are so tight that you hardly ever feel cheated.. in fact, more often, you feel you got away with glancing off of that koopa-troopa and flying up into the wild blue. Mario 2 in particular has somewhat of a mythical quality about it's difficulty that really makes it quite a pleasure to grind off of. Damn you 2-2, indeed.

Creating the website took a bit longer than 20 minutes, as you might have heard elsewhere. Most anything serious does. As a little test for myself, I ported a simple mp3 announce script to use banshee instead of audacious and timed it. Essentially, the real meat was porting a single, simple, "mp3" object to use one information finding backend to another, and even with the copious code pasting, I was still only able to put down the 95 line script after 30 minutes of frenetic toil.

So, the mario site took from 5 - 7 hours, including creating the html, css, forms, content, designing the database models, and writing the views and urls from scratch. Some featuers:

• administration backend that can edit all site content and users
• user authentication
• milestone system and user modification of personal milestones
• info pages for most objects created
• cute pictures of mario

In all, I made 3 models, 3 custom template tags, 2 forms, and about 7 views and templates. With that project I think I have really entered the 'intermediate' level of django development. I'm comfortable enough with the framework to do most anything with it, but I still do not know it well enough to really start saving me a lot of time (developers with deadlines!). For instance, I could have saved about 2 hours and implemented a few more features if I had known about generic views and manipulators. I could have saved another hour or two if I had known about the user authentication system beforehand, generic form creation, and the various scaffolding scripts that automatically generate form template code from your models.

Eventually, I'd like to build up enough familiarity where most of my time is spent figuring out how to organize the app and not how to write it, but the framework looks very promising towards actually approaching this. Thankfully, even the 5 - 7 hours I spent ons uch a simple site was still a wild improvement from other projects i've done: doing the site in PHP with the user authentication and database code would have been a nightmarish 20+ hour ordeal that would still be rickety and sql-injection prone.

I spent most of today investigating different ways to read EXIF data in Python for the up and coming gallery. A few days ago, as noted elsewhere, I spent a day investigating Mono and writing trivially simple GUI applications in boo. These were kind of liesurely activities, but their usage was immediately relevant to anyone.. even a non-coder. When my images are immediately available to everyone along with the ISO setting, shutter speed, and timestamp, normal people will understand. When I show my parents GUI apps that I have written, they understand its uses.

The complexity of programming is shifting from technical complexity to a more user-centric complexity. I spent more than half of today looking at different existing image gallery software and image gallery presentations. Some css only galleries are actually quite impressive. After quite some time, I finally got something working and grabbing exif data from photos taken from various cameras. Figuring out how exactly to present this (What should I cache, and how? What will I do as far as thumbnailing, fixing orientation, etc?) is going to take quite some time yet; at least some number of weeks.

Speaking to amit about this this evening, he came up (independently) with the same idea that Ceache and I had been kicking around ever since he got his SLR: combine GPS data w/ pictures and integrate a google maps mashup into your gallery. As more and more complex and useful API's are made public and available, the hard part is increasingly knowing what is out there and what you need to leverage them, and not really writing the program.

In the end, PIL did what I wanted but was missing a few vital pieces of information from the vendor specific tags (notably, the ShutterSpeed and ISO number are MakerNote's for canon cameras), so I settled for using (and modifying slightly) EXIF.py. I was a bit disappointed not to find anything simple for using ImageMagick for python (I could have sworn I had seen it), but PIL looks like it does most of the things I will need for image manipulation.

I finally understand apache, and what really happens internally when a request is made. I wasn't able to understand mod_python until today. Until the limitations of the server page model are ground against in such a way that you actually wonder, "There has to be a better way!" you aren't really ready to write your own Apache request handler. Even if it's totally fucking easy.

PHP has to deal with novices by design. These profiteering entrepeneurs are going to make it big with their bulletin board software if they could just figure out loops, and they just need a little help stepping over this barrier. Apache's mod_php was built to "do the right thing" in an easy to understand way (server pages), and although I have my beef with the language I feel that the module writers were largely successful. But what you don't know as a PHP developer is that the entire PHP environment is but one solution to how you could treat page requests.

When I wanted to write my page in Python, I wished that I could just "write the page" and be done with it. For simple stuff, PHP and the server page paradigm work flawlessly. I read the mod_python documentation in with no lack of dread, wondering why they were asking me to code something that wasn't even part of my page. Getting over that hump took a while; it takes grinding up against the limitations of being confined to a server page paradigm. Without getting into dogma and holy war too much, lets just look at something neat you can do by writing your own handler.

When Apache gets a request, you can essentially hook into it and do what you like with it. With mod_python's PSP handler, you can get a decent server page style environment to write quick scripts with. But mod_python also lets you write your own handler, which can essentially be only a slightly customized version of any of the default handlers (and most likely usually is). Let me provide an example:

from mod_python import apache, psp

l = ['a complex', 'object', 1]
vars = {
    'sitename' : 'jmoiron.net',
    'baseurl'  : l,
}
def handler(req):
    global vars
    req.content_type = "text/html"
    template = psp.PSP(req)
    template.run(vars)
    return apache.OK

I won't explain this too much, because I feel a lot of it is self-evident. I'm defining a handler function that will handle incoming requests for "*.py" (this option is set in a .htaccess file). When a request for "index.py" is received, mod_python will call my handler with a request object. The python server page module (psp) is then used to build a template from the request (which has the filename in it), and then the template is "run". The "template.run(vars)" line is really the only one of any importance. It "runs" the python code in the template, and passes in the vars (that i created) as global variables. So, suppose that the psp file that we have requested is this:

<%
sys.stdout = req

...
# lets pretend we have printed out the HTML tags

print "%s, %s" % (sitename, baseurl)
%>

This will print out "jmoiron.net, [ 'a complex', 'object', 1 ]". Once you realize that mod_python is a python interpreter that is resident within apache, and thus always running (and with a somewhat constant environment), it's easy to see the usefulness of such a simple peice of code as the handler above. Since you can pass arbitrary objects, it should be possible to pass things like database handles and other things that you might desire to be persistent.

The average latency for blogposts has increased for me and for others, but the bandwidth has stayed about the same, regardless of what format I'd rather employ. The problem lately is that I detest my ¼ complete backend and the entire structure behind it; Jerm's decision, although heavily ridiculed by myself and yet others, was the right one, because he and I are of a particular breed of perfectionist that can either have everything absolutely perfect, or nothing being satisfactory at all. I am just not capable of doing small, incremental design changes; when I get into a process, I put my all into it, and when I back out, I forget about it and use that space for other things. I work best in batch mode, a-la Knuth.

I've been reading some python blogs recently, mostly about the weird monkey-typing system they plan on implementing. I'm not so sure I like this; this is not what python is. Before I explain that a little more clearly, I'd like to give some background information.

I've finally started Graduate Classes at Stevens Tech, and decided to get the ball rolling with CS 571 (Java) and CS 765 (A special topics course called "_Foundations of Programming Paradigms_"). The former is taught by an industry man, and the latter by Andrew Koenig, who is more of an academic.

Since Greenshpun is opinionated, he gives his opions on the Java language; why the lack of multiple inheritance is bad, or why interfaces are good but lack the power of simpler and more elegant solutions (my favorite "interface" solution is duck typing). Koenig on the other hand is much more drawn to the human side of things, the "how does this (or that) play with our cognitive process", and inspecting why certain structures work better than others. In the respect of simply studying programming languages, which is one of my favorite topics of study, these two classes are serving me very well.

Having two classes from completely different ends of the spectrum gaze upon one issue (language design) really leads to interesting insights, and sometimes these insights were even known to me (and definitely to others) previously, but never explored more deeply. But I have come recently to realize, much to the help of Jerm (who probably doesn't know it), that languages are tools in more than one dimension. When you understand why something is, you start to see a bit deeper and realize what it is.

There is this cliched maxim in CS about choosing the right tool for the job, but theres also a completely competing strategy that seems to work quite often as well; create the right job for the tool. If you have don't know how to use a screwdriver but are a whiz with a hammer, maybe you can grind down that screw to make it a nail. This has all been explored before in famous papers like the "right-way / wrong-way method of programming", but with today's languages there's an interesting paradox; the language is also a tool; not only so far as semantics or libraries go, but in the ways you deal with the language. For languages that have a runtime, or better are interpreted, this can give you considerable power.

A long time back, while writing his blog backend and reworking it to use almost exclusively PEAR (because, you know, you might want to run msql or postgre or firebird someday...), Jeremy was getting fed up with its templating system. When I spoke to him later about templating, telling him that I felt that CSS was enough and that mixing source and code was OK in projects our size, he told me his solution: use PHP as the templating language.

In languages where you can dynamically load new code and then deal with that code as you wish, using the language itself (or more clearly the interpreter) to do things you'd normally have to write code for is a pretty interesting shortcut, but in this case it was especially clever. PHP already is a templating language, he told me, and I have to agree.

So with all of this language design floating around in my head I plopped down in my chair at work today to do a rush job. We have this visualization that I wrote in PyQt 6 to 8 months ago, and even in the time since I've written it I've become exponentially better as far as code organization and program design that I detest the way it is written and wish I could redo it; this thing is a classic example of what happens when you have no idea what the specifications for your program are going to be, or when you have features that you are adding to the featureset as time goes on. We were told that we have to give some manner of demo on monday, but we don't have the hardware, so none of the daemons for the visualization can be set up. My boss asks me about the feasibility of creating a few dummies for inclusion into a presentation in flash, since he's the new proprietor of a presentation that has to be rushed and completed in 3 days.

"Why don't we just use our existing visualization and then feed it the necessary data to create what we want; then capture it over VNC with vnc2swf and include movies that show something that we actually have done?" -- me

When the idea hit me, I realized that had it also involved Guinness someone probably would have moved their hinge-jointed arm and exclaimed "Brilliant!". In order to actually create these movies though, we'd need to have something dynamic enough that it could change its behavior in time such that we could simulate real behavior. Most of this; bandwidth, etc, could just be randomly generated; but to make all of it grow in time; the simple console that I wrote for the program was just not equipped to do this in batch mode, and besides when you ran information from the logs you lose things like context.

So I set to work creating a demo module that would import the minimum ammount of hooks it'd need to provide functionality to time and loop all of this. I knew that I'd have to be able to save each demonstration (5 in all) as a file of some sort, so that we could replay and tweak in case the recording came out bad, or had bad settings, or we just wanted some timing to change, etc. Then it hit me; why not just let Python do that? I needed some language that would have access to time, loops, and pass back strings. Python itself has more than enough functionality, and if I formed my individual demo files as valid Python code, I could dynamically import whatever I want and then make a call to a predefined function to run it all.

As it turns out, with the nature of the program (3 or 4 concurrent threads all running infinitely, gathering data and listening for signals), this solution was beyond ideal! My demo templates could themselves import a trivial ammount of shared code, and then I'd be off to the races. I don't have to create a language, or write a parser; I have python already. Oh what I would have given to have thought about this back when I wrote the command console for the application in June!

But this whole experience has taught me a valueble lesson; know what your tools are and why they are. Python works well as a general purpose language, but its real functionality is as a quick prototyping and glue language. When you need a quick prototype yourslef, use the interpreter you already know is running. Sometimes, your hammer really is a screwdriver.

I read the whole archive of Elf Only Inn the other day, mostly at the impersonal behest of Tycho, who's on comic has left me somewhat empty these days. This archive reading of course reminded me that I haven't read the Thin H Line/Sexy Losers in a while, so maybe I'll eventually get to doing that.

Unlike Lorde Omlette, having a blog doesn't help me remember my past any better, and as such I have no idea what I've been doing the past week. I picked up my graduation tickets and worked lots; I'll start full time on monday, which is good. Nico (my boss) told me that he could find me an RA-ship, which makes things all the more interesting. Oh my, what to do? Well, I know what not to do at least; a PhD at drexel.

I'm finding that the math I did not care about as an undergrad will probably make grad school a bit harder, and that I would have to take comp arch. That prospect alone is giving me second thoughts. Still, being an RA would be fun, and at least give me a taste of what its like. If it's more agreeable than coorperate work, which I'm sure it is, I might stick around a bit longer. In any case, if I get an MS, I can teach in New York, which might be possible, too. I wonder if they'd still take me in Japan.

I've been coding solely in python for a few months now; I can't even remember the last significant C program I wrote. I still haven't finished anything, which is a problem, but I have the first levels of software development down pat, so in just a few days at work I was able to cook up some python/qt sweetness at work.

What I have been finishing, contrary to my actions since college started, is video games. I had the pleasure of beating Eternal Darkness, whoose story was a joy that I cannot convey without simply giving you the game. I've beaten Shikigami no Shiro countless times (mostly with dev), recently on "Hard" difficulty using 0 continues. I also had the pleasure of completing Prince of Persia - The Sands of Time. While I wouldn't call the game's ending masterful, I'd have to say it is very good. Sorry ubisoft, you won't get more than that out of me.

I've been having some political conversations lately, and they've reminded me that I need to both read more and to keep up on current events a bit more. The "me" of last year would have been reading the red cross reports about Abu Ghraib for the last 6 months, making everyone else around me know about it before the pictures came out.

Why the hell can I not design context menu's in designer? The idea is so idiotic that I've spent 2 days trying to figure out how, because it's something I think just has to be there. I'm just missing it, or something, because I'm new. But after surfing freenode for a while and asking some questions there, I couldn't figure it out. It has become a major roadblock in the first steps I was about to take towards making my program actually do anything.

I have done all of the swaret.conf reading and inserting into the preferences screens, but I haven't taken care of any writing back to swaret.conf. This is somewhat worrying, because writing in place might be a bit more complicated than I originally thought, since it will involve adding and removing comments.

After a few days of hacking around in python again, I must say that the language, while extremely easy to use, is dangerously productive in a master's hands. I have to reference the library manuals for almost everything I do, and I'm pretty sure a lot of the things I have done can be done much better. Even with all of this time consuming language re-learning, I'm still 100 times more productive in python than I would be in say C++. The libraries are so expansive, and the interpreter still so mysterious, that I'm not sure I will ever be able to say I'm as good with python as I am with C, but if I was, I'd be a dangerous man.

Another side effect is I find myself using classes because of convenience, not because of stupid reasons like wanting to use the languages garbage collector. A good example is that all of the configure file reading and writing is going to have to be done regardless of whether or not I'm using Qt or GTK+, so its all going into the no toolkit file with the package stuff. I figured that it'd be good to have an object that takes care of everything in there and sets up itself with a nice dictionary (callable through overloading "getitem" of course) that has all of the settings hashed by name. This was only about 20 lines of code for the read portion, and I don't imagine it taking me more than 30 or 40 to write the export or writing portion. This kind of thinking and behavior only makes my program more coherent elsewhere, and re-reinforces the readability that most python programs share.

Overloading, once something I thought of as a cardinal sin in C++ and tolerable in Java becomes the normal thought pattern in python because it makes things more coherent rather than less. Overloading language constructs like getitem and getslice (which is deprecated as of python2 but replaced by a process I am not familiar with yet) is something that I've talked about in the past (while creating the database library for this site, probably), but it's still a joy to do.

The two biggest roadblocks in my way to the completion of my project is to figure out how to fork off swaret and constantly update the UI for things like progress bars and information about what's going on and to design the context menu for the package list by hand. Interacting with swaret while its running is also something I don't know how to do yet (for instance: answering "y" to a question it asks me in manual mode). I have a popup menu coming up (that was really easy), but packing it with entries might be difficult (especially since I want pixmaps in it). As soon as I get preliminary (no error checking) swaret.conf file writing working, I will begin to entertain posting code, since then it will do something useful. I don't foresee this happening until at least the weekend, but a 1 week time schedule isn't that bad for a project like this. After this is done, and after (hopefully) Jason Woodward gets back to me, I'll get onto designing the slapt-get portion. At this point, I'm not planning a plugin based architecture.

I am no GUI designer. I have new found respect for all of the people from whom I've consciously or unconsciously stolen the ideas that go into the screens coming up shortly. Actually, I always had respect for a well built program, but at this point I think I have more respect for poorly built programs: in order for them to get wherever they are, a lot of people had to think quite a bit.

I've gotten pretty used to QtDesigner; the only thing I'm not sure of at this point is if it will allow me to design the context menu's I want to make. I'm still undecided whether or not I want to add a toolbar to my program. Things that were annoying before have become almost second nature already; part of the designing thought process. Things seem more natural in Designer because it feels like you have more control over your widgets. In actuality, I think the container method of GTK+ is still superior because of its simplicity, power, and elegance; three things I value very much.

As I was working on some preliminary configuration screens, schizm said that they looked professional, which is really cool because he has an eye for things like this. Johnny said that Qt itself has become more professional; although I'm not sure that's true. I think KDE alienated a lot of people by choosing Keramik as the default; and themes like DotNET or Baghira which copy off of Microsoft and Apple, while looking nice, don't really give the user anything special. Maybe opinions will change when they migrate to Plastik.

Anyway, It took me about 6 hours, but I finally think I have it figured out. Turning an RC file into some settings dialogs is a lot harder than I thought. Here are some screenshots:

• The swaret general configuration screen... I think "slackware version" is too ambiguous or misleading, but I can't think of a better caption.. maybe I should add a tooltip?
• Configuring some stuff about packages
• Configuring the network options
• Configuring swaret servers for packages.. In the end it was merely coincidence that I ran out of space and realized that I should add another tab for these anyway
• Configuring dependencies this has some more server configuration, but I figured its easy enough to find here

I'm really concerned with usability on this, even though its something that probably only "root" should run (or it should run with "root" privileges). People say that free software developers don't try or don't care about polish, just about getting working software. I agree; its more important to me that this works rather than it looks nice: but at the same time, I want to have some pride in it. We'll see how things turn out; hopefully I can finish it before I get bored with it.

I mentioned that I hated having to install Eric3 in order to compile the *.ui files that QtDesigner creates. I did some exploration and realized that Eric3 used "pyuic" (PYthon User Interface Compiler), which, wouldn't you know it, comes standard with PyQt. There goes one huge complaint I had about the whole system as well as 3 packages. I find myself thinking that PyQt did it right, and that libglade should develop a python generator for the glade UI files.

A pitfall that I had was naming signal handlers I accidentally collided with the object's namespace for the actual objects I was handling. For instance, the "file -> new" button would be called "self.fileNewAction" (dislaimer: I don't like camel caps but many tools I'm using use them, so I will too for consistency), and I was trying to create a function "self.fileNewAction()", which of course will work (and overload the old object instantly). This provides a problem when setting the signal handler, as you are setting a function to be the signal handler for what was an object but what is now simply the function. Oh No! I have to be careful and follow the naming conventions a little more closely.

Qt is pretty nice, and after you get over the stupidity issues with Qt designer (why are things static by default? this makes every application break on window size, and requires a good deal of forethought and action to fix!), the whole system is rather nice.

As a side thought, I wanted to create some kind of web-based developers journal, where I'd talk only about my coding (so that if I want to cross reference a solution to a problem I had, I can go to one section), but I actually think that this is that section, so expect more updates to be uncommentable until I completely redo the whole system (the way I did it now is retarded).

UPDATE: here's a screenshot of an extremely early version of "Pacman".. there is no functionality implied by the snapshot currently not implemented, but I'm probably going to muck up the screen a bit before I'm done.

When I first started using python+gtk+glade/libglade, I didn't think that any other programmers had it better than me. I was still in a platform dependent stupor, a sublime state of anti-idiocy insanity where my own idiocy was incalculable and yet undetectable, but then again I had never had any experience developing Gui's even though simple tools were most likely available. My PyGTK Tutorials, available still in the docs section, are a testament to how much I enjoyed the system, as well as how much I began to understand it.

With PyGTK/glade/libglade, I was introduced to:

• My first experience with Python
• My first experience with a good GUI designer
• My first experience with a graphical toolkit

It might sound silly, but I never viewed anything before in those contexts, so I never really participated in the glorious insanity of it all. Getting PyGTK and libglade set up required the compilation of 3 packages, which I was forced to do several times, although mostly because I forgot the --prefix=/usr configure option. Still, I got it set up with relatively few problems, and about ¾ of the sample code worked, with the other ¼ listed as broken on the PyGTK site.

I joined some mailing lists, read some O'Reilly books on Python programming, read all of the FAQ's about GTK and how it worked, and basically learned the whole system as if I was a C programmer merely writing executable pseudo code. The best part is that I basically was. I got a text editor up and off the ground (and working) 4 days from wanting to start playing with it, which for someone with really only 1 core language is quite good. We'll compare it to my time with QT shortly.

The PyGTK/glade days introduced me to many tools, ideas, and people. It made me grow infinitely as a programmer, able to express myself in the then almost forgotten object oriented approach, which would be an invaluable addition to my programming knowledge. I went onto serverside python programming, dealing with the cgi module and postgreSQL module and redoing this site in python. The dormancy of these tasks, which I have only followed up on in a cursory way recently was radically shaken this week.

Trying out SWareT, I decided to write a front end in, of course, Python. Python would give me the flexibility and development speed to surpass the only other established project as well as give me the opportunity to develop for both toolkits quickly. In less than 4 hours this morning, I got all of the packages I needed for Qt(only) development in Python with similar constructs to my PyGTK/glade environment. There are a few key differences, however:

• The Qt UI generation seems simultaneously far more powerful and far less optimal than glade's
• The step for taking designer .ui files to usable code (in one way or another) is done at compile time, much like C's implementation of libglade. This increases speed and allows for widget tweaking to be pre-initial bytecode interpretation step: the speed gains for this have been thusfar indiscernible
• The PyGTK set of tools was 3 packages; the Qt set of tools was 5, one of which is currently incompatible with the rest of the "distribution".

So, my overall difficulty with Qt so far? Definitely the widget placement is not as intuitive as the GTK+ "container" module, and not as elegant (although I'm not sure if it allows me to do things that GTK+ won't), but the major difficulty in programming in python using Qt is, oddly enough, C++. G++ is entirely inadequate in terms of its speed facilities; otherwise I'm sure its an entirely viable and in fact probably excellent compiler. However, waiting 45 minutes for bindings to compile is not how these things are supposed to work. And with 5 packages to compile, almost all of them having C++ source, this extends the process mightily.

My problem with Qt designer can be best described with an example from the excellent yet esoteric manual.

> * Click the decimalsSpinBox and press Shift+Click on the spacer next to it. Click the Lay Out Horizontally (Ctrl+H) toolbar button. > * Click the form (to deselect anything that's selected), then click the form to the left of the Decimals text label and then drag the rubber band so that it touches every label and widget, including the decimals spacer, but avoiding the other spacers and the push buttons. Release the mouse and notice that the desired widgets are selected. Click the Lay Out in a Grid (Ctrl+G) toolbar button. > * Click the Clear push button in the Object Explorer window (Objects tab) and then Shift+Click the Calculate push button, the Quit push button and the Spacer between the push buttons. Click the Lay Out Horizontally toolbar button. > * Click the form to deselect all the widgets and layouts and then click the Lay Out Vertically (Ctrl+L) toolbar button. > * Finally, click the form and then click the Adjust Size (Ctrl+J) toolbar button.

What the fuck? The process describes how to get UI's designed with Qt Designer to not act stupidly; ie: to not have static sizes and positions, but grow with the window. The design of Qt should probably take care of this by default and at the very least Designer should get it out of my hair.

My other large gripe is that the whole development environment does not seem "unix-like". It was easy for me to use only Glade to create python programs, since Glade itself was independent of the source code implementation. Qt Designer works in a like way, but rather than create python libraries that come with PyQt to handle the XML output of Qt Designer, most tutorials that I have read thus far rely on an IDE named Eric to convert the UI to a python module. The end result is cleaner than the current GTK+ solution, but at what cost?

At the cost of adding 2 extra dependencies, and, if you are like me and did not anticipate this, another recompiling (45+ min on an Athlon 1.4 GhZ w/ 1 gig of ram). Not only this, but these dependencies seem like they are for things that would not be needed by most people using Qt Designer; the requirement? Scintilla and the QScintilla bindings. Why the Eric developers would go for a text widget that handles facilities that native C++/Qt widgets already handle seems puzzling, although it becomes apparent that the whole tool set wants to remain as far from KDE as possible. Hence, no Kate text widget. This all means, of course, that I need GTK+ anyway, since Scintilla is written for GTK+.

The IDE itself is largely useless to me, as non-antialiased text is largely an exercise in pain on my LCD monitor. So there are 3 things I have on my computer related to this IDE (including the IDE) that I need to do the things I want to. Heres to hoping that something is done about this by the PyQt devel group!

One of the nice things was that all of the sample code worked, the documentation for the actual programs that I am using is rich, which cause less lookups to the mailing list or to google. The system itself seems almost as easy and friendly to develop in as GTK+: I just wish that PyQt was a more standard component of Qt/KDE (although it is my wishes that PyGTK also gets recognized a bit more than it is now).

I've been trying out a few console only programs in the past week; a few programs that I had been meaning to try for some time. One of them was swaret, the SlackWARE Tool, and the other is Screen, the UNIX tool.

Swaret is pretty good and best of all maintains the transparent nature of the slackware package scripts. The only thing I don't really understand at this point is how it does its dependency resolving: since dependency checking is not built into slackware packages, I'm assuming that the only answer would be that dependency files are stored on the server. A short description, for those of you that do not care to click the link, is that its kind of like a mixture of portage and apt-get for Slackware.

Screen is of course is a rather old application that I first came across reading about Ratpoison and ways of using Linux efficiently with little or no use of X or a mouse. It's the kind of tool that simplify's almost everything you do by virtue of enhancing the building block of what makes you function: in this case, your shell. Screen creates a shell "session", where once inside it makes itself mostly unknown and gets out of your way. However, by hitting ^a, you can enter a few commands into screen. Some of the more useful ones are "c" and "0-9", which allow you to create new windows and flip back and forth between them, much like a tabbed terminal window. Say goodnight, gnome-terminal.

Lastly, screen lets you "detatch" and "re-attach" to sessions (much like loggin in and out), and keeps the terminals within screen alive even after you exit the shell you've called screen from. This allows you to pick up where you left off on any process that will consume your terminal (like IRC or a swaret --upgrade) from anywhere that has SSH (assuming you have sshd), and not only that but use the same ssh session to do multiple concurrent process, all potentially taking up a "window".

The combination of Screen, Swaret, and Sysadmin Class have got me thinking about another possible project in Python, but I'm hesitant to start something after I've left other things (like the admin module to my website or JTEX) so unfinished. However, I do have a large amount of newfound freetime, mostly gained by giving up my workaholic attitude towards the GBA project. The new project would probably be foolishly named "pacman", and the details as to what exactly it does and how it does it will be released when I decide on said details.

I was debating on whether or not to use PyGTK for this project or to try PyKDE and PyQT. Part of me said PyGTK because of familiarity, and another part of me said QT because kde is the default DE on slackware (in fact, initially I used this argument for gnome because so many people who use slack run gnome I forgot it wasn't default). I slightly entertained the Idea of using both, and developing my idea as a backend or a set of classes with which two separate frontends will interact. I'll have to see about how to get around installation difficulties this might give people who do not keep both QT and GTK (AND PyQT and PyGTK) around. Which is just about everyone.

The subject of this post directly correlates to the research I was doing, coming across random KDE+Python websites, and finally reading some comments to an article written at Lateral's site. Someone recommended gnome to this obvious KDE guy, and he politely replied "Sorry, I'm a KDE guy! Thank's though!". Holy shit!, said I, this man is able to be a maniac without being a zealot. This is something I cannot do; so I thought I'd at least attempt to explain some of the positions I take.

I use Linux because within 6 months of using it, I understood more about the way it worked than after 6 years of Windows. I can see through my Linux system and fix problems in ways that, with windows, would demand a re-intsall. I know how my system displays things, and understand how most of my programs work, and for me, this is an improvement over a system that "just works" only most of the time.

I prefer C to C++ because I personally have a strong dislike for syntactical inconsistencies brought on by C++'s confusing class syntax, and because I don't find the need for a low-level, high-level language. If you want to have a large project with object oriented methods of organization, then use a language who's object features are rich and simple to handle like Java or Python. If you want speed, then figure out which parts of your program need it, and program those in C. If you need to be at the machine level (no interpreter), the bite the bullet and use C which is (much like Linux vs. windows) transparent to the coder, as opposed to C++ which hides much of what it does behind syntax, sometimes with obfuscated results.

I use multiple De's and WM's. I've found that changing my workspace around is a good way to not only be preoccupied from work, but to constantly educate myself on how to improve performance within these environments. Whenever something new comes out, I usually try it. Whenever I find out something new, I try it. It happened with Blackbox, Fluxbox, Gnome 2.0, 2.2. 2.4, KDE 3.0, 3.1, and 3.2, and it will happen when Gnome 2.6 comes out soon, and again when the next version of Fluxbox comes out, and when the newest KDE comes out. De's and Wm's are one area where I do not fall into the trap of being a zealot. Konqueror kicks Nautilus' ass, and Kopete has nothing on Gaim. I'll use Evolution until I can't stand it's long load times, and then try out KMail. Whatever I need, I'll use.

And finally, and you knew it was coming, I like python because it's the cleanest language syntactically that I've ever used, and I never fully understood classes or strong/weak typing until I learned Python. I do believe that RAD and more importantly rapid application patching will be the tool that allows Open Source software to surpass proprietary software simply by the virtues of being Open Source and having the source to amend, and along with this I find Python to be the only language that I can truly develop rapidly with.

Ahh, its that time of year again, when the freshman orientation week ends, they get their laptops, and then fuck our connection right in the balloon knot. If this was last year at this time, only this year (bear with me), I'd still be fine with this, because I'd be downloading packages pertaining to whatever interest I have this week at lightning I2 speeds. Unfortunately, Stevens needed the valueble I2 network money to water the grass and plant flowers.

Anyway you cut it, 3 and 2, 1 and 4, 5k a second just does not do. Instead of checking out porn, the freshman could be educating themselves in something. Whether its important or not, there is a wealth of information out there, a wealth of information that doesn't steal my precious bandwidth. Here's a pop quiz for the freshmen of the world: How do you get this girl into that outfit?. Answer? Practice

I seemed to have missed the bulletin about females actually having interest in me and me being attractive. I always assumed this was because these things were in fact not true, and in my humblest of opinions, they aren't. I know that people will have their opinions, but some have their opinions which are very contrary to my own; and I exist only to tell you this, they lie.

Anyway, Pygame interests me terribly, so maybe when I wake up god damn morning my 900k download will be finished and I can play with it. SJbrown, the one who wrote one of the PyGTK tutorials that I modified (not linked here due to laziness) to work with today's libraries, wrote some PyGame tutorials that I will probably be perusing in the near future. And maybe I'll update my website. Until then, keep firing assholes!

It is my contention that Lorde Omlette is a fake; a ninny with soiled knickers. Let me explain, please.

The year: 1999. The day; approximately September 16th. With his charismatic greeting of "Sup Yo?!", Amit Jain 1 announces to the world his plans to code "mad RPGz". 4 years later, nothing. His site, a barren waste land drier than the Mohabi, contains no RPGz. It would all be dissapointing if one were to continue under the guise that this man is actually competent.

You see, this man isn't competent. His skills as a coder languish in mediocrity, tarnished further by poor algorithmic thinking and even worse coding styles. Namespace conflicts, nondescript variables, poor decisions; they riddle code overflowing with examples that show he just doesn't know what the functions he uses do.

But unlike this man, I will not simply say that my software is better than yours without any hard proof. Lets look at some evidence of his folly, shall we?

First, a snippet from his hompage, the source of which can be found here.

//header("Pragma: no-cache");
HTTP/1.0header ("OmletteSoft: Copyright (C) 2003 Amit Jain");

Here is evidence that Mr. Jain clearly does not understand what a HTTP header is, what its for, or what the php header() function actually does. Truly, disgraceful. We could examine the ever famous "webchat" next, but its probably not even sane, calling a text file ( $fp = fopen ($dir ."messages.txt", "a"); ) a webchat.

Lets look at something a little more telling of prowess; an algorithm. Randomness sounds good, does it not? Afterall, it finds abundance in all walks of programming, and especially RPGz. Looking at the same file as before, we find this gem of human thought:

function GetRandomLine ($file){
    $zerg = file ($file);
    return trim ($zerg[time() % sizeof ($zerg)]);
}

That's just lovely. So the random quote isn't exactly random, its a function of whatever time it is; more specifically, the UNIX time, or the time since the Unix equinox; a little odd for a Microsoft pup to be using, no? But Unix time is for just that; time, not randomness. Afterall, time is sequential, and when you are measuring time in seconds as the php time() function does, you end up with a nice sequential list of random numbers, one a second, counting in a clock (%). Refresh any page and you will see that his quotes count downwards. Fun! I guess rand() was too sheik for him. This is not even mentioning that his variable's name is zerg. As a final looksee into the life of this halfwit simpleton, let us go back and take a look into what drives his "webchat". Through the power of his viewsource "engine", we actually can.

if ($_GET['pika'] == 'chu' || $_POST['pika'] == 'chu'){
    ForceAuthentication ();
}
elseif ($_GET['pika'] == 'zerg'){
    RetryForm ('chu');
    exit;
}

Uh huh..

else{  
    print "<html>";  
    print "<head>";
    print "<title>Prepare to feel the full power of the .OMLETTE System!</title>";
    print "<link rel="stylesheet" href="http://www.omlettesoft.com/css/dotomlette.css" type="text/css" />";
    print "</head>";
    print "<body style="background-color: #000000; color: #FFFFFF">";

Ah, I see that smarty, as SQL (since his comments, webchat, passwords, and journal seem to be kept in clear text files), is not really used all that much, and that .OMLETTE appears to fully harness the powers of deprecation and uselessness. Hats off to you, my good man! Needless to say, my cantankerous lambasting of your credibility was all too easy.

And since Mr. Jain took the time to bash Linux for not following Microsoft standards, if you can eve call Th em that, I'll take the time to retort. Just because Microsoft doesn't join the rest of the world and abide by standards that we in the *nix community have had for 25 years (true fs level hard links, etc) doesn't mean you need to get in a tizzy next time Linux doesn't support something created by a company that Microsoft bought out. You know, sorry if we can't support all of the bugs and undocumented "features" of fileformats that after years of dealing with Microsoft users have learned how to avoid. But hey, I guess Microsoft is great for pretend coders like Amit Jain 1 since it comes with so many compilers.

In closing, some source code to my own rant page; so as to provide Omlette with some means of retort.

#!/usr/bin/python
import sys
sys.path = sys.path + [](/home/jmoiron/nondeus/common/)

import cgi
import db
import globals
import header, menu, footer
import time
import journal

print "Content-type: text/html"

print header.head
print menu.menu

post_object = journal.journal()
form = cgi.FieldStorage()
# if the "id" is set; display news ONLY
if (form.has_key("id")):
    id = int(form['id'].value)
    post_object.fetch(id)
    post_object.getrow()
    post_object.post.print_title()
    post_object.post.print_content()
    post_object.post.print_footer()

# if "pg" is set; display page number "pg"
elif (form.has_key("pg")):
    journal.print_nav(int(form['pg'].value))
    page = int(form['pg'].value)
    last_entry = page * 5
    first_entry = last_entry - 5
    post_object.fetch[first_entry:last_entry]
    while object.getrow() != None:
        post_object.post.print_title()
        post_object.post.print_content()
        post_object.post.print_footer()

# display first page of news
else:
    journal.print_nav(1)
    post_object.fetch[:5]
    while object.getrow() != None:
        post_object.post.print_title()
        post_object.post.print_content()
        post_object.post.print_footer()

My appologies if its too clean and readable.

Its been quite a few days since I set out to rewrite this site in python, and so far, I have only constructed a minimal scaffolding where I want my great statue to stand. How the brilliant fire of fervor can die down to the smoking head of a recently lit match, blown out by the wind or put out of its misery by a single rain drop, is beyond me. How my mind that seemingly lives only in pursuit of its own destruction due to overactivity manages to latch on to something so completely only to let it go is also not yet within my understanding.

So what happens? The site sits idly by, a testament to my innability to procure white stripes tickets. My projects stand first and foremost, tall and proud, ivy clad mossen monuments of the effects of time and the ability for someone to break down and be unable to build back up. And even now I can feel it; my mind is a stagnant pool where there was not too long ago a roaring ocean, but a body that was once made of water is now made of gasoline, poised to ignite in a catastrophic display of self destructive feverous activity.

What will ignite it? What was it last time? Was it a person, or was it a thing; a song, an injury, a movie, a flat white rock with golden letters, half rubbed out by attention? If it was love, surely today I wouldn't feel like this.

And until I can make that synapse fire off an exploding ocean of production, i can only sit here in the cold, cold night, drifting on the sea explosive potential energy, observing life in high fidelity, fishing for an idea without any bait. I can't say its not a relaxing trip though, but perhaps I force myself back to the frenetic pace too early; It feels as though its like I always try to go back to work JUST before I really should; and then all of that time spent gathering energy is simply wasted.

Or spent playing freecell. Or playing Zelda. My current freecell record is 97 and 4; I have quite a few failed games of minesweeper to my credit as well. The fucked up thing about minesweeper is that sooner or later it comes back down to a guess. And this rambling of course details the reason my brain doesn't work well.

I started writing this 'weblog' again after a long absense because I wanted to write; and I wanted a forum for my work that was frequently updated enough for people to actually read it; and I think thusfar I have been successful. A major tenet of this rewriting is that I would write about what I want to write about; its useless to me to outline the day's fucked up happenings around the world as I got into the habit of doing previously; after all, there are much better places to gather that information, and much worse places as well.

It brings me so much more pleasure to bring up some small, unimportant truth or insight that people don't already see than it does to bang people over the head with yet more proof that the things they already know are right. Its a dangerous trap to fall into; trying to be informative and actionary ends up sounding repetitive; and it ends up not being interesting; a trap that others are heading towards; its not just me.

And the only way to bring up unimportant truths is to be blatantly truthful. I only usually put one side here; and lo, Wendy noticed. I mention her in passing, or when I'm upset about something, but the truth, if I were to use so bold a word, is that she's far more important a friend than her coverage here would suggest. Why? Perhaps because I wouldn't want to mention her over and over superficially, or perhaps because theres nothing ever interesting to say that you would want to read. But, as long as we're being truthful, might as well set the record straight: Wendy, you're important to me, and I love you.

Maydrea made a long and conclusive post showing how my innocent musings were actually socially unconcious attacks on a feminist movement that I have not read enough material on to understand. I can't say that is really the truth, although it sure sounded straight on; but I'm not really perplexed by the questions I posed; I was just wondering if anyone else found them to be sad ironies. Apparently, people do.

Which reminds me; sooner or later, I wish to escape the legacy of a long line of my ancestors that persecuted women, and the long line of half of my ancestors that might or might not have persecuted black people. Its obviously a grievance that most women have grown up with a puritan ethic and some socialy debilitating feeling that they need to be "the good girl", or that hundreds of thousands of Africans were shipped here against their will for a life of hard work and humiliation for no pay; but I grew up 80 years after the initial womens rights movement; 70 years after womens sufferage, 30 years after the destruction of Jim Crow, 20 years after the civil rights movement hit warp speed, and 5 years after it was generally accepted that these movements were in fact right. I do not have first hand knowledge of a time or place where it was OK to be prejudiced. Forgive me if I don't understand why my peers are so sensitive to these issues, but perhaps I just have the unique position of not being concerned with these things because they do not impact me. Forgive me if I know more guys that are p-whipped than girls who are beat by their asshole boyfriends; forgive me if I don't live in your overbearingly patriarchal society.

It would appear that submission is working, although in a non-final rudimentary form. Still, something had to be done, and even more will soon have to be done because the main page and other pages work off of different databases entirely. Unfortunately, there are no plans to get any of the functionality in the comments or rant section ported to the new system within the next 5 days; but I don't think that will kill anybody.

The insatiable urge to code and get stuff done is surpassed only by the insatiable urge to not do shit. Unfortunately these urges compete against one another with quite the fervor, and I don't want the one that's in the lead to win. That having been said, I haven't been able to accomplish anything recently; and I'm not exactly sure why. I just kinda sit around, knowing what has to be done, but never doing it. I think I need to write things down again; unfortunately, I wrote things down during my most productive times here, so I guess that doesn't work.

I have found the first 'features' of python that rub me the wrong way. First, regular expressions. In order to use one, you must create a 're' object with the expression in question, then you have to run some member function of the expression on whatever other ojbect (usually a string) you want to use it with. It seems clunky and I can see where it could cause people to trip over the syntax.

The other grip I have is, surprise, with regular expressions. Regular expressions and string objects have their own set of escape characters. Suppose you want a regular expression to find "p>". Well, a regular expression has as a special character, so you need to escape it... with a. Ok, so your regular expression needs to be "p>". Unfortunately for you, you're probably going to build it with a string; and in a string, is a special character, so you need to escape it... with another. The string to build your regular expression end sup being "\p>". If you want to dodge that game you can use python raw strings, like this: r"p>". Of course, god only knows what's going on at that point.

Everything else so far has been beautiful. The code running this website is more elagant and probably runs faster than any previous rendition.

In other news, there was some heat rising from ye olde bug trail earlier this week; much entertainment was to be had, and indeed, much entertainment was had. The chickens are a bit restless it seems. Its all well and good for me, at least, since I can kinda sit back and watch it happen from where I'm sitting.

So tomorow is my cousin's wedding. Unfortunately for me, this means that the constant barrage of "When are you going to get married?" now falls upon me, the oldest single man of the family. This question is no laughing matter; any family event, including tomorows, will see yours truly have to field this question in an uncomfortable, sometimes defensive, sometimes offensive way. Every event. Big fat Greek wedding style. Thank's a lot, Nel.

However, in the pinnacle of good fortune, I have a few things protecting me from an otherwise lifetime (lifetime, assuming what they say is true, that life ends after marriage) of danno-style questioning. The first, is that Nelson is childless. This means that he doesn't have any kids, for the uninitiated or english-ly challenged. It also means that for a year or so he will protect me from the brutal barage by shouldering the "When are you going to have kids?" question in place of my marital status update. The second, is that compared to other members of the family, I'm quite unapproachable. I think I tend to make other people more uncomfortable than they think they make me; a notion many family members of mine seem to have, but few realize is false. The third, is that my family insists on doubting my Portuguese skills. Its not enough for them that I listen intently (and understand) all conversations at the dinner table, or that I only make vocabulary based inquiries on rather difficult or region specific words, or that I often correct my father's portuguese. And thankfully, this means I can usually dodge most of their questioning. For instance:

Relative 1: When are you going to get married.
Relative 2: He doesn't speak portuguese.
Relative 1: Oh.
Me: Hmm.

I think you all can see how this can work to my advantage. Finally, I have a brother who's around more. Familiarity goes a long way when asking prying personal questions, and so does being comfortable and not intimidated. Its impossible to not be comfortable, equally as impossible to be intimidated, around my little brother. And he doesn't know portuguese, which for him seems to be an invitation to more questioning rather than an end to a conversation.

So, in my mind, weddings usually suck. Of course, I've never been to a wedding that I didn't like, so I don't know in what way they suck. Looking back on any appearance I've made at a wedding (that I actually remember), I can't think of one that I didn't thoroughly enjoy. Maybe its the fact that I hate wearing a suit. Or maybe its the fact that I detest the large waste of time it is to go through the inevitable 2 day preperation for some large scale family event. In many ways, the cousin who is getting married shares my opinion on many of these issues, so it will be interesting to see how shit goes down.

In an effort to keep the design of Jtex simple, and further reinforce that it is not competition for other projects, I've taken a few features out of the release plan. The project is more exciting to me as a tutorial to myself and others on building PyGTK applications rather than reinventing the text editor yet again. The removed features follow:

• undo
• redo
• changing colors
• properties dialog (its unnecessary anyway)

All of the XML stuff is still going to be there for now; if only because I have already started on it. Its not really central to the tutorial, but I figure that I might as well since almost any application will need to save preferences data, and what better format to use?

I've finished the design of the preferences dialog and removed the properties button from the menu. Since some of the minor revisions on the release plan (below a few entries) are obviously now empty, "connect signals and finish preferences dialog" is moving to revision 0.6 and replacing "undo" For a run down, 0.6, 0.7, and 0.9 are now unnecessary. The new schedule from 0.5 follows:

Features for 0.6

• connect signals and finish preferences dialog

Features for 0.7

• cut, copy, paste (with builtin clipboard widgets)
• begin public release or code archives

Features for 0.8

• finish off tutorial

Features for 1.0

• final testing and release of code

I can't seem to remember what it is that I wanted to have done by Friday night, but whatever it is, I probably don't have it done. The tutorial is still languishing on page 3, and although I didn't get much actual development done, I did learn quite a bit about how to use pango to change fonts in the TextView, among other things.

Another thing I finished is the XML Doctype for JTex preferences settings; and with that, basically, the design of the preferences dialog is pretty much done... all I have to do now is build it in glade and tie it to the activate event.

The DTD language, while looking quite complicated, is actually made up of an extremely simple (and small) grammar. I was able to learn it in little over an