Archive for December, 2007

Python Magazine’s Article Wish List

Sunday, December 30th, 2007

Welcome to the second occasional PyMag Article Wish List, where the editors of Python Magazine get together to come up with a list of topics that we’d like to see covered, but which currently have no coverage in our magazine. If you’re familiar with any of the topics listed below, or you’re doing something else that’s cool, you are invited to drop us a line with your idea!

This edition of the list has some similarities to the last edition of the list. This is good news and bad news. The good news is that we’re getting really creative article ideas that touch on the high-level topics we’ve identified. The bad news is that even projects that have large and highly active communities have failed to come forth with even a proposal for an article about how to do something interesting with their project. To date, for example, not a single Django proposal has been received. Same with Zope and Plone. Throw Moin in there for good measure. Our regular “Welcome to Python” columnist, Mark Mruss, just did a really great newbie’s intro to PyQT, but aside from that, silence. Same goes for wxPython.

OK – on with the list!

High Performance Computing (HPC)
* Parallel Python (pp) module
* SciPy
* Django
* TurboGears
* CherryPy
* Zope
* Writing a Zope product
* Plone
* Writing a plugin for trac
Web Services
o simplexmlrpcserver
o xmlrpclib
* Flickr (Beej’s API?)
* Amazon
* Yahoo
System Administration
o python-ldap
o Luma (extending?)
* User/Group management
GUI Frameworks
* wxPython
* PyQT
* PostgreSQL (writing stored procedures with Python)
* sqlite
* py2exe
* py2app
* setuptools vs. distutils
* Web testing
* unit testing
* code coverage analysis
* Manipulating [MS/Open]Office Documents
* Design Patterns with Python
* New api’s – opensocial, facebook, google charts, etc.

Blogged with Flock

December issue of Python Magazine Out, and DRM-Free

Wednesday, December 26th, 2007

After the October and November issues of Python Magazine were both out for some time, I was pretty surprised when I joined IRC channels only to find lots of people who still didn’t know about Python Magazine.

Well, the December issue has just been released in both print and PDF, and I think it’s a killer issue. It’s got a pretty broad range of coverage, and all of the authors we’ve worked with so far have been wonderful. If you are, or would like to be an author in Python Magazine, drop us a line!

First, the cover article is Jesse Noller’s article about threading. Jesse does an outstanding job of making somewhat heady material easy to grasp, even for those who are not indoctrinated in the ways of the Python. If this were the only article in the December issue, the wit and wisdom conveyed would make it worth the cost of the magazine alone.

If you’re a sysadmin in an environment using RPM-based Linux distributions, be sure to check out John Berninger’s article (his second for PyMag) on using Python’s RPM module to help insure the integrity of the packages installed on the system (without the pitfalls of configuring, using, and generally coping with Tripwire).

Still not interested? Ok, how about a look at how Python is enabling the next generation of Accessibility architectures for applications that target (or would like to target) users with disabilities. Steve Lee shows us (literally – with lots of images) what accessibility structures *do*, how they work, how you can use them, the types of disabilities they account for, and the like. This is an incredibly enlightening look at a topic that is often overlooked.

In addition, Saša Dimitrijević shows us what options are available for Pythonistas who would like to go mobile with their applications, and app development. He covers multiple platforms, pointing out the distributions to use for each, the coding and deployment tools available, etc. Again, lots of images of mobile IDEs, emulators, etc., are provided. Excellent work!

Finally, our columnists and editors have rounded out this issue with some great columns.

Doug Hellmann, our senior technical editor, has also been participating in the Google Highly Open Participation contest, and has lots to say about it – including how you can take part, and how the contest is benefitting Python as a language, in addition to various Python projects.

Mark Mruss walks the less senior Python coders through the basics of creating GUIs with PyQT4. I actually call myself a ‘less senior Python coder’ myself, and don’t do much GUI programming, so this was a great read for me. Thanks, Mark!

Steve Holden also gave us an enlightening look at how his site gets the news. The combination of simplicity and effectiveness is a nice break from some of the online coverage I’ve seen about “How to use <huge framework> to deploy <miniscule app>”.

Finally, yours truly writes that front page “from the editor” piece that I’m pretty sure nobody reads. To combat this trend, I’ve taken a look this month at…. trends! Specifically, I went back to look at some old (and informal) language usage statistics from the past 5 years, came to some unsurprising conclusions, and spiced things up with a plea to make Python *more* of a no-brainer for web hosts (and, thereby, web developers).

Have a look! You can buy a single issue, of course – or you can subscribe to Python Magazine and get it delivered to your doorstep each month.

The beginnings of a CLI interface to Google Spreadsheets

Wednesday, December 19th, 2007

I’m really re-posting this from an O’Reillynet posting I made a while back in order to test out the code highlighting capabilities of my new wordpress installation. It seems that it’s not really working correctly [UPDATE: Seems to be working now :-D].

Anyway, I hate (HATE!) spreadsheets. Even the Google variant. The interfaces to these things are so full of quirks and clumsiness and idiosyncrasies that I just find them unusable. So when I was required to take charge of a Google spreadsheet, I decided to create a CLI interface to it using the GData API from Google. This is completely unfinished, and requires the gdata python client, but I’ll post more code as I have time to work on it. As it stands, it only *adds* rows. It doesn’t update existing ones or delete rows or anything. This is just a “Hello World!” that contains the functionality I happen to use the most.

#!/usr/bin/env python

  from xml.etree import ElementTree
except ImportError:
  from elementtree import ElementTree
import gdata.spreadsheet.service
import gdata.service
import atom.service
import gdata.spreadsheet
import atom
import getpass
import string
from optparse import OptionParser

parser = OptionParser()
parser.add_option("-a", "--addrow", action="store_true", dest="addrow", default=False)
(options, args) = parser.parse_args()

gd_client = gdata.spreadsheet.service.SpreadsheetsService() = raw_input('nEmail: ')
gd_client.password = getpass.getpass()
gd_client.source = 'pymag-test-1'

def PromptForSpreadsheet(gd_client):
  # Get the list of spreadsheets
  feed = gd_client.GetSpreadsheetsFeed()
  input = raw_input('nSelection: ')
  return feed.entry[string.atoi(input)].id.text.rsplit('/', 1)[1]

def PrintFeed(feed):
  for i, entry in enumerate(feed.entry):
    if isinstance(feed, gdata.spreadsheet.SpreadsheetsCellsFeed):
      print '%s %sn' % (entry.title.text, entry.content.text)
    elif isinstance(feed, gdata.spreadsheet.SpreadsheetsListFeed):
      print '%s %s %sn' % (i, entry.title.text, entry.content.text)
      print '%s %sn' % (i, entry.title.text)

def PromptForWorksheet(gd_client, key):
  # Get the list of worksheets
  feed = gd_client.GetWorksheetsFeed(key)
  input = raw_input('nSelection: ')
  return feed.entry[string.atoi(input)].id.text.rsplit('/', 1)[1]

def ListGetAction(gd_client, key, wksht_id):
  # Get the list feed
  feed = gd_client.GetListFeed(key, wksht_id)
  return feed

def AddRow(columnfeed, spreadsheet, worksheet):
  # take the columnfeed.entry object and prompt for a value for each column
  # Build a dict from the resulting column:value pairs.
  dict = {}
  for key in columnfeed.entry[0].custom.keys():
      dict[key] = raw_input("%s: " % key)
  gd_client.InsertRow(dict, spreadsheet, worksheet)

spreadsheet_id = PromptForSpreadsheet(gd_client)
worksheet_id = PromptForWorksheet(gd_client, spreadsheet_id)
columnfeed = ListGetAction(gd_client, spreadsheet_id, worksheet_id)
if options.addrow:
  AddRow(columnfeed, spreadsheet_id, worksheet_id)
  for attr, val in enumerate(columnfeed.entry):
    for key in val.custom.keys():
      print "%s:   %s" % (key, val.custom[key].text)
    print "n"

How to Build Your Own MySQL 5 Server in AppLogic

Wednesday, December 19th, 2007

I’m building an infrastructure on a grid using AppLogic, which is a grid OS that provides both GUI and command line interfaces to build your infrastructure. I use both interfaces, depending on the task I’m trying to perform at the time.

One thing I have discovered is that there are a lot of somewhat nebulous concepts, abstractions, lingo, etc., and a number of default configuration settings that make doing really really simple things extremely unintuitive if you’re coming from a (more or less) physical server environment.

Another thing I discovered is that, while AppLogic provides a pretty nice collection of template ‘appliances’ that you can drag around the GUI and connect to each other in various ways, the MySQL appliance in particular is somewhat lacking. Specifically, it’s a MySQL 4.0 server. Lame.

There might be better ways to do all of this, and if you know of any, I welcome the input!! For now, here’s what *I* did to build my own MySQL 5 server:

  1. In the GUI, at the top of the left pane, choose ‘proto’ from the drop down menu to show the appliances in the ‘proto’ catalog.
  2. Drag the “LUX5” appliance to the work area in the right pane. This is a very bare-bones appliance. There’s almost nothing installed on it. However, it *is* a CentOS *5* install, and the version of MySQL distributed for this version of CentOS is 5.0.x. Unfortunately, you still don’t have network access, or read-write access to /usr, or enough disk space. The /usr volume is mounted read-only, and is something like 120MB in size.
  3. Right-click the appliance and choose “Branch Class”. This is super important. If you don’t do this, you can lose any changes you make to the appliance later. Don’t proceed until you do this. Seriously.
  4. Once done, the first thing to do is right click the appliance, choose ‘Edit Class’, and click the ‘Interfaces’ tab. Create an interface called ‘net’, choose ‘Out’ for the Direction, ‘any’ for the Protocol, and click the ‘Gateway’ button in the Options column. In order to enable this appliance to reach the internet so it can do things like ‘yum install’, you need to connect that ‘net’ interface to a ‘NET’ appliance, which is another AppLogic-supplied appliance whose sole purpose is to be a gateway to the internet for your appliances. If you don’t have one of those yet, you’ll need to add one. See this doc on how to use NET: – creating this gateway interface will cause a new line to appear in the output of the ‘route’ command when you log in on the device showing your new gateway.
  5. While you’re in this tab, you can also change the protocol for the ‘in’ interface to ‘mysql’.
  6. Next, you’ll need to resize “/” and “/usr” on the system. The only way I know how to resize a volume on a specific appliance is using the AppLogic controller’s command line interface. Assuming you didn’t change the class name of your appliance from ‘LUX5’, here are the commands to use:
    # ca myapp
    # vol resize LUX5.boot size=1G
    # vol resize LUX5.usr size=1G

    Replace ‘myapp’ with the name of your application. Also note, the app will need to *not* be running when you do this (yes, this means volume resizes require an app restart. Yes this kinda degrades the benefits of virtualization. No, I’m not a fan of how this is implemented)
  7. [UPDATE] forgot to mention: by default, LUX5 has like 40MB of RAM, so when you run ‘yum install whatever’, it’ll be killed at different stages in its work by the Out of Memory (OOM) killer in Linux. Right click the appliance, go to ‘Resources’, and up the minimum and default to at least 256MB. I used 512MB.
  8. Finally, right click your appliance, go to ‘Edit Class’ again, and then go to ‘Volumes’ and unclick the ‘read only’ button in the Options column for the ‘/usr’ volume.

Once you’ve done all of that, you can finally get some real work done. Run ‘yum install mysql-server’ to get the server installed, and you’re on your way. There are still problems to solve with this setup, such as how to manage the volumes for your data directory, how to set up, for example, phpMyAdmin if you want to use that to manage your MySQL database, and how to use things like memcached in this environment. I guess I’ll blog those as I get to them. Wish me luck!

Holy Crap I’m Busy

Wednesday, December 19th, 2007

I feel guilty that I’m even taking this little bit of time to post this. I’m *BUSY*. I’m simultaneously:

  • learning how to deploy a fully-virtualized, production infrastructure on a grid using AppLogic
  • planning and scripting the bits and pieces that need to happen to migrate an infrastructure to a grid architecture.
  • Doing code review for some PHP we have lying around
  • Writing/testing some ETL code for some MySQL databases, and looking into memcached.
  • Writing/testing Python code which will become a middle-tier application server some day.
  • Mucking with every single social, web 2.0, buzzword-worthy…. “thing” I can find.
  • Mucking with Perl code I wrote a long time ago to get it to do things I need to do *now*.
  • Maintaining the current infrastructure services (mostly externally-facing LAMP stuff).
  • Looking for more interesting ways our product can be used, interesting data we can offer users about how users interact with their content, prototyping, testing, banging head on desk.

When I’m not doing that, I’m working on Python Magazine, and I still have a couple of consulting clients who occasionally need me to do bits of things here and there. Throw the holiday season into the mix, and it makes for a really outrageously busy couple of months.

So, forgive me if I’ve neglected this blog just a tad. I’ll try to keep up. Now that I’ve got the code syntax highlighting working, I’ll try to post s’more code bits as I learn/find/discover them.

…So little time

Friday, December 7th, 2007

I’m not usually this bad about writing on my blog, but the past few weeks have been a little nuts. But it’s the ‘good’ kind of nuts.

I’m working on MySQL, memcached, database replication, optimizing SQL queries (and, in one case, the database schema itself), writing Python (using ctypes, and trying to parse the C-encrusted madness that is libpurple), setting up Subversion, performing security audits, troubleshooting PHP code, doing architecture design and scalability planning, and planning deployment of a bunch of stuff in this architecture to a grid solution.

Oh, and I have a magazine to run! I also gave a talk about Python for my local LOPSA chapter last night, and I’m refining some Linux administration training material because I have a client who might want me to do some on-site training in the next few months.

For those who aren’t regular readers, I’m also a new father, and I recently took over bottle-washing duties, which, it seems, can keep you busy pretty much full time all by itself!

I’m sure I’ve left multiple things out, but you get the idea. In the meantime I have a bunch of posts that are just waiting to fly off of my fingers. Gimme another week to get the December issue of Python Magazine out the door, and I’ll catch up with you then 🙂