Ordered Dictionaries with Python 2.4-2.6

OrderedDict is a super handy data structure.

An OrderedDict is a dict that remembers the order that keys were first inserted. If a new entry overwrites an existing entry, the original insertion position is left unchanged. Deleting an entry and reinserting it will move it to the end.

Problem is, this stuff is only available in the standard library since Python 2.7 while my project also needs to support Python 2.6. Fortunately there’s a back-port available and it is only a pip install away:

# make OrderedDict available on Python 2.6-2.4
$ pip install ordereddict

ordereddict is based on the awesome recipe by Raymond Hettinger, works with Python 2.4-2.6 and, most importantly, is a drop-in replacement for OrderedDict.

However if you want your code to run seamlessly on all Pythons there’s still some work to be done. First of all you want to make sure that the appropriate OrderedDict is imported, either the standard library version (for Python 2.7 and above) or the back-port release.

This is easily accomplished, and in a very pythonic way:

try:
    # try with the standard library
    from collections import OrderedDict
except ImportError:
    # fallback to Python 2.6-2.4 back-port
    from ordereddict import OrderedDict

Fixing setup.py

If you are shipping your code as a package then you also want to make sure that setup.py properly handles the different Python versions. Since setup.py itself is nothing but standard Python module, we can make it more dynamic by applying the same technique above.

#!/usr/bin/env python
from setuptools import setup, find_packages

# your standard required modules
install_requires = [
    'simplejson',
    'cerberus',
    'events',
    ...
]

try:
    from collections import OrderedDict
except ImportError:
    # add backport to list of required modules
    install_requires.append('ordereddict')

setup(
    name='appname',
    version='0.1',
    packages=find_packages(),
    ...
    # no matter which Python, we're now good to go
    install_requires = install_requires,
    ...
)

Handling requirements.txt

When it comes to pip’s requirements.txt, what I think works best is to simply add a diff file which targets old Python versions like so:

# py26-requirements.txt
# install from 'canonical' requirements.txt first (DRY)
-r requirements.txt
# add specific Python 2.6 dependencies
ordereddict

A developer using Python 2.6 would then go with

$ pip install -r py26-requirements.txt

Whereas someone on a recent Python would simply run

$ pip install -r requirements.txt

Since py26-requirements.txt explicitly lists Python 2.6 dependencies only and then relies on requirements.txt, most likely you will only need to update the main requirements when/if new dependencies are needed.

You can check out the commit where Python 2.6 support for OrderDict has been introduced.

Taming Portable Class Libraries and .NET Framework 4

If your project is a Portable Class Library and you want it to run with the .NET Framework 4 well, you are in for a few surprises. Especially so if you are using InstallShield for building your deployment package. We’ve been going through this a few days ago and it’s been kind of  a wild ride. I thought I could pin the whole thing down so that others might enjoy a painless journey through all this mess.

Portable Class Libraries and .NET Framework 4

The first thing you should know is that while the .NET Framework 4 does support PCLs, in fact it won’t run them without a patch. For whatever reason, Microsoft decided that PCL compatibility wasn’t a worth a 4.0.4 update. That leaves us with the need to not only make sure that target machines are running the up-to-date .NET4 release (v4.0.3) but also that they’ve been updated with KB2468871.

You might be wondering why this is an issue in the first place. We could simply install the .NET Framework 4.5 which is backward compatible with the .NET4 and includes the afore mentioned KB2468871. Even better, we could just target the .NET 4.5 on our PCL. Problem is that besides iOS, Android, WinPhone and Silverlight we also want our libraries to run seamlessly on as many Windows editions as possible, Windows XP included. Here is the catch: .NET4 is the last framework version to run on Windows XP. And yes, we got the memo, Microsoft officially abandoned Windows XP a while ago so why bother? Well it turns out that millions of users are still running XP, especially so in the enterprise and SMB. These PCL are targeting exactly that, precisely the accounting software segment, and believe me there’s a huge number of users happily invoicing and accounting on their old-fart-but-still-splendidly-doing-its-job-for-cheap boxes. Oh and the .NET Framework 3.5 is not an option as it doesn’t support Portable Classes at all.

All things considered, it’s still good news. We can build PCL which run everywhere, Windows XP included. We only need to make sure that both the .NET Framework 4 and the KB2468871 are installed on target machines. Easy enough, right?

The strange story about KB2468871 InstallShield Prerequisite

We rely on InstallShield for building our distributions, so I was delighted to find that it comes with a KB2468871 Setup Prerequisite out of the box. All we had to do was add the prerequisite to our setup and we would be done. In fact, our first test was encouraging. We ran the setup on a pristine Windows XP. It installed the .NET Framework 4, then the KB patch and then, finally, our own application which included the PCL libraries. Everything was running smoothly. We then moved on to test the same identical build on a fresh Windows 7 machine. Again, it installed the .NET4 just fine… and then it crashed. Actually, the setup itself did not crash. It was the KB2468871 which was crashing while the main setup itself was left idle, waiting for the KB install to complete. So, what was going on there?

CPU Architecture Does Matter

To make a long story short, after a lot of investigation and an embarrassingly high number of tests we found that our setup was only crashing on 64bit editions of Windows. It turned out that the issue was with the InstallShield Prerequisite itself. It was broken. In a bad way.

The KB2468871 comes in three flavors:

  • NDP40-KB2468871-v2-x86.exe
  • NDP40-KB2468871-v2-IA64.exe
  • NDP40-KB2468871-v2-x64.exe

Three executables, each one targeting a different architecture. Upon inspection of the stock prerequisite however, we discovered that it was launching the x86 executable no matter what the target edition of Windows was. That explained the crashes on x64 systems.

The solution was to create a new custom prerequisite which would download and launch the x64 KB edition on 64bit systems. We then had to update the stock prerequisite too, so that it would only run on x86 systems. So we now had two specialized KB2468871 prerequisites. One for 32 and another for 64 bit systems. They would be launched alternatively depending on the target system. We proceeded to add them both to our InstallShield project and rebuild it. Then we went and tested it against freshly installed Windows. It installed the .NET Framework 4, then totally skipped the KB (like the prerequisite didn’t even exist) and finally proceeded to install both PCL and main application – which of course would crash on execution as there was no KB on the system.

Beaten, but not ready to admit defeat yet, we went back to the drawing board.

Execution Order Does Matter. Or Not.

Our custom launch conditions for both our KB prerequisites were there, and they looked good. Then there was this other conditional triplet. It was validating two registry keys and then making sure that mscorlib.dll existed in .NET4 folder. So, the idea was that the KB installation should only be executed if the .NET4 was on the target system. That sounded perfectly reasonable. As we could configure the order in which prerequisites were executed, we just had to make sure that the .NET4 prerequisite was assigned a higher priority, so it would run before the KB prerequisites themselves. The prerequisites order was fixed, a new setup was built and… nothing changed. KB were still not being installed.

Prerequistes order did not seem to matter. In fact, if we removed that check-if-net4-is-there conditional triplet the KB would install. However that was not an acceptable solution because then the KB would always be installed, causing a reinstall (and a waste of time and resources) on most systems.

Then I had my epiphany. Maybe launch conditions were being evaluated all together at boot time, for all prerequisites, before they were installed and regardless of their execution order? Non-sense you might think (I did). Why allow me to set an execution order in the first place, if launch conditions for each item are not going to be evaluated in that order? Luckily, validating this theory was going to be quick and easy. We just had to reboot the system after the faulty installation was completed, then run the setup again. And guess what? On second run, after the reboot, the KB installation would be executed without a glitch. Bingo. Since the .NET4 had been installed on the previous run, now registry keys were there and the mscorlib.dll was in the right place, so the KB launch conditions were finally met.

We ended up replacing that bogus conditional triplet completely and changing the validation logic. Instead of checking if the .NET4 was installed (on pristine XP and Windows 7 systems, which don’t have the .NET Framework 4 preinstalled, it just could not be there yet), we were now simply checking if KB itself was installed. After all it just needs to be there, no matter the Windows or framework version (on .NET 4.5+ the KB will be installed by default).

So now, armed with our brand new custom KB2468871-x64 prerequisite and a totally re-imaginated set of launch conditions we were finally able to build a setup that would deliver a fully functional Portable Class Library which would run on all possible Windows: XP, Windows 7, Windows 8… independently of the CPU architecture. Victory!

If you read this far you probably noticed that I didn’t include instructions on how to apply these changes to the stock prerequisite, let alone create a new one. You can find those instructions on the InstallShield website, or you can simply use our modified prerequisites. Of course we are providing them without any guarantee that they will work for you. They did for us, and that’s it.

Too long; didn’t read

  1. Portable Class Libraries won’t run on plain vanilla .NET Framework 4.0 unless KB2468871 is installed;
  2. If using InstallShield KB2468871 Setup Prerequiste you’re in for a wild ride;
  3. However, and until an official fix is released, you can opt to use our modified prerequisites instead.
  4. Get the custom KB2468871 (64x) InstallShield prerequisite.
  5. Get the custom KB2468871 (x86) prerequisite.

Conclusion

The stock KB2468871 InstallShield prerequiste has been out for a good while so I’m baffled that to this day I still cannot find any reference about these issues on the internet. Portable Class Libraries are probably still a niche and the fact that most of Macrovision KB resources are hidden behind a wall does not help. Soon or later an official prerequisite update will be released. Until then, feel free to use our mods.

I’ll just add that if we were dealing with an Open Source project, we’d just open a pull request and be done with it.

REST APIs for Humans at FOSDEM

Yesterday I gave a talk at FOSDEM 2014 in Brussels. The conference itself was amazing, with over 5000 attendees literally swarming and taking over the ULB Campus. I was stoked at how smoothly everything was going on despite the incredible number of simultaneous sessions and the number of attendees continuously flowing between buildings and conference rooms. Everybody involved, volunteers and attendees,  has been very welcoming, charming and helpful. In short, I had a blast.

My REST APIs for Humans™ talk was given in a fully packed Python DevRoom, which granted a lot of smart questions both in the final QA session and later during the day. Thanks everybody for the great feedback!

Giving back to my community

A few days ago I tweeted:

Now the project is out in the wild and I’m very excited about it. It’s all italian yes, but do know that CoderDojo is a global movement, and starting a kids coding club in your own town would probably be great idea.

Come meet me next week. I will be giving a short talk about CoderDojo Ravenna and, most importantly, we’ll have a good pizza afterwards. Oh, and we’re looking for mentors to join us.

Cerberus 0.5 is out (and it breaks stuff)

The new release changes the way validation errors are reported. Please note that these changes will also affect future releases of Eve, the Python REST API Framework.

What we had before was basically a list of human-readable errors. Each item in the list, while perfectly fine for human reading, wasn’t really ideal for algorithmic parsing. Why would you want to parse the errors with an algorithm? A common case would be when your client is using business objects to represent API resources (think a client-side ORM), and would have a hard time binding validation errors to the objects themselves. Continue reading

On Why I Still Prefer Traditional Books to eBooks

I’ve been using a Kindle for a long time now, and I love it. But I keep buying paper books. Lots of them. Actually I buy a lot more books than ebooks and it doesn’t even stop there. Admittedly, I am guilty of repeatedly buying paper editions of ebooks I’ve read on the Kindle.

For a very long time I’ve not been able to tell the precise reason why I keep going back to traditional books. It’s not about the reading experience or the sexiness of turning physical pages. Well of course that matters too, as I’ve always been kind of a book fetishist, but since the very beginning of my love-hate relationship with the Kindle I’ve always known there had to be something else, and I couldn’t pin it down.

Then yesterday it finally struck me. I was visiting the awesome Silent Books exhibition with the kids, and there was this Suzy Lee quote pinned on the wall, and I couldn’t stop reading it over and over.

In turning the pages of a book,
a little world opens and closes,
enclosed in a frame.

Story ends, the book closes.
The world closes too.
Then it’s quickly placed on a shelf.

Art can be placed on a shelf.
Shelf-sized art.
Isn’t that beautiful? (*)

So yes, it’s about the shelves. It’s about surrounding myself with those little worlds I’ve been exploring. I suddenly realized that several times every day, while I’m furiously coding or when I’m just walking around the house or heck, even while I’m reading a book, each day my eyes like to indulge on those shelves, the vivid memories of my reading experiences. That’s something the files stored on my precious little Kindle can’t offer me.

That also explains why I only tend to buy what I call “throw away” books on the Kindle. These aren’t necessarily bad books (or I wouldn’t buy them in the first place). They are mostly leisure readings, you know, thrillers and the like: something that’s entertaining to read but it isn’t likely to leave me with a long lasting memory, or emotion.

(*) turns out the Border Trilogy, Suzy Lee’s book from which the quote is taken, has not been published in english, which is a shame. This is a (probably poor) translation of mine.

MongoDB and REST API go for a picnic (video and slides)

I had the opportunity to give my RESTful WeB APIs and MongoDB Go For A Picnic talk at both MongoTorino and NoSQL Day. The folks at PUG Friuli where so nice to record all the NoSQL Day sessions, so here you have it: the full length video of yours truly speaking to a fully packed room crowded with 120 very attentive attendees.

Unfortunately audio is horrible and while all MongoTorino talks were in english, NoSQL Day was an italian-only event. The slide deck is in english however, and is available on both SpeakerDeck and SlideShare.