Why Microservices Matter

In spite of being a marketing spin and link bait, Heroku article on microservices does a pretty good job at explaining why microservices matter and, at a very high level, what challenges they pose.

However, microservices aren’t a silver bullet, and they won’t make a sluggish IT organization fast. While individual services become more robust and less complex, the overall system takes on the many challenges of distributed systems at the network level. Despite their challenges, they’re here to stay because they map better than anything else to the software landscape of the future: parallel development, platform-as-a-service deployment, and ubiquitous use.

Better Pull Requests

I am not a big fan of top tens but 10 Tips for Better Pull Requests is well worth reading.

A good Pull Request involves more than just some code. In most cases, there’s one or more reviewer(s) involved, who will have to review your Pull Request in order to evaluate whether it’s a good fit for inclusion in the code base. Not only must you produce good code, but you must also cater to the person(s) doing the review.

Here’s a list of tips to make your Pull Request better. It isn’t exhaustive, but I think it addresses some of the more important aspects of creating a good Pull Request.

Announcing Eve-SQLAlchemy the official SQL extension for the Eve REST Framework

Powered by SQLAlchemy and good intentions, Eve-SQLAlchemy is an official Eve extension which allows to effortlessly build and deploy highly customizable, fully featured RESTful Web Services with SQL backends.

As with all Eve extensions, once installed with

$ pip install eve-sqlalchemy

using Eve-SQLAlchemy is very simple:

from eve import Eve
from eve_sqlalchemy import SQL

app = Eve(data=SQL)
app.run()

On a fresh virtualenv (of course you are using virtualenvs, right?) the install will also setup Eve and all its dependencies for you. For a complete tutorial you can visit the Eve-SQLAlchemy Support Website.

If you have been following the Eve development you probably know that the original plan was to release SQL support within the Eve core, alongside with the native MongoDB layer. In fact, in a long rant on open source and code responsibility I took the chance to outline the path forward.

However, if you check the discussion which spread in the comments below that post you can see how both my friend and fellow MongoDB Master Flavio and the main contributor to the sqlalchemy branch Andrew were suggesting that a plugin system for data layers might be more appropriate. Not coincidentally that was easy to accomplish since the Eve data layer system was designed precisely with that goal in mind, right from the start.

So, to make a long story short, in agreement with Andrew who volountereed for the task, we changed plans. He refactored the sqlalchemy branch into a separate repository… and then the official Eve SQL extension was born. This approach also allows the extension and Eve to develop asynchronously which is ideal for a faster release cycle.

I like this setup so much that, if in time it shows to be as good as I think it will be, I will probably want to experiment with taking the MongoDB layer out of Eve core too. So with Eve 0.6 MongoDB support would still be preinstalled along with core (so nothing changes as first time user experience) but, being a separate package, users could easily switch to a different driver as they need, still keeping their Eve setup as light as possible.

So if you are willing to help with the development of a mature SQL driver please feel free to join Andrew and other contributors. Your help is more than welcome. Allow me thank them all individually, for the work they’ve been doing on the SQL code is absolutely amazing: Andrew Mleczko, Tomasz Jezierski (Tefnet), Bruce Frederiksen, Jacopo Sabbatini and Peter Zinng.

Get Eve-SQLAlchemy v0.1 while it’s hot!

If you want to get in touch, I am @nicolaiarocci on Twitter.

How to: Add custom JeSuisCharlie header to API responses

A lot of servers have been including a JeSuisCharlie header with their responses. If you haven’t already, try with Charlie Hebdo site itself:

$ curl -I charliehebdo.fr
Date: Mon, 12 Jan 2015 15:56:13 GMT
Content-Type: text/html; charset=iso-8859-1
Content-Length: 221
Connection: keep-alive
Location: http://www.charliehebdo.fr/index.html
Vary: Accept-Encoding
X-Charlie-fr: Je suis toujours Charlie.
X-Charlie-en: I am still Charlie.
X-Charlie-es: Todavia soy Charlie.
X-Charlie-de: Ich bin immer Charlie.
X-Charlie-ro: Inca sunt Charlie.
X-Charlie-cz: Jsem stale Charlie.

I find this to be a great way for us techies to somehow contribute and show support for the ongoing anti-terrorism campaign. So if you feel like doing it here is a quick rundown on how to serve custom headers with your Eve-powered REST API. It is actually a very easy task to accomplish.

Eve provides its own set of callback hooks but since we don’t need fine-grained control (we are good with including the new header with all responses), this time we will leverage Flask’s native callback system instead.

@app.after_request
def after_request(response):
    response.headers.add('X-Charlie', 'Je Suis Charlie.')
    response.headers.add('X-Ahmed', 'Je Suis Ahmed.')
    return response

A simple run script would then look something like:

from eve import Eve
app = Eve()

@app.after_request
def after_request(response):
    response.headers.add('X-Charlie', 'Je Suis Charlie.')
    response.headers.add('X-Ahmed', 'Je Suis Ahmed.')
    return response

if __name__ == '__main__':
    app.run()

All API responses now include the custom headers:

$ curl http://localhost:5000
HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 141
Server: Eve/0.5 Werkzeug/0.9.6 Python/2.7.8
Date: Mon, 12 Jan 2015 15:39:36 GMT
X-Charlie: Je Suis Charlie.
X-Ahmed: Je Suis Ahmed.

If you want to see it in action try sending a GET request with curl, Postman or similar tool to the Eve demo instance (source). Remember Eve is a Flask subclass so whatever works with Flask generally works with Eve too.

If you want to get in touch, I am @nicolaiarocci on Twitter.

Validating user objects with Cerberus

People keep telling me that they want to validate class and instance attributes (object properties) with Cerberus. While it certainly wasn’t conceived with that goal in mind, it is actually very possible to leverage both the Python data model and Cerberus extensibility to achieve object validation.

Nuts & Bolts

Let’s say that we have a simple class:

>>> class Person(object):
...     pass

We create a Person instance and add a few properties and values:

>>> p = Person()
>>> p.name = "bill"
>>> p.age = 44

Now let’s instance a Cerberus Validator and set up some validation rules for it:

>>> from Cerberus import Validator
>>> schema = {
...     'name': {'type': 'string'},
...     'age': {'type': 'integer', 'min': 0}
... }
>>> v = Validator(schema)

As you probably know already, all Python user objects have a __dict__ magic method which exposes class and instance attributes as a dictionary. This means that we can also query our class like this:

>>> p.__dict__
{'name': 'bill', 'age': 44}

>>> p.__dict__['age']
44

You see where we are going with this: we can exploit the __dict__ method in order to let Cerberus perform validation on our object:

>>> v.validate(p.__dict__)
True

Validation succeeds because current attribute values do not break any rule. However, if we break the rules we do get what we deserve:

>>> p.age = -1
>>> v.validate(p.__dict__)
False

>>> v.errors
{'age': 'min value is 0'}

This works, but is somewhat clumsy. We can do better.

A Custom Object Validator

How about letting the Validator do the work for us? We could subclass the standard Validator and extend it to natively support object validation.

>>> class ObjectValidator(Validator):
...     def validate_object(self, obj):
...         return self.validate(obj.__dict__)
...

>>> v = ObjectValidator(schema)
>>> v.validate_object(p)
False

>>> v.errors
{'age': 'min value is 0'}

>>> p.age = 44
>>> v.validate_object(p)
True

Much better. But what happens if we add a new property and then validate the object?

>>> p.lastname = 'white'
>>> v.validate_object(p)
False

>>> v.errors
{'lastname': 'unknown field'}

Validation fails because by default unknown fields are not allowed. This might not be the desired behaviour. If this is the case, we simply need to update the helper method in our custom validator class:

>>> class ObjectValidator(Validator):
...     def validate_object(self, obj):
...         self.allow_unknown = True
...         return self.validate(obj.__dict__)

>>> v.validate_object(p)
True

By setting allow_unknown to True we let unknown fields be ignored by validation. If we are not concerned by state changes between calls we might conveniently choose to move the setting of allow_unknown to the __init__ method so it gets executed only once.

Looks good so far. We can validate simple objects. But what about complex ones like those exposing other objects as attributes? This is going to require some more tinkering.

Validating Complex Objects

It would be super handy if we could add support for an object data type and then provide a validation schema for it, like we already do with the dict and list types. A revised validation schema would then look like this:

>>> schema = {
...     'name': {'type': 'string'},
...     'age': {'type': 'integer', 'min': 0},
...     'address': {
...         'type': 'object',
            'schema': {
...             'street': {'type': 'string'},
...             'zip': {'type': 'integer'}
...         }
...     }
... }

We could then validate it like so:

>>> class Address
...     pass

>>> addr = Address()
>>> addr.street = 'Lexington'
>>> addr.zip = 50238
>>> p.address = addr

>>> v.validate_object(p)
True

>>> p.address.zip = 'not a number'
>>> v.validate_object(p)
False

>>> v.errors
{'address': {'zip': 'must be of integer type'}}

It turns out this is also very achievable. We can leverage Cerberus data type extensibility model to add support for the object type. Then it is just a matter of handling the new type when validating the schema rule. I’m not going into details here but you can check Validating complex objects with Cerberus, a trivial implementation I posted as a GitHub gist.

Closing note

You might be wondering why don’t I add object validation to Cerberus core. Actually, I don’t rule out this possibility but see I like to keep tools as simple, targeted and focused as possible. Besides, there are other object validation tools out there, so adding a new flavour does not seem very useful to me (but let me know if you feel otherwise.) Though as we have seen, if you want to validate user objects with Cerberus, you can do that easily enough.

If you want to get in touch, I am @nicolaiarocci on Twitter.