The project in question was Cerberus. The ‘critical’ designation happens when a project has been in the top 1% of downloads over the prior six months. Given that there are currently 388K packages on the Python Package Index, I must admit that having one of my projects in the top 1% does feel good.

However, I was initially a bit baffled in that I would lose control of the package if I didn’t take action. I understand the motivation behind this move, which is to improve the general security of the Python ecosystem. Still, the imposition from above didn’t feel quite right to me. After reading about the rationale for the new requirement, I surrendered, went on PyPI and activated 2FA¹. In hindsight, I should have done that long ago, as I already use 2FA on many other services.

A couple of days later, I was relieved to learn that my initial “1984-dictatorship-alert” ring bell wasn’t the only one to go off. On that day, Armin Ronacher’s thoughts on this very topic made the headlines on Hacker News:

The message to me as a maintainer is quite clear: once a project achieved criticality, then the index wants to exercise a certain amount of control […] However when I create an Open Source project, I do not chose to create a “critical” package. It becomes that by adoption over time. Right now the consequence of being a critical package is quite mild: you only need to enable 2FA. But a line has been drawn now and I’m not sure why it wouldn’t be in the index best interest to put further restrictions in place.

I think Armin’s post is worth reading. I share some of his concerns, although I don’t consider ‘users vetting,’ as he suggests, a suitable alternative for authentication (vetting solves a different problem).

I really feel like I should put more effort into Eve, Cerberus and satellite projects Eve-Swagger, Flask-Sentinel, Eve.NET, etc. I love working on these projects and I know a lot of people rely on them. Also, I have big ideas that I would like to play with. At this point in time however, I cannot afford allocating more time to not-paying-the-bill activities.

Well, I went and looked at the possibilities, and finally settled down on a model which would probably be best defined as a blend between the Vue.js and the Django REST framework models.

Effective now, the Eve REST framework, Cerberus, and the whole eco-system are collaboratively funded projects. If you run a business and are using either Eve or Cerberus a revenue-generating product, it would make business sense to sponsor their development: it ensures the project that your product relies on stays healthy and actively maintained. Individual users are also welcome to make either a recurring pledge or a one-time donation if Eve and/or Cerberus have helped you in your work or personal projects. Every single sign-up makes a significant impact towards making Eve possible.

Rest assured, Eve continues to be open-source and permissively licensed. It is also maintained and developed in my free time, and by the awesome community which has matured around it. I firmly believe, however, that it is in the commercial best-interest for users of the project to invest in its ongoing development (make sure you check the exciting Things I Would Be Working On section on the Patreon page).

To join the backer ranks, check out Eve campaign on Patreon, or donate via PayPal.

Thank you, Nicola.

Join the newsletter to get an email alert when a new post surfaces on this site. If you want to get in touch, I am @nicolaiarocci on twitter.

I just wish to reiterate my gratitude towards all the contributors to the project. The ones who specifically worked on this awesome release, those who made it all possible, are: Matthew Ellison, Dominik Kellner, David Kirkendall, Damián Nohales, calve, Jonathan Huot, Roman Redkovich and of course the one and the only Frank Sachsenheim, whose role was pivotal to this release.

If you want to get in touch, I am @nicolaiarocci on Twitter.

All things considered, I would dare to claim that Cerberus is battle tested to death. This is, in fact, one reason why I believe that the time for a canonical and stable release has come. Another reason is that next release is a major one. It brings a ton of important new features along with very significant code refactoring and a redesigned, powerful API. Third, next release breaks backward compatibility, and we want to signal that in the version number.

So next Cerberus release will be 1.0. If you have been following the development this will come as no surprise, as a Release Candidate has been out for a while. As a Cerberus user you will want to take the plunge and upgrade to 1.0 because well, it is just too cool to be true. If new to Cerberus you will also want to adopt 1.0 right away, for the same reason. If you are new however, make sure you get the basics covered before reading further. By the way, at latest PyCon Italy I gave a talk on Cerberus which also included a preview of several 1.0 features. You can check the slides to get a general idea of the tool, its usage, and upcoming features.

Let’s now look at some of the relevant features and changes introduced with Cerberus 1.0. For a (mostly) accurate list of changes and new features, have a look at the changelog.

Transformation and normalization

Big news in the normalization department. Similar to validated(), the new normalized() method returns a normalized copy of a document without validating it.

Renaming of Fields

You can now define a field to be renamed before any further processing.

To let a callable rename a field or arbitrary fields, you can define a handler for renaming. If the constraint is a string, it points to a custom method.

If the constraint is an iterable, the value is processed through that chain.

Purging Unknown Fields

After renaming, unknown fields will be purged if the purge_unknown property of a Validator instance is True.

You can set the property per keyword-argument upon initialization or as rule for subdocuments like allow_unknown. The default is False.

Default Values

You can set default values for missing fields in the document by using the default rule.

You can also define a default setter callable to set the default value dynamically. The callable gets called with the current (sub)document as the only argument. If the constraint is a string, it points to a custom method.

Value Coercion

Coercion has been introduced with 0.9. It allows you to apply a callable to a value before the document is validated. The return value of the callable replaces the new value in the document. This can be used to convert values or sanitize data before it is validated.

If the constraint is an iterable, the value is processed through that chain.

Please note that coerce kicks in with validate(), not with normalized().

Schema Validation

This is another area the is seeing a lot of changes and new, powerful things.

Registries

Schema registries are awesome if your schemas shall include references to themselves (recursion) and if they contain a lot of reused parts and are supposed to be serialized. There are two default registries in the cerberus module namespace. You can use schema_registry to store definitions for schemas which can later be re-used:

And you can extend rules_set_registry with rules-sets which can then be referenced in validation schemas:

Schema Constraints in docstrings

Validation schemas themselves are validated when passed to the validator or a new set of rules is set for a document’s field. A SchemaError is raised when an invalid validation schema is encountered.

Now you can provide constraints as literal Python expression in the docstring of the rule’s implementing method to validate the arguments given in a schema for that rule. Either the docstring contains solely the literal or the literal is placed at the bottom of the docstring preceded by the following sentence: The rule's arguments are validated against this schema.

The example below is comes directly from Cerberus’ own test suite:

Validation Rules

forbidden

Opposite to allowed, this new rule validates if a value is any but one of the defined values:

min, max

Up to 0.9 you could only use min and max to compare numeric types. Now they define minimum and maximum values allowed for any types that implement comparison operators.

keyschema

For better consistency, the propertyschema rule has been renamed to keyschema. This is the counterpart to valueschema and validates the keys of a dict.

type

Data type validation now also supports the binary type.

Breaking Changes

This is a major release which breaks backward compatibility in several ways. Don’t worry, these changes are for the better. However, if you are upgrading, then you should really take the time to read the list of changes in the changelog and consider their impact on your codebase, especially so if you have custom validators sitting around. For your convenience, there are also some upgrade notes available.

Acknowledgements

Cerberus 1.0 would not exist in its current form without the incredible work done by Frank “funkyfuture” Sachsenheim. After his initial, timid, set of v0.9 contributions, Frank went on a rage. He really adopted the project and went on touching on every aspect of it: documentation, new features, proposal, fixes, refactoring, ticket triaging, you-name-it. His efforts have been instrumental in taking the project to the next level. As a token of appreciation for his work, I’m tagging this release with the funkyfuture codename.

Of course he was not alone. Other contributors were Matthew Ellison, Dominik Kellner, Damián Nohales, calve, Roman Redkovich. And then are all the other people who helped by opening tickets and spreading the word around. Thank you all, folks.

Closing notes

Cerberus 1.0 is an important milestone. I consider the API to be reasonably stable and, once it is released, I plan to let it settle down for a while. Release Candidate will stay out for a couple more weeks, which is ideal for you to experiment a little bit before the final release.

If you are a Eve user you probably see the potential that new Cerberus can offer to your RESTful API, especially in the normalization department. Don’t get too excited though as I do not plan on immediately adding Cerberus 1.0 to Eve. Next Eve release (v0.7), which is also upcoming, will be focused on MongoDB Aggregation Framework and other things. That is more than enough for a major release. I am targeting the following release (v0.8) for Cerberus 1.0 support.

So, have fun with New Cerberus. As usual, please report any issue on the ticket system.

If you want to get in touch, I am @nicolaiarocci on Twitter.

Collection rules

First up is the new set of anyof, allof, noneof and oneof validation rules. anyof allows you to list multiple sets of rules to validate against. The field will be considered valid if it validates against one set in the list. For example, to verify that a property is a number between 0 and 10 or 100 and 110, you could do the following:

>>> schema = {
...     'prop1': {
...         'type': 'number',
...         'anyof': [
...             {'min': 0, 'max': 10}, 
...             {'min': 100, 'max': 110}
...         
...         ]
...     }
... }

>>> doc = {'prop1': 5}
>>> v.validate(document, schema)
True

>>> doc = {'prop1': 105}
>>> v.validate(document, schema)
True

>>> doc = {'prop1': 55}
>>> v.validate(document, schema)
False

allof is the same as anyof, except that all rule collections in the list must validate. Same pattern applies to noneof (no rule in collection must validate) and oneof (only one rule in collection must validate).

Type coercion

Type coercion allows you to apply a callable to a value before any other validators run. The return value of the callable replaces the new value in the document. This can be used to convert values or sanitize data before it is validated.

>>> v = Validator({'amount': {'type': 'integer'}})
>>> v.validate({'amount': '1'})
False

>>> v = Validator({
...     'amount': {
...         'type': 'integer', 
...         'coerce': int
...     }
... })
>>> v.validate({'amount': '1'})
True
>>> v.document
{'amount': 1}

>>> to_bool = lambda v: v.lower() in ['true', '1']
>>> v = Validator({
...     'flag': {
...         'type': 'boolean', 
...         'coerce': to_bool
...     }
... })
>>> v.validate({'flag': 'true'})
True

>>> v.document
{'flag': True}

Properties (keys) validation

propertyschema is the counterpart to valueschema (also new, it replaces the now deprecated keyschema) and validates the keys of a dictionary.

>>> schema = 'a_dict': {
...     'type': 'dict', 
...     'propertyschema': {
...         'type': 'string', 
...         'regex': '[a-z]+'
...     }
... }

>>> document = {'a_dict': {'key': 'value'}}
>>> v.validate(document, schema)
True

>>> document = {'a_dict': {'KEY': 'value'}}
>>> v.validate(document, schema)
False

List of types

The type rule can now be a list of types, to allow for different type of values for the field.

>>> v = Validator({
...     'quotes': {
...         'type': ['string', 'list']
...     }
... })

>>> v.validate({'quotes': 'Hello world!'})
True

>>> v.validate({'quotes': ['Do not disturb my circles!', 
...                        'Heureka!']})
True

>>> v = Validator({
...     'quotes': {
...         'type': ['string', 'list'], 
...         'schema': {'type': 'string'}
...     }
... })

>>> v.validate({'quotes': 'Hello world!'})
True 

>>> v.validate({'quotes': [1, 'Heureka!']})
False

>>> v.errors
{'quotes': {0: 'must be of string type'}}

And there is more so make sure you check the changelog before upgrading. No breaking changes but there’s at least one deprecation, as mentioned.

Fun fact: Cerberus is currently getting 3x the downloads of his sister project Eve, the REST API Framework for which the tool was originally conceived. Interesting if you consider that Eve is featuring 10x the GitHub stars. Fun, but not really surprising since Cerberus probably has a broader audience.

Special thanks to Frank Sachsenheim, Tobias Betz, Brett and C.D. Clark III for their valuable contributions to this release.

If you want to get in touch, I am @nicolaiarocci on Twitter.

So, suppose we set the following validation schema:

schema = {
  'test_field': {
    'dependencies': [
      'a_dict.foo', 
      'a_dict.bar'
    ]
  },
  'a_dict': {
    'type': 'dict',
      'schema': {
        'foo': {'type': 'string'},
        'bar': {'type': 'string'}
      }
  }
}

Then, we can validate a document like this:

>>> v = Validator(schema)
>>> document = {
      'test_field': 'foobar', 
      'a_dict': {'foo': 'foo'}
    }

>>> v.validate(document, schema)
False

>>> v.errors
{'test_field': "field 'a_dict.bar' is required"}

This release will not work with Eve 0.5.2 or less so if you want to use Cerberus 0.8.1 with Eve make sure you upgrade to Eve 0.5.3, released today. By the way, yesterday we hit 2K stargazers and 70 contributors on the Eve repository, quite the milestone!

If you want to get in touch, I am @nicolaiarocci on Twitter.

Nuts & Bolts

Let’s say that we have a simple class:

>>> class Person(object):
...     pass

We create a Person instance and add a few properties and values:

>>> p = Person()
>>> p.name = "bill"
>>> p.age = 44

Now let’s instance a Cerberus Validator and set up some validation rules for it:

>>> from Cerberus import Validator
>>> schema = {
...     'name': {'type': 'string'},
...     'age': {'type': 'integer', 'min': 0}
... }
>>> v = Validator(schema)

As you probably know already, all Python user objects have a __dict__ magic method which exposes class and instance attributes as a dictionary. This means that we can also query our class like this:

>>> p.__dict__
{'name': 'bill', 'age': 44}

>>> p.__dict__['age']
44

You see where we are going with this: we can exploit the __dict__ method in order to let Cerberus perform validation on our object:

>>> v.validate(p.__dict__)
True

Validation succeeds because current attribute values do not break any rule. However, if we break the rules we do get what we deserve:

>>> p.age = -1
>>> v.validate(p.__dict__)
False

>>> v.errors
{'age': 'min value is 0'}

This works, but is somewhat clumsy. We can do better.

A Custom Object Validator

How about letting the Validator do the work for us? We could subclass the standard Validator and extend it to natively support object validation.

>>> class ObjectValidator(Validator):
...     def validate_object(self, obj):
...         return self.validate(obj.__dict__)
...

>>> v = ObjectValidator(schema)
>>> v.validate_object(p)
False

>>> v.errors
{'age': 'min value is 0'}

>>> p.age = 44
>>> v.validate_object(p)
True

Much better. But what happens if we add a new property and then validate the object?

>>> p.lastname = 'white'
>>> v.validate_object(p)
False

>>> v.errors
{'lastname': 'unknown field'}

Validation fails because by default unknown fields are not allowed. This might not be the desired behaviour. If this is the case, we simply need to update the helper method in our custom validator class:

>>> class ObjectValidator(Validator):
...     def validate_object(self, obj):
...         self.allow_unknown = True
...         return self.validate(obj.__dict__)

>>> v.validate_object(p)
True

By setting allow_unknown to True we let unknown fields be ignored by validation. If we are not concerned by state changes between calls we might conveniently choose to move the setting of allow_unknown to the __init__ method so it gets executed only once.

Looks good so far. We can validate simple objects. But what about complex ones like those exposing other objects as attributes? This is going to require some more tinkering.

Validating Complex Objects

It would be super handy if we could add support for an object data type and then provide a validation schema for it, like we already do with the dict and list types. A revised validation schema would then look like this:

>>> schema = {
...     'name': {'type': 'string'},
...     'age': {'type': 'integer', 'min': 0},
...     'address': {
...         'type': 'object',
            'schema': {
...             'street': {'type': 'string'},
...             'zip': {'type': 'integer'}
...         }
...     }
... }

We could then validate it like so:

>>> class Address
...     pass

>>> addr = Address()
>>> addr.street = 'Lexington'
>>> addr.zip = 50238
>>> p.address = addr

>>> v.validate_object(p)
True

>>> p.address.zip = 'not a number'
>>> v.validate_object(p)
False

>>> v.errors
{'address': {'zip': 'must be of integer type'}}

It turns out this is also very achievable. We can leverage Cerberus data type extensibility model to add support for the object type. Then it is just a matter of handling the new type when validating the schema rule. I’m not going into details here but you can check Validating complex objects with Cerberus, a trivial implementation I posted as a GitHub gist.

Closing note

You might be wondering why don’t I add object validation to Cerberus core. Actually, I don’t rule out this possibility but see I like to keep tools as simple, targeted and focused as possible. Besides, there are other object validation tools out there, so adding a new flavour does not seem very useful to me (but let me know if you feel otherwise.) Though as we have seen, if you want to validate user objects with Cerberus, you can do that easily enough.

If you want to get in touch, I am @nicolaiarocci on Twitter.

What we had before was basically a list of human-readable errors. Each item in the list, while perfectly fine for human reading, wasn’t really ideal for algorithmic parsing. Why would you want to parse the errors with an algorithm? A common case would be when your client is using business objects to represent API resources (think a client-side ORM), and would have a hard time binding validation errors to the objects themselves.

So for example, previously we had:

[
    "min value for field age is 10",
    "value of field name must be of string type"
]

With Cerberus 0.5+ we now have:

{ age: min value is 10, name: must be of string type }

Let’s look at more complex structures, like lists. Imagine we have a schema defined like this:

schema = {
    'list_of_values': {
        'type': 'list',
        'items': [{'type': 'string'}, {'type': 'integer'}]
    }
}

And this is the document we want to validate against it:

document = {
    'a_list_of_values': ['a string', 'not an integer']
}

Validation will of course fail and, given the new dictionary format, inspecting the errors property will return the following:

>>> v = Validator(document, schema)
False

>>> v.errors
{1: 'must be of integer type'}

Let’s look at a document that contains a list of dictionaries instead:

document = {
    rows: [
        {'sku': 1, 'price': 100},
        {'sku': 'hello', 'price': '1'}
    ]
}

Validation errors will be reported like this:

{
    rows: {
        0: {
            'sku': 'must be of string type',
            'price': 'must be of integer type'
        },
        1: {
            'price': 'must be of integer type'
        }
    }

}

By contrasts, on top of my memory, any previous Cerberus release would report:

[
    "rows[0]": 'field "sku" in must be of string type',
    "rows[0]": 'field "price" in must be of integer type',
    "rows[1]": 'field "price" in must be of integer type'
]

As you can easily see, the old implementation was forcing the client to properly parse the list in order to retrieve line number, offending field and the actual error. Overall Im fairly confident that this is an important step forward. Checkout the documentation, which has been updated to reflect the changes.

Get Cerberus 0.5 while its hot.

Sono due le caratteristiche che rendono Cerberus interessante: è facilmente estensibile (la documentazione include un esempio di estensione del sistema di convalida), e non si blocca sollevando un’eccezione al primo problema riscontrato: gli errori, quando presenti, sono tutti riportati in una lista a fine convalida.

Cerberus è un progetto open source tutt’ora in pieno sviluppo, al quale è possibile partecipare. E’ alla base del sistema di convalida impiegato dalla nostra REST API, ma può e dovrebbe vivere di vita propria – per esempio nei vostri progetti. A proposito, la stessa RESTful Web API, quella di cui ho parlato a EuroPython, sarà rilasciata come progetto open source. Stay tuned.