Two things stood out: the PR is in draft, and it includes an AI disclosure:

the fix and the test were created by Claude. I don’t have mongo available or all the necessary python versions for testing, so I’m making this a draft PR so that I can verify all the tests/checks pass before marking it as ready to review.

I’m not at all surprised that Claude was used for the fix (I use it daily myself, to great benefit), but I appreciate the disclosure. A careful review of the code wouldn’t have revealed AI involvement.

The idea of submitting a draft to let the CI run and intervene where needed before marking it as ready for review is also smart — especially since, as the author explains, they can’t run the tests locally.

All in all, an exemplary case of AI use in open source, and welcome news at a time when maintainers are under pressure, often flooded with auto-generated junk PRs (in my own small corner, it has happened to me too). As always, it’s not the tool that’s the problem — it’s who uses it, and how.

But the kicker was just before the session when, at the coffee break, an attendant approached me to shake hands and tell me he had known me for a long time because he discovered and used Eve in a research project he was working on years ago. As I recall, he had attended one of my Eve sessions at EuroPython, maybe in Berlin in 2014. As is often the case, he had no immediate need for my project and shelved it as “interesting, and cool that it’s by an Italian author, but I don’t need really it”. Then, a couple of years later, in a time of need, he recalled that little presentation of mine, and, long story short, Eve ended up being adopted by a big, unnamed corp or research institute. I thanked him from the bottom of my heart and told him it was worth preparing for the talk, waking up at 4 am and travelling to Rome to meet him and hear his story. It’s moments like this that make it all worth it. Also, hearing such a Python story at a .NET event was unexpected.

Nothing gratifies me more than seeing my open-source work appreciated and actively used in the wild.

Special thanks to Fabio Spaziani and the whole DotNetCode team for accomodating for my “daily commuter from afar” special needs.

• FatturaElettronica for .NET 3.6 has been released. It adds support for the upcoming technical specifications v1.9 that are coming into effect on April 1, 2025 (not a joke.) The changelog is here.
• I did some maintenance work on Eve. The CI workflow has been switched to ubuntu-latest from 20.04, as the latter is about to be sunsetted by GitHub. I also merged two pull requests (#1541 and #1547), one of which was long-standing. There isn’t enough material for a release, although the guys who submitted the PRs might think otherwise (if that’s the case, let me know.)
• I ran two DevRomagna meetups. The first one was on OpenTelemetry and was kept by Alessandro Mengoli, of whom I’m very proud (I’ve been encouraging him to start speaking for a long time.) The second was on Linux Containers but before Docker. The speaker was Gabriele Santomaggio, my go-to buddy regarding low-level networking stuff. I found both events to be quite successful and enjoyable. I hope the other attendants agree.
• I finally deactivated my Twitter/X account. I abandoned the platform a while ago and did not miss it. I maintain a presence on Mastodon and Bluesky, mainly to propagate whatever content I post on my website, but don’t count on me following or reading you there (same with LinkedIn.) I’m not active on social media and don’t see myself getting back into them. I’m not interested anymore (and the content there is mostly trash.)
• I went on a nice hike a couple of weeks ago. It was cold and overcast, and that’s why, I suspect, I did not meet a single person the whole day.
• Speaking of hiking, I finally got myself a Garmin inReach Mini 2 device. It’s meant to be used in case of emergency. It allows me to call for help and send sms messages even when no cell signal is available (it uses the Iridium satellite network.) I resisted getting it so far because of the high cost, not so much of the device itself, but the mandatory subscription. Only recently, I found that one can buy Garmin data plans from other vendors, and ProteGear has a nice option to suspend the subscription when not in use. So, I bought the device from Garmin, activated it, and subscribed to ProteGear. It’s looking good so far, and sending sms messages (and emails!) when there’s no cell signal feels like black magic.
• I’ve been reading good books, and I’m grateful for them.
• Well, my dad is not doing well. He’s okay now, but he’s been going through a lot, and more is expected soon. That’s life, I know.
• I am having difficulty making peace with what is happening in the world right now. American friends, I cannot understand how you could re-elect Donald Trump for a second term. It beats me. I am in shock and worried about the geopolitical consequences. I hope time will prove me wrong, but the future looks grim.

• Package info
• Docs

The Eve project has been out for ten years. As said elsewhere, I believe it’s mature and stable enough for most use cases. I consider it done in terms of features, and it is now in ‘maintenance mode’.

I know. I recently stated that Eve was in maintenance mode. All of those considerations still apply, but what can I say? I want Eve 2 out.

So the question is:

Is Eve still maintained?

My reply goes like this:

Hello, yes, Eve is in ‘maintenance mode’, as I call it. I don’t actively develop new features anymore. Still, I am more than willing to code-review and merge relevant pull requests, especially so if they are bug fixes or improvements over existing features.

Long story short, after many years of development, the funding campaign¹ that would have allowed me to keep working on the project failed miserably. People are using the framework worldwide, probably making a profit out of it, but they are not interested in investing in it². The good news is that the framework is, I think, stable and mature enough to meet most needs.

With Cerberus, I was lucky enough to find a worthy, skilled maintainer in the person of @funkyfuture, who’s keeping the project going. I am hoping the same happens with Eve. So, if anyone familiar with the code-base is willing to help, this is your moment to come forward. Maybe start with triaging open tickets, maybe pull-request for some low-hanging fruits, and then challenge yourself with more demanding tasks, like updating Eve to the latest version of Cerberus.

I had so many things planned for Eve. Hopefully someone will join the ranks, maybe take the helm, and help bring it to the next level.

I really feel like I should put more effort into Eve, Cerberus and satellite projects Eve-Swagger, Flask-Sentinel, Eve.NET, etc. I love working on these projects and I know a lot of people rely on them. Also, I have big ideas that I would like to play with. At this point in time however, I cannot afford allocating more time to not-paying-the-bill activities.

Well, I went and looked at the possibilities, and finally settled down on a model which would probably be best defined as a blend between the Vue.js and the Django REST framework models.

Effective now, the Eve REST framework, Cerberus, and the whole eco-system are collaboratively funded projects. If you run a business and are using either Eve or Cerberus a revenue-generating product, it would make business sense to sponsor their development: it ensures the project that your product relies on stays healthy and actively maintained. Individual users are also welcome to make either a recurring pledge or a one-time donation if Eve and/or Cerberus have helped you in your work or personal projects. Every single sign-up makes a significant impact towards making Eve possible.

Rest assured, Eve continues to be open-source and permissively licensed. It is also maintained and developed in my free time, and by the awesome community which has matured around it. I firmly believe, however, that it is in the commercial best-interest for users of the project to invest in its ongoing development (make sure you check the exciting Things I Would Be Working On section on the Patreon page).

To join the backer ranks, check out Eve campaign on Patreon, or donate via PayPal.

Thank you, Nicola.

Join the newsletter to get an email alert when a new post surfaces on this site. If you want to get in touch, I am @nicolaiarocci on twitter.

I am glad to report that the Eve-SQLAlchemy community extension, which allows SQL databases to serve as Eve backends, has seen a surge of activity around it. There is a new maintainer, Dominik Kellner, and a bunch of active contributors. Work is being done to align the extension with latest Eve release. If interested, please join the efforts there. Speaking of community extensions, only a few days ago Eve-Elastic version 2 has been released too.

Next major release will of course be version 0.8 and it should come relatively soon. It will be focused on adding Cerberus 1.0 support. That will add superpowers and more flexibility to areas like data validation and API endpoints definition. Work on adding Cerberus 1 to Eve has actually been ongoing. In fact, we were ready to release the feature with the current release, but I decided against it as I wanted to deliver all the new features before the inevitable breaking changes that will come with Cerberus 1. The idea is to let most users take advantage of new features with as little effort as possible. They will then have the option to bite the bullet and upgrade to 0.8 at their own pace.

The current release has been cooking for a long time, way more than I anticipated. I have simply been too busy with normal day work, the kind of activity that pays the bills. That is good for me I guess, but not so much for the project. I really feel like I should put more effort into Eve, Cerberus and satellite projects Eve-Swagger, Flask-Sentinel, Eve.NET, etc. I love working on these projects and I know a lot of people rely on them. Also, I have big ideas that I would like to play with. At this point in time however, I cannot afford allocating more time to not-paying-the-bill activities. Rest assured the project is not going stale. Worse case scenario, nothing changes and it will just proceed at current pace. But I have been wondering what would allow me to put more into it.

I am particulary fond of the collaboratively funded project model adopted by the Django REST Framework, and I wonder if it would work for Eve too. Of course I know that Eve is a smaller project with a smaller ecosystem. I also know however that many companies and institutions, big and small are using Eve in production and with good results. In the past I even did the occasional consulting gig for them. Maybe some would be willing to help with some form of financing and/or sponsorship, adopting the project in exchange for some visibility and, more importantly, project growth?

If you have some advice to share on this topic, please do so. I am very hesitant and unsure about this whole thing. Part of me just wants to keep going the way it’s always been, as with any form of financing also comes more responsability. The other half of me wants to work more on this thing, make it bigger. Eventually we will get there but, unless something changes, it will take quite some time.

By the way, next March I am speaking about RESTful Web Services at Codemotion Rome 2017. Presentation will include a Eve live demo, so if you are interested in Eve and/or want to see me fail miserably and blow it up on stage, join me there. Tickets are still available, talk will be in English, and March is a great season to visit Rome. See ya’ll there.

If you want to get in touch, I am @nicolaiarocci on Twitter.

• PostAsync() supports bulk inserts
• DeleteAsync() supports bulk deletes
• GetAsync() has a softDelete option to include soft-deleted documents with query results
• GetAsync() has a rawQuery option to pass raw Eve queries to the server
• BearerAuthenticator class adds support for Bearer Token authentication

Several fixes made it into this release and, most importantly, I switched to portable Profile259 which offers support for the following platforms: Xamarin.iOS, Xamarin.Android, WinPhone 8.1, Windows 8, .NET 4.5+. Yes, that means no more support for .NET 4.0. See the changelog for details.

Eve.NET v0.2 is on NuGet so you can install it either from Visual Studio or with the Package Manager console. Enjoy!

If you want to get in touch, I am @nicolaiarocci on Twitter.

Eve’s schema definitions are full of features, but can take a good amount of time to create when dealing with lots of complex resources. From our experience, it’s often helpful to describe an endpoint in JSON before creating it as an Eve schema. This allows you to make quick decisions about the structure of your entities without spending time moving schema code around. This is where EveGenie comes in.

EveGenie saves time by creating a ready to use Python Eve schema from a very simple JSON representation of a set of resources.

Interested? Good! Matt Tucker (@ultimateboy) has a getting started with EveGenie article up for you, so make sure to check it out.

Work on v0.7, which will include MongoDB Aggregation Framework support (docs) and many other new features, continues steadily.

But what is Swagger, and why is it useful to your RESTful API? With a Swagger-enabled API you can get interactive documentation, client SDK generation and discoverability, all for free. From Swagger website:

Swagger is a simple yet powerful representation of your RESTful API. With the largest ecosystem of API tooling on the planet, thousands of developers are supporting Swagger in almost every modern programming language and deployment environment. With a Swagger-enabled API, you get interactive documentation, client SDK generation and discoverability.

When Eve-Swagger is installed and properly configured your Eve API exposes a special /api-docs endpoint which returns a 100% swagger-compliant JSON response. This JSON can then be processed by the swagger tools like Swagger UI and Swagger Editor. Here we can appreciate Swagger UI providing API documentation out of the box (click to zoom):

Like most Eve extensions Eve-Swagger comes as a standard Flask Blueprint and, as such, using it is very simple:

Once the blueprint has been registered all you have to do is add a SWAGGER_INFO node to your settings. It maps to a swagger infoObject and contains some simple, human readable information. The extension will also scan your endpoint settings to figure out the API graph and document it. Of course, in order to not to clutter your launch script, you could (and probably should) set SWAGGER_INFO in your configuration file.

Installation is super easy:

This is just version 0.0.2 so many parts might still be moving, but you are encouraged to start using it right away. Also if you appreciate this extension please let me know by either starring it on GitHub or with a tweet or email, so I will know that I should really try hard and allocate more time to this project. As always, feel free to contribute via pull request!

If you want to get in touch, I am @nicolaiarocci on Twitter.

Work on v0.7 is also ongoing. It will include new features such as support for the MongoDB Aggregation Framework (docs) and a few breaking changes, so you might want to check it out in advance.

Special thanks to Arnau Orriols, Cyril Bonnard, Hamdy, Luca Di Gaspero, Manquer, Nick Park, Patrick Decat, Prayag Verma, Ralph Smith, Stratos Gerakakis, Valerie Coffman and Wei Guan who contributed to this release.

How do I setup endpoints without any binding to a data entity, just connected to a custom method?

They would like to call something like /mycustomendpoint and get the response from a method they have defined somewhere in the Python sources. The standard endpoint-entity mapping provided by Eve covers 90% of their needs, but sometimes they just need endpoints that have nothing to do with data entities.

This is very easy to achieve. I’m writing it down so I can point people to this post in the future.

Eve is Flask

Eve is a Flask application and I really mean it since it is, in fact, a Flask subclass. Eve adds out-of-the-box RESTful capabilities to the Flask micro-framework. Most of the time you will be happy with just Eve’s own features but remember, the full Flask (and Werkzeug at a lower level) features set is also part of your arsenal.

Whatever you can do with Flask, you can do with Eve

So yes, of course you can easily set some functions to be invoked when a custom endpoint is hit with a request. The following snippet is a slightly modified version of Flask’s very own Quickstart tutorial:

By decorating the hello_world method with the route decorator we added a new endpoint to the application. Each time a request to /hello comes in, it will be routed to our custom method. Of course the regular API endpoints (either defined in settings.py, passed as a dict at launch, or registered live by calling the register_resource method) will also be available.

If you have an Eve authentication class securing regular API endpoints, you can apply it to your custom endpoint too. Just add a requires_auth decorator:

Say that you saved the above snippet as run.py and also have your RESTful endpoints properly configured. Launch the script:

You can now point your browser to http://localhost:5000/hello/ and enjoy the application greeting. Or you can consume any other configured API endpoint.

As all of Flask features are at your fingertips you might as well opt for registering a blueprint, which is what the awesome Eve-Docs extension is doing in order to provide a static HTML /docs endpoint on top of any Eve powered API.

Collection rules

First up is the new set of anyof, allof, noneof and oneof validation rules. anyof allows you to list multiple sets of rules to validate against. The field will be considered valid if it validates against one set in the list. For example, to verify that a property is a number between 0 and 10 or 100 and 110, you could do the following:

>>> schema = {
...     'prop1': {
...         'type': 'number',
...         'anyof': [
...             {'min': 0, 'max': 10}, 
...             {'min': 100, 'max': 110}
...         
...         ]
...     }
... }

>>> doc = {'prop1': 5}
>>> v.validate(document, schema)
True

>>> doc = {'prop1': 105}
>>> v.validate(document, schema)
True

>>> doc = {'prop1': 55}
>>> v.validate(document, schema)
False

allof is the same as anyof, except that all rule collections in the list must validate. Same pattern applies to noneof (no rule in collection must validate) and oneof (only one rule in collection must validate).

Type coercion

Type coercion allows you to apply a callable to a value before any other validators run. The return value of the callable replaces the new value in the document. This can be used to convert values or sanitize data before it is validated.

>>> v = Validator({'amount': {'type': 'integer'}})
>>> v.validate({'amount': '1'})
False

>>> v = Validator({
...     'amount': {
...         'type': 'integer', 
...         'coerce': int
...     }
... })
>>> v.validate({'amount': '1'})
True
>>> v.document
{'amount': 1}

>>> to_bool = lambda v: v.lower() in ['true', '1']
>>> v = Validator({
...     'flag': {
...         'type': 'boolean', 
...         'coerce': to_bool
...     }
... })
>>> v.validate({'flag': 'true'})
True

>>> v.document
{'flag': True}

Properties (keys) validation

propertyschema is the counterpart to valueschema (also new, it replaces the now deprecated keyschema) and validates the keys of a dictionary.

>>> schema = 'a_dict': {
...     'type': 'dict', 
...     'propertyschema': {
...         'type': 'string', 
...         'regex': '[a-z]+'
...     }
... }

>>> document = {'a_dict': {'key': 'value'}}
>>> v.validate(document, schema)
True

>>> document = {'a_dict': {'KEY': 'value'}}
>>> v.validate(document, schema)
False

List of types

The type rule can now be a list of types, to allow for different type of values for the field.

>>> v = Validator({
...     'quotes': {
...         'type': ['string', 'list']
...     }
... })

>>> v.validate({'quotes': 'Hello world!'})
True

>>> v.validate({'quotes': ['Do not disturb my circles!', 
...                        'Heureka!']})
True

>>> v = Validator({
...     'quotes': {
...         'type': ['string', 'list'], 
...         'schema': {'type': 'string'}
...     }
... })

>>> v.validate({'quotes': 'Hello world!'})
True 

>>> v.validate({'quotes': [1, 'Heureka!']})
False

>>> v.errors
{'quotes': {0: 'must be of string type'}}

And there is more so make sure you check the changelog before upgrading. No breaking changes but there’s at least one deprecation, as mentioned.

Fun fact: Cerberus is currently getting 3x the downloads of his sister project Eve, the REST API Framework for which the tool was originally conceived. Interesting if you consider that Eve is featuring 10x the GitHub stars. Fun, but not really surprising since Cerberus probably has a broader audience.

Special thanks to Frank Sachsenheim, Tobias Betz, Brett and C.D. Clark III for their valuable contributions to this release.

If you want to get in touch, I am @nicolaiarocci on Twitter.

Michael has a great lineup ready for the next episodes, so make sure you subscribe to his show. It is also available on iTunes.

If you want to get in touch, I am @nicolaiarocci on Twitter.

So, suppose we set the following validation schema:

schema = {
  'test_field': {
    'dependencies': [
      'a_dict.foo', 
      'a_dict.bar'
    ]
  },
  'a_dict': {
    'type': 'dict',
      'schema': {
        'foo': {'type': 'string'},
        'bar': {'type': 'string'}
      }
  }
}

Then, we can validate a document like this:

>>> v = Validator(schema)
>>> document = {
      'test_field': 'foobar', 
      'a_dict': {'foo': 'foo'}
    }

>>> v.validate(document, schema)
False

>>> v.errors
{'test_field': "field 'a_dict.bar' is required"}

This release will not work with Eve 0.5.2 or less so if you want to use Cerberus 0.8.1 with Eve make sure you upgrade to Eve 0.5.3, released today. By the way, yesterday we hit 2K stargazers and 70 contributors on the Eve repository, quite the milestone!

If you want to get in touch, I am @nicolaiarocci on Twitter.

So what is the Resource Owner Password Credentials Grant pattern? According to the official RFC:

The resource owner password credentials (i.e., username and password) can be used directly as an authorization grant to obtain an access token. The credentials should only be used when there is a high degree of trust between the resource owner and the client (e.g., the client is part of the device operating system or a highly privileged application).

A typical use case would be when the remote API maintainer controls the client application. Say that you have a REST API being consumed by your own iOS, Android, WinPhone, desktop or web applications. Users register to your service by creating their accounts. Then, they consume the service using your applications.

Even though this grant type requires direct client access to the resource owner credentials, the resource owner credentials are used for a single request and are exchanged for an access token. This grant type can eliminate the need for the client to store the resource owner credentials for future use, by exchanging the credentials with a long-lived access token or refresh token.

So let’s get back at the proprietary client scenario. The user has just installed the application on his/her device. On first run he/she is asked to provide his/her username and password. These are sent to the OAuth2 server through a SSL/TLS encrypted channel. If the user is registered for the service and the client id, which has also been sent along with the user credentials, is recognised, then the server sends back a valid access token. Otherwise responds with a 401 Unhautorized.

From now on the application will only be using the access token for all requests until, eventually, the token expires. If that happens, the cycle repeats. Please note that in this scenario the client does not need to (and probably should not) store the username and/or password on the local cache. The User Credentials pattern usually relies on long lived tokens so asking again for username and password is not a big deal (you could also opt for permanent, revokable tokens.)

Flask-Sentinel provides two endpoints: one for token creation which defaults to /oauth/token and is consumed by clients, and another for users and clients management, accessible at /oauth/management:

Note that the password is hashed and salted on the server, so no plain password is stored on either sides of the channel.

Only existing users and recognised clients will be provided an access token. A typical token request would be as follows:

$ curl -k -X POST -d \
  "client_id=9qFbZD4udTzFVYo0u5UzkZX9iuzbdcJDRAquTfRk&
   grant_type=password&
   username=jonas&
   password=pass" \
   <api_url>/oauth/token

And the response would be like so:

{
    "access_token": "NYODXSR8KalTPnWUib47t5E8Pi8mo4", 
    "token_type": "Bearer", 
    "refresh_token": "s6L6OPL2bnKSRSbgQM3g0wbFkJB4ML", 
    "scope": ""
}

The client can now use the access token to access protected API endpoints:

$ curl -k -H \
  "Authorization:Bearer NYODXSR8KalTPnWUib47t5E8Pi8mo4" \
  <api_url>/endpoint

200 OK

There are a number of configuration options of course, for example you can change the url of token and management endpoints, set token expiration, setup database connection, stuff like that. Redis is used to store active access tokens, allows for optimal performance.

While you can use Flask-Sentinel to extend an existing API, you might want to instance it as a stand-alone Flask application to optimize for scalability. You would end up with a distributed network of three different (micro)services: the OAuth2 server, the resouces API with protected endpoints as needed, and the Redis instance bridging the two. Check out the project page on GitHub for details.

Of course Flask-Sentinel integrates seamlessly with any Eve powered REST API. Check out the Eve-OAuth2 sample, a fork of the original Eve-Demo project with a couple protected endpoints and a static HTML page, also protected.

The project is very new and lacks a few little things, but I suspect it is already usable even at this stage. Enjoy!

If you want to get in touch, I am @nicolaiarocci on Twitter.

As with all Eve extensions, once installed with

$ pip install eve-sqlalchemy

using Eve-SQLAlchemy is very simple:

from eve import Eve
from eve_sqlalchemy import SQL

app = Eve(data=SQL)
app.run()

On a fresh virtualenv (of course you are using virtualenvs, right?) the install will also setup Eve and all its dependencies for you. For a complete tutorial you can visit the Eve-SQLAlchemy Support Website.

If you have been following the Eve development you probably know that the original plan was to release SQL support within the Eve core, alongside with the native MongoDB layer. In fact, in a long rant on open source and code responsibility I took the chance to outline the path forward.

However, if you check the discussion which spread in the comments below that post you can see how both my friend and fellow MongoDB Master Flavio and the main contributor to the sqlalchemy branch Andrew were suggesting that a plugin system for data layers might be more appropriate. Not coincidentally that was easy to accomplish since the Eve data layer system was designed precisely with that goal in mind, right from the start.

So, to make a long story short, in agreement with Andrew who volountereed for the task, we changed plans. He refactored the sqlalchemy branch into a separate repository… and then the official Eve SQL extension was born. This approach also allows the extension and Eve to develop asynchronously which is ideal for a faster release cycle.

I like this setup so much that, if in time it shows to be as good as I think it will be, I will probably want to experiment with taking the MongoDB layer out of Eve core too. So with Eve 0.6 MongoDB support would still be preinstalled along with core (so nothing changes as first time user experience) but, being a separate package, users could easily switch to a different driver as they need, still keeping their Eve setup as light as possible.

So if you are willing to help with the development of a mature SQL driver please feel free to join Andrew and other contributors. Your help is more than welcome. Allow me thank them all individually, for the work they’ve been doing on the SQL code is absolutely amazing: Andrew Mleczko, Tomasz Jezierski (Tefnet), Bruce Frederiksen, Jacopo Sabbatini and Peter Zinng.

Get Eve-SQLAlchemy v0.1 while it’s hot!

If you want to get in touch, I am @nicolaiarocci on Twitter.

$ curl -I charliehebdo.fr
Date: Mon, 12 Jan 2015 15:56:13 GMT
Content-Type: text/html; charset=iso-8859-1
Content-Length: 221
Connection: keep-alive
Location: http://www.charliehebdo.fr/index.html
Vary: Accept-Encoding
X-Charlie-fr: Je suis toujours Charlie.
X-Charlie-en: I am still Charlie.
X-Charlie-es: Todavia soy Charlie.
X-Charlie-de: Ich bin immer Charlie.
X-Charlie-ro: Inca sunt Charlie.
X-Charlie-cz: Jsem stale Charlie.

I find this to be a great way for us techies to somehow contribute and show support for the ongoing anti-terrorism campaign. So if you feel like doing it here is a quick rundown on how to serve custom headers with your Eve-powered REST API. It is actually a very easy task to accomplish.

Eve provides its own set of callback hooks but since we don’t need fine-grained control (we are good with including the new header with all responses), this time we will leverage Flask’s native callback system instead.

@app.after_request
def after_request(response):
    response.headers.add('X-Charlie', 'Je Suis Charlie.')
    response.headers.add('X-Ahmed', 'Je Suis Ahmed.')
    return response

A simple run script would then look something like:

from eve import Eve
app = Eve()

@app.after_request
def after_request(response):
    response.headers.add('X-Charlie', 'Je Suis Charlie.')
    response.headers.add('X-Ahmed', 'Je Suis Ahmed.')
    return response

if __name__ == '__main__':
    app.run()

All API responses now include the custom headers:

$ curl http://localhost:5000
HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 141
Server: Eve/0.5 Werkzeug/0.9.6 Python/2.7.8
Date: Mon, 12 Jan 2015 15:39:36 GMT
X-Charlie: Je Suis Charlie.
X-Ahmed: Je Suis Ahmed.

If you want to see it in action try sending a GET request with curl, Postman or similar tool to the Eve demo instance (source). Remember Eve is a Flask subclass so whatever works with Flask generally works with Eve too.

If you want to get in touch, I am @nicolaiarocci on Twitter.

Written and maintained by the same author of the Eve Framework itself, Eve.NET is delivered as a portable library (PCL) and runs seamlessly on .NET4, Mono, Xamarin.iOS, Xamarin.Android, Windows Phone 8 and Windows 8. We use Eve.NET internally to power our iOS, Web and Windows applications.

And now please forgive me while I take you on a quick motivational, strongly opinionated, probably boring and overzealous detour

Python and C# Motherfucker, Do You Speak It? *

If you don’t then well, you should consider doing just that. Especially so these days, with C# and the whole .NET platform going open source and cross platform. Actually, thanks to technologies like Mono and Xamarin (also based on Mono) we have been able to run C# code on all major platforms for a while: iOS, Android, OSX, Linux, Windows, Windows Phone, you-name-it. And what’s even better, on mobile platforms C# is compiled to native so performance is a non-issue.

What makes C# a perfect match for a REST API is precisely that: it’s ubiquity. You can have a portable client library like Eve.NET which runs seamlessly (and untouched) on all these mobile desktop and server platforms.

If you already have a Web Service running on Eve and are now looking at the client side of things then well, you should consider C# and Eve.NET because you know, you can’t have a native iOS app written in Python anyway. On the other hand if you are a C#/.NET shop consider this: you can have a powerful Web Service up and running in minutes (even if you don’t grok Python yet – trust me on that) and a complete out-of-the-box, cross platform client library ready to go with it.

A few years ago I gave a talk about leaving my Comfort Zone (**) and getting out of my .NET nest. That opened the path to Python, MongoDB, Node and so many other technologies and best practices and, what’s even more relevant, most of what I learned down that path I ended up using in my .NET projects in the long run. But the point I’m trying to make is don’t be afraid of change, it can only improve your skills making you a better all-around professional programmer.

Never mind the naysayers. Polyglot is the way.

(*) I’m paraphrasing Zed A. Shaw’s Programming, Motherfucker. You should get a T-Shirt by the way. They are so cool.

(**) Since then there have been plenty of talks on the same subject. Mine was an 5 minutes ignite talk and was in Italian, so you probably don’t care (it’s on my Talks page anyway).

Back to business now.

Usage

Initialization is as simple as instantiating a new client and providing it with the web service entry point.

// Initialize and set API address.
var client = new EveClient();
client.BaseAddress = new Uri("http://api.com");

// Set target resource for subsequent requests.
client.ResourceName = "companies";

Getting a list of objects is pretty straightforward:

// List<T>
companies = await client.GetAsync<Company>();

// Objects changed since DateTime.
var ifModifiedSince = DateTime.Now.AddDays(-1);
companies = await 
  client.GetAsync<Company>(ifModifiedSince);

// Refresh an object
company = await client.GetAsync<Company>(company);

// Raw, conditional GET request
var companyId = "507c7f79bcf86cd7994f6c0e";
var eTag = "7776cdb01f44354af8bfa4db0c56eebcb1378975";

company = await 
  client.GetAsync<Company>("companies", companyId, eTag);

Other CRUD operations are easy too:

// Create (POST)
company = await 
  client.PostAsync<Company>(
    new Company { Name = "MyCompany" }
  );

// Update (PUT)
company.Name = "YourCompany";
var result = await client.PutAsync<Company>(company);

// Delete (DELETE)
var result = await client.DeleteAsync(company);

As you can see all methods are Async and return full object instances parsing JSON in and out on for you. If you need more control you can query the HttpResponse property to inspect the original JSON, response headers, status code, etc.

Behind the scenes data integrity and concurrency control are transparently handled so for example PutAsync performs a If-Match check and same happens with DeleteAsync. On PostAsync new objects are returned with fresh meta-fields such as ETag, DateCreated, DateUpdated and UniqueId. Mapping object properties to JSON fields and Eve metafields is just a matter of setting the JSonPropertyAttribute and RemoteAttribute:

public abstract class BaseClass
{
  [JsonProperty("_id")]
  [Remote(Meta.DocumentId)]
  public string UniqueId { get; set; }

  [JsonProperty("_etag")]
  [Remote(Meta.ETag)]
  public string ETag { get; set; }

  [JsonProperty("_updated")]
  [Remote(Meta.LastUpdated)]
  public DateTime Updated { get; set; }

  [JsonProperty("_created")]
  [Remote(Meta.DateCreated)]
  public DateTime Created { get; set; }   
}

public class Company : BaseClass
{
  [JsonProperty("n")]
  public string Name { get; set; }

  [JsonProperty("p")]
  public string Password { get; set; }
}

For a complete list of usage examples see the README

Current Status and License

Eve.NET is a Nicola Iarocci and Gestionali Amica open source project distributed under the BSD license. It is a work in progress but it’s pretty stable already. It is available on NuGet as a pre-release package. The test suite can be run against a local Eve instance or, if you don’t grok Python yet, you can use a free test instance which is available online, see the README for details.

Did you read this far? Well thank you! And please, consider showing your appreciation by starring the project on GitHub. It feels so lonely out there.

If you want to get in touch, I’m @nicolaiarocci on Twitter.

At some point during the debate, I argued that when a maintainer merges a pull request, he (or she) implicitly agrees on being responsible for that code. That seemed to strike some surprise into most attendees.

Yes, in theory any contributor is just a ping away so in case trouble arises one can always reach him, or her. Unfortunately this is not always the case. While some contributors will fully embrace your project and keep helping after their initial contribution, truth is that a good number of them will just move on, never to be seen again.

There’s nothing wrong with that. Not everyone has spare time to devote to your project, which is perfectly fine. It is natural for most people to contribute what they need to a project and then go on their way. Actually, one could argue that most projects grow and prosper precisely thanks to this kind of contributions.

However this attitude can become an incumbent when big chunks of code get merged, usually as new (big) features. Good practices advice against merging huge pull requests. In fact they are rare and when they do come, it is a good idea to ask for them to be split into smaller ones. But no matter the format, a huge contribution is likely to hit a project one day or another. It might even come from more than one person: a disconnected and distributed team of contributors who have been patiently tinkering on a side branch or a fork for example. When this happens, and provided that the contribution is worth merging, the maintainer should then ask him/herself the obvious question: am I willing to deal with the consequences of this merge?

In fact this is the exact scenario I’m dealing with right now. The Eve project has always been a MongoDB shop. Since its inception however people have been asking if (when) SQL support was going to be added. I think we were in Alpha when someone started contributing SQL code. Over time I ended up devoting a specific branch to this feature. Several people have been hacking at it since then, and what a splendid job they did! To say that it was a huge commitment is an understatement but, in time, they managed to deliver. So now we have this awesome sqlalchemy branch which is feature complete and ready to be merged ahead of the new Eve release. We’re talking 4K+ lines of code and 44 files changed. Code quality is not under discussion. I know that several companies and individuals have been using that branch in production with good success, even when it still was at its early stages.

This is very exciting as adding SQL support has a good chance to greatly improve the audience of the project. At the same time however, I’m a little bit nervous if not scared, and I have been for a while. Am I ready to deal with the consequences of this supermerge? Inevitably SQLAlchemy tickets will be opened and Stack Overflow questions will be asked. SQL-related pull requests will come in and mailing list posts will flock. To be honest I don’t think I can handle that, let alone allocate more of my free time to the project. Also, I’m not very confident with SQLAlchemy itself so I would not be the best person deal with that code anyway. In the recent past while discussing SQLAlchemy support on the mailing list, I have been very clear about my concerns, so much that I probably scared a few people away. What worries me the most, I suspect, is the risk of new code becoming stale one day or another. In time that would probably impact the reputation of the whole project.

To think about it, we already had something similar happen in the past, although for a smaller feature. The Document Versioning pull request, contributed by the amazing Josh Villbrandt, had been daunting me with similar thoughts. New code was going to be be quite intrusive, adding a good deal of complexity to an otherwise relatively simple codebase. Everything went amazingly well though. Josh is still an active contributor. He helps with improving his own feature and, even more importantly, other contributors are now helping with Document Versioning as we speak. Overall, the Eve project as a whole as been enjoying a growing number of skilled contributors and adopters. It’s been a joy to see people commenting on open tickets, offer support on the mailing list and even on Stack Overflow. So that should be encouraging.

So here I sit, with 4K LOCs ready to be merged. What do I do with them? I considered a few options. One is leaving the SQL feature in a separate branch. Another is to ask the team to refactor the whole thing into an external extension (we have a few already). By doing any of these Eve-core would remain MongoDB only and I could keep managing it on my own. But then again, none of these options would add native SQL support to Eve. Also, an extension or a branch would run even a greater danger of becoming stale.

At some point I guess even mildly successful projects like Eve have to decide wether they want to outgrow their author. I strongly believe that growing and trusting communities is all that open source is about. You release your work out there and, even at that early stage, you are already entrusting people. You trust that they will take notice and then that they will validate your project (or not). Eventually, someone will review your code, adopt it and, in time, contribute. The project then grows up to a point where its community becomes so predominant that you, as the author and maintainer, just have to let some control go.

So yes, SQL support is coming to Eve, and as a native feature. I trust that the contributors to the SQLAlchemy backend will stay around and, if they won’t, that someone else will stand up and take the torch. I am also confident that the community as a whole will adopt the feature, make it grow and well… we’ll see what happens next.

If you want to get in touch, I’m @nicolaiarocci on Twitter.

What is the OpLog?

The OpLog is a special resource that keeps a record of operations that modify the data stored by the API. Every POST, PATCH, PUT and DELETE operation can eventually be recorded by the oplog.

At its core the oplog is simply a server log, something that’s always been on the Eve roadmap. What makes it a little bit different is its ability to be exposed as a read-only API endpoint. This would in turn allow clients to query it as they would with any other standard endpoint.

Every oplog entry contains few important informations about the document

involved with the edit operation:

• URL of the endpoint hit by the operation.
• Kind of operation performed.
• Unique ID of the document.
• Date when the document was updated.
• Date when the document as created.
• User token, if User Restricted Resource Access is enabled for the endpoint

Like any other API-maintained documents, oplog entries also contain:

• Unique ID
• ETag
• HATEOAS meta fields, if enabled.

A typical oplog entry would look something like this:

{
    "o": "DELETE", 
    "r": "people", 
    "i": "542d118938345b614ea75b3c",
    "_updated": "Fri, 03 Oct 2014 08:16:52 GMT", 
    "_created": "Fri, 03 Oct 2014 08:16:52 GMT",
    "_id": "542e5b7438345b6dadf95ba5", 
    "_etag": "e17218fbca41cb0ee6a5a5933fb9ee4f4ca7e5d6"
    "_links": {...},
}

To save a little storing space field names have been shortened when possible (what can I say, I’m a MongoDB guy): o stands for operation, r stands for resource, i stands for unique ID and c stands for changes. Other keys are defined by the configuration settings, and their default names are shown here.

How is the oplog operated?

Three new settings keywords are available:

• OPLOG

  Sets the oplog name and defaults to oplog. This is the name of the collection on the database and also the default url for the oplog endpoint.

• OPLOG_METHODS

  A list of HTTP methods for which oplog entries are to be recorded. Defaults to ['DELETE'].

• OPLOG_ENDPOINT

  Set it to True if an oplog endpoint should be made available by the API. Defaults to False.

• OPLOG_AUDIT

  If enabled, IP addresses and changes introduced with PATCH and PUT methods are also logged. Defaults to True.

So by default the oplog is stored on a conveniently named oplog collection, it only stores informations about deleted documents.

Since the eventual oplog endpoint is a standard API endpoint, if it is enabled the API maintainer can also fiddle with the endpoint settings as he/she would do with any other Eve endpoint. This allows for setting custom authentication (you probably want this resource to be only accessible for administrative purposes), changing the url, etc. Just add an oplog entry to the API domain, like so:

'oplog': {
    'url': 'log',
    'auth': my_custom_auth_class,
    'datasource': {'source': 'myapilog'}
}

Note that while you can change most settings, the endpoint will always be read-only, so setting either resource_methods or item_methods to something else than ['GET'] will serve no purpose. Also, unless you need to do so, adding an oplog entry to the API domain is not required as it will be added automatically for you.

Why the OpLog?

Clients have always been able to retrieve changes by simply querying an endpoint with a If-Modified-Since request. So why do we need an operations log? Of course because server-side logging is cool, and so is auditing, but it’s not only about that.

Single entry point for all API updates

From the client perspective and for most use cases logging inserted, edited and replaced documents is probably a waste of both space and time, and this is the main reason why only DELETE operations are logged by default. However, I believe there are scenarios where remote access to a full activity log can be useful.

Imagine an API which is accessed by multiple apps (say phone, tablet, web and desktop applications) and all of them need to stay in sync with each other and the server. Instead of hitting every single endpoint with a IMS request they could just access the oplog. That’s a single request vs several, and since the oplog itself is a standard API endpoint, they could also perform IMS requests against it for optimal gains:

Server, please send me all the changes occurred to the API since my last access. Sincerely, Your Client.

Again this is not always the best approach a client could take. Sometimes it is probably better to only query for changes when they are really needed, but it seems cool to have both approaches available (and remember, the oplog endpoint is disabled by default).

Fixing 304s

And then there are deleted documents which are a completely different beast. With no oplog we would have no way to tell if and when any document has been deleted, let alone inform clients about that. Actually, there is an open ticket precisely about this, and it’s been sitting there for a while.

When a If-Modified-Since request is received, the API is expected to respond with a 304 Not Modified status if no changes occurred, so that clients can conveniently fallback to cached data. Up to version 0.4 (the official release at the time of this writing) Eve has been doing exactly that, with one caveat: missing documents were being ignored as, in the contest of the IMS request, there was no way to know about them.

The operations log will allow Eve powered APIs to take deleted documents into account, returning perfectly proper 304 codes as needed. The impact on performance should be minimal as we will only query the oplog when and if no changes have been detected on the target collection.

This solves only one half of the problem however. What happens when a IMS request comes in and deleted documents are found in the backlog? How do we report them back to the client? Three options come to mind which would address this scenario:

1. Respond with a 200 OK and a the usual “changes since IMS date” payload, which might happen to be empty if only deletions occurred in the time window. The client can then go and query the oplog endpoint with the same IMS date, finally getting the list of deleted documents IDs.

2. Include deleted documents IDs in the standard payload (within the _items list), maybe with a deleted status tag. This status tag is something new though, and for consistency we should probably add it to other objects in the payload.

3. Add support for a new _deleted meta field in resource payloads. When deleted documents are spotted in the backlog the response payload will include them in their own list. Something like this:

First option is so bad I should probably not be listing it at all. It would take two roundtrips to get the whole update down. Also, it would kind of force API maintainers to open the oplog endpoint to their clients.

I’m not convinced #2 would be a good idea either, as objects in the items list would not be homogeneous anymore and we would have to add support for a new meta field anyway (the status tag).

Option #3 on the other hand looks quite good to me. It does not require multiple requests to handle the case of deleted documents on IMS requests, and it is still easy and clean for clients to process. I am going to go with #3 unless feedback is negative and for good reasons, so let your opinion be heard.

Closing concerns

I am slightly concerned about the performance impact, not so much on IMS requests but rather on write operations, especially when a complete, all-operations log is being recorded.

In MongoDB world OpLog is probably an ideal candidate for a capped collection. I’m not entirely convinced about that though, as by its nature a capped collection is bound to lose data over time, which again might lead to inaccurate 304 handling.

I am not implementing the OpLog at the data layer level however. It is a business layer feature to let other engines take advantage of it. Nothing prevents the MongoDB admin from setting the oplog as a capped collection anyway. Also, keep in mind that like all other resources maintained by the API, indexes are not handled by Eve itself so you will have to do your homework in that field too.

So here you have it. I’m currently done on both configuration and logging parts and will be working on 304 handling and response payloads in the coming days so that all of this can be included with next version 0.5. Be warned that at the moment the develop branch has no support for IMS requests on resource endpoints. It’s been disabled to avoid providing clients with inaccurate responses (see the ticket above).

If you have any comment or feedback to provide, please let me know in the comments below. I’d really appreciate that.

PS. In case you are wondering yes, the Eve OpLog is heavily inspired by the awesome MongoDB OpLog.

My REST APIs for Humans™ talk was given in a fully packed Python DevRoom, which granted a lot of smart questions both in the final QA session and later during the day. Thanks everybody for the great feedback!

What we had before was basically a list of human-readable errors. Each item in the list, while perfectly fine for human reading, wasn’t really ideal for algorithmic parsing. Why would you want to parse the errors with an algorithm? A common case would be when your client is using business objects to represent API resources (think a client-side ORM), and would have a hard time binding validation errors to the objects themselves.

So for example, previously we had:

[
    "min value for field age is 10",
    "value of field name must be of string type"
]

With Cerberus 0.5+ we now have:

{ age: min value is 10, name: must be of string type }

Let’s look at more complex structures, like lists. Imagine we have a schema defined like this:

schema = {
    'list_of_values': {
        'type': 'list',
        'items': [{'type': 'string'}, {'type': 'integer'}]
    }
}

And this is the document we want to validate against it:

document = {
    'a_list_of_values': ['a string', 'not an integer']
}

Validation will of course fail and, given the new dictionary format, inspecting the errors property will return the following:

>>> v = Validator(document, schema)
False

>>> v.errors
{1: 'must be of integer type'}

Let’s look at a document that contains a list of dictionaries instead:

document = {
    rows: [
        {'sku': 1, 'price': 100},
        {'sku': 'hello', 'price': '1'}
    ]
}

Validation errors will be reported like this:

{
    rows: {
        0: {
            'sku': 'must be of string type',
            'price': 'must be of integer type'
        },
        1: {
            'price': 'must be of integer type'
        }
    }

}

By contrasts, on top of my memory, any previous Cerberus release would report:

[
    "rows[0]": 'field "sku" in must be of string type',
    "rows[0]": 'field "price" in must be of integer type',
    "rows[1]": 'field "price" in must be of integer type'
]

As you can easily see, the old implementation was forcing the client to properly parse the list in order to retrieve line number, offending field and the actual error. Overall Im fairly confident that this is an important step forward. Checkout the documentation, which has been updated to reflect the changes.

Get Cerberus 0.5 while its hot.

Unfortunately audio is horrible and while all MongoTorino talks were in english, NoSQL Day was an italian-only event. The slide deck is in english however, and is available on both SpeakerDeck and SlideShare.

Get it on PyPI, go straight to the source code or more likely, visit the project homepage.

Ad EuroPython 2012 ho poi raccontato la nostra esperienza e condiviso ciò che avevamo imparato. In quell’occasione mi sono reso conto di quanto l’argomento fosse d’attualità, tanto che da quel 4 luglio le slide che avevo preparato (le trovate su Speaker Deck o Slideshare) ed i video su YouTube (italiano; inglese) hanno totalizzato 30mila visite. Da allora ricevo continue richieste di chiarimenti, suggerimenti, approfondimenti, consigli. Soprattutto, mi si chiedono esempi di codice. In fin dei conti prima o poi qualunque sviluppatore web si trova, volente o nolente, a fare i conti con una REST API.

Così ho pensato che avrei potuto prendere il nostro codice proprietario (nome interno Adam), sistemarlo un po’ in modo che si adattasse ai casi d’uso più comuni, e rilasciarlo come progetto open source. Be’ c’è voluto un bel po’ di lavoro, ben più di quello inizialmente preventivato, ma ora ci siamo. Il progetto è finalmente disponibile, e naturalmente si chiama Eve.

Costruire e lanciare una API con Eve è semplice. Servono tre cose; un database, un file di configurazione, uno script di lancio. MongoDB è già supportato; sviluppare estensioni per altri database SQL/NoSQL non dovrebbe essere troppo complicato. Le personalizzazioni della vostra API sono salvate in un modulo standard Python. Lo script di esecuzione è in genere davvero semplice:

Le API basate su Eve supportano l’intero range di operazioni CRUD (Create, Read, Update, Delete). Ogni end-point è personalizzabile e supporta filtri, paginazione e ordinamenti. Per massima accessibilità i client possono chiedere che le risposte pervengano in JSON oppure XML. Sono supportati gli inserimenti multipli, le direttive di caching per i client, il versioning e… tante altre cosette.

Non mi dilungo oltre poiché su GitHub trovate tutto quel che serve: documentazione (al momento piuttosto scarna), codice e progetti satellite. In particolare date un’occhiata a Eve-Demo, una API funzionante messa su in pochi minuti grazie a Eve. Potete consultarla direttamente, via cURL per esempio, o anche via browser (in questo caso vi beccherete del XML in risposta, mentre via cURL o altro potrete scegliere JSON). Il README di eve-demo include una serie completa di esempi relativi alle varie operazioni di lettura, inserimento, cancellazione, aggiornamento.

C’è ancora tanto da fare affinché Eve diventi il framework completo, stabile e maturo che ho in mente ma credo che il progetto ora sia pronto per una anteprima pubblica. In ogni caso sono lieto di dare finalmente risposta alle richieste di questi mesi: ora avete il codice; usatelo come volete. Spero anche e soprattutto di raccogliere critiche, consigli e suggerimenti; magari a qualcuno verrà anche voglia di collaborare!

PS: Buon 2013!