> Tutorial <

The aim of this tutorial is to provide a comprehensive overview of Resource API components and how they are supposed to be used to implement real-life applications.

Note

Some of the sections and examples of this tutorial intersect with the information provided by the actual code documentation. This is intentional.

Intro

For the sake of this tutorial lets try to design a simple school course management system backed with a primitive python shelve backend that is reasonable to use during prototype phase. We shall implement a granular authorization using means of Resource API framework. After the implementation is done we shall play with direct Python API and HTTP interface.

1: collect the use cases

Our course simple management system must be able to support the following actions:

  • students can sign up to courses
  • teachers can kick the students out from the courses
  • teachers can grade the students during the courses
  • students can make comments about the course and rate its teacher respectively
  • teachers are supposed to have information like qualification, diploma, etc.

2: determine entities

Based on usecases mentioned above we can list the following entities (in singular):

  • student
  • teacher
  • course
  • comment
  • student grade
  • teacher rating

Also the following links are supposed to exist:

  • student can sign up for MANY courses and each course can be attended by MANY students
  • teacher can teach MULTIPLE courses but each course can be taught only by a SINGLE teacher
  • teacher can grade a student ONCE for every course the teacher owns
  • every student can grade every teacher ONCE for every course the teacher was teaching
  • every student can make comments about the courses as much as he wants

3: entity diagram

Note

Designing the perfect course management system is not among the goals of this tutorial. It aims to demonstrate how Resource API facilitates dealing with implementation issues.

There are several ways to model the system. For the sake of this example we shall look at teachers and students as at separate entities.

_images/entity_diagram.png

Entity diagram

4: ontology to code

After we managed to come up with an idea on what kind of resources we are supposed to have and how they are expected to link to one another we need to write python code that would correspond to these structures.

from resource_api.interfaces import Resource, Link


class Student(Resource):
    """ A pupil """

    class Links:

        class courses(Link):
            """ Courses the student has ever attended """
            target = "Course"
            related_name = "students"
            master = True

        class comments(Link):
            """ Comments made by the student """
            target = "Comment"
            related_name = "student"

        class ratings(Link):
            """ Ratings given by the student """
            target = "TeacherRating"
            related_name = "student"


class Teacher(Resource):
    """ A lecturer """

    class Links:

        class ratings(Link):
            """ Ratings given to the teacher """
            target = "TeacherRating"
            related_name = "teacher"

        class courses(Link):
            """ Courses the teacher is responsible for """
            target = "Course"
            related_name = "teacher"


class Course(Resource):
    """ An educational unit represinting the lessons for a specific set of topics """

    class Links:

        class teacher(Link):
            """ The lecturer of the course """
            target = "Teacher"
            related_name = "courses"
            cardinality = Link.cardinalities.ONE
            master = True
            required = True

        class comments(Link):
            """ All comments made about the course """
            target = "Comment"
            related_name = "course"

        class ratings(Link):
            """ All ratings that were given to the teachers of the specific course """
            target = "TeacherRating"
            related_name = "course"

        class students(Link):
            """ All pupils who attend the course """
            target = "Student"
            related_name = "courses"


class Comment(Resource):
    """ Student's comment about the course """

    class Links:

        class student(Link):
            """ The pupil who made the comment """
            target = "Student"
            related_name = "comments"
            cardinality = Link.cardinalities.ONE
            master = True
            required = True

        class course(Link):
            """ The subject the comment was made about """
            target = "Course"
            related_name = "comments"
            cardinality = Link.cardinalities.ONE
            master = True
            required = True


class TeacherRating(Resource):
    """ Student's rating about teacher's performance """

    class Links:

        class student(Link):
            """ The pupil who gave the rating to the teacher """
            target = "Student"
            related_name = "ratings"
            cardinality = Link.cardinalities.ONE
            master = True
            required = True

        class course(Link):
            """ The subject with respect to which the rating was given """
            target = "Course"
            related_name = "ratings"
            cardinality = Link.cardinalities.ONE
            master = True
            required = True

        class teacher(Link):
            """ The lecturer to whom the rating is related """
            target = "Teacher"
            related_name = "ratings"
            cardinality = Link.cardinalities.ONE
            master = True
            required = True

As you can see every relationship (or link) consists of two edges of a bidirectional relationship. Even though we will dive into implementation details of the relationships later on it is critical to highlight that the main purpose of Resource API is to maintain relational integrity in a similar fashion to graph or SQL databases.

There are couple of important things to note in the code above.

First, all links must have a target and related_name defined. A combination of these two attributes lets the framework bind two edges into a single link entity.

Second, one of the edges must be marked as a master one. Structural validation (if there is any structure) and authorization are performed against the master edge only. Storing data in one place is a logical way to save storage space. And the employed approach of authorization lets the following scenario be possible: if someone can add a student to the course then the same user could add the course to the student’s course list.

Third, there is a cardinality attribute. There are two possible values for this one. ONE and MANY. The edge with cardinality ONE does not differ from the MANY implementation-wise. However, the framework returns a single object for the ONE and a collection for MANY via object interface.

Check interface documentation to find out more about link attributes.

5: structure of the entities

Apart from the relationships between the resources there is another bit of knowledge which is vital for modeling the system. It is the structure of individual resources (e.g.: Student must have an email, first name, last name and a birthday).

Lets expand our graph to include structural information as well.

_images/entity_diagram_with_structure.png

Entity diagram with structure

As you can see from the relationship between Course and Student, links may have attributes as well. Resource API supports link properties in a similar fashion as Neo4j.

6: structure to code

Full version of the code can be seen here. Below we shall focus on the critical bits of the implementation.

Lets have a look at Student’s schema :

class Student(Resource):
    """ A pupil """

    class Schema:
        email = StringField(regex="[^@]+@[^@]+\.[^@]+", pk=True,
                            description="Addess to which the notifications shall be sent")
        first_name = StringField(description="Given name(s)")
        last_name = StringField(description="Family name(s)")

As you can see the structure of the entity is exposed declaratively. Instead of writing multiple functions to validate the input we just say what the input is supposed to be. This approach of describing entities’ structures is similar to the one used by Django models for example.

The key difference from Django’s approach is usage of the inner class called Schema. The nested class exists to prevent naming collisions between user defined fields and the fields that are used by framework internals. Links are defined in a separate nested class for the same reason.

7: persistence

When we have resource structures defined in our code it is still not enough to make the entities do anything useful. We need to program how these entities are supposed to be stored an fetched to/from the persistence layer, file system, etc.

To make the components work we need to implement Resource/Link interface.

A full version of the implementation can be found here. Below we shall focus on critical implementation details.

First, lets have a look at Resource implementation:

class Resource(BaseResource):

    def __init__(self, context):
        super(Resource, self).__init__(context)
        self._storage = context["storage"]

    def exists(self, user, pk):
        return pk in self._storage.get(self.get_name(), {})

    def get_data(self, user, pk):
        return self._storage.get(self.get_name(), {}).get(pk)

    def delete(self, user, pk):
        self._storage.get(self.get_name(), {}).pop(pk)
        self._storage.sync()

    def create(self, user, pk, data):
        if self.get_name() not in self._storage:
            self._storage[self.get_name()] = {}
        self._storage[self.get_name()][pk] = data
        self._storage.sync()

    def update(self, user, pk, data):
        self._storage[self.get_name()][pk].update(data)
        self._storage.sync()

    def get_uris(self, user, params=None):
        return self._storage.get(self.get_name(), {}).keys()

    def get_count(self, user, params=None):
        return len(self.get_uris(params))

Next, lets check Link implementation:

class Link(BaseLink):

    def __init__(self, context):
        super(Link, self).__init__(context)
        self._storage = context["storage"]

    def exists(self, user, pk, rel_pk):
        return rel_pk in self._storage.get((pk, self.get_name()), {})

    def get_data(self, user, pk, rel_pk):
        return self._storage.get((pk, self.get_name()), {}).get(rel_pk)

    def create(self, user, pk, rel_pk, data=None):
        key = (pk, self.get_name())
        if key not in self._storage:
            self._storage[key] = {}
        self._storage[key][rel_pk] = data
        self._storage.sync()

    def update(self, user, pk, rel_pk, data):
        self._storage[key][rel_pk].update(data)
        self._storage.sync()

    def delete(self, user, pk, rel_pk):
        self._storage.get((pk, self.get_name()), {}).pop(rel_pk)
        self._storage.sync()

    def get_uris(self, user, pk, params=None):
        return self._storage.get((pk, self.get_name()), {}).keys()

    def get_count(self, user, pk, params=None):
        return len(self.get_uris(pk, params))

As you can see each abstract method of both interfaces are implemented to use Shelve database.

Warning

Note, compensating transactions are one of the TODO features to be added to Resource API in the future. Now any error in the implementation of the resource when creating/deleting the entity with multiple associated links has high chances to cause relational inconsistency.

Now let examine the service class:

class ShelveService(Service):

    def __init__(self):
        super(ShelveService, self).__init__()
        self._storage = shelve.open(SHELVE_PATH, writeback=True)

    def _get_context(self):
        return {"storage": self._storage}

    def _get_user(self, data):
        return None

    def __del__(self):
        self._storage.close()

ShelveService implements abstract Service. We had to override two abstract methods.

First, _get_context. This method must return an object that shall be passed to all resources during initialization. The context shall be available as a context attribute of resource objects. It makes sense to put service-wide singletons like DB connections, persistence layers or open sockets into the context.

Second, _get_user. More on it later. But in short it is expected to return a user that would be used for authorization later on.

In addition to service and entity implementations there are a few more important lines:

srv = ShelveService()
srv.register(Student)
srv.register(Teacher)
srv.register(Course)
srv.register(Comment)

The lines above provide an overview of how to notify the system that certain resources are supposed to be exposed. Each resource must be registered with a respective method call in order to become a part of the API.

Also notice a setup() method call. It must be invoked after all the required resources are registered. The method validates that resource relationships point to registered resources. Meaning: if we registered a Student but did not register a Course - Resource API would raise a ResourceDeclarationError.

8: primary key

When addressing the resource Resource API follows the standards and employs a URI concept. In the example above the URI is represented by a field marked as a primary key. What the framework does by default - it takes the value of the field and passes it to resources set method. Resource is a synonym of word entity within the context of Resource API.

In contrast with a student resource the following Comment entity has a primary key that does not have any direct value (unlike the Student’s email) for the end user. Thus passing it together with the rest of the data during entity creation does not make sense. For this purpose we need to override URI creation mechanism.

9: custom UriPolicy

To change the way URI is generated and processed for a specific resource we need to subclass UriPolicy and implement a bunch of its methods.

A full version of the service with custom UriPolicy can be found here. Below we shall focus on important details of the implementation.

Lets have a look at Comment definition:

class Comment(Resource):
    """ Student's comment about the course """

    UriPolicy = AutoGenSha1UriPolicy

In order to override URI creation mechanism we explicitly changed UriPolicy from the default one to AutoGenSha1UriPolicy.

Lets have a closer look at AutoGenSha1UriPolicy:

class AutoGenSha1UriPolicy(AbstractUriPolicy):
    """ Uses a randomly generated sha1 as a primary key """

    @property
    def type(self):
        return "autogen_policy"

    def generate_pk(self, data):
        return os.urandom(16).encode('hex')

    def serialize(self, pk):
        return pk

    def deserialize(self, pk):
        if not isinstance(pk, basestring):
            raise ValidationError("Has to be string")
        if not RE_SHA1.match(value):
            raise ValidationError("PK is not a valid SHA1")
        return pk

There are three abstract methods that were implemented.

First, getnerate_pk. It returns a random SHA1 string.

Second, serialize method. Since we do not change the URI anyhow when storing the resource we return it as is.

Third, deserialize method. Here we validated that input value is a string and that it fits a SHA1 regular expression.

10: authorization

Since we want to limit the access to various resources only to specific categories of users, we need to implement authorization using granular can_ methods of Link and Resource subclasses.

Full implementation of authorization can be seen here. Below we shall focus on authorization details.

Lets have a look at the methods that limit read-only access to the entities only for authenticated users.

    def can_get_data(self, user, pk, data):
        """ Only authenticated users can access data """
        if user.is_anonymous:
            return False
        else:
            return True

    def can_get_uris(self, user):
        """ Only authenticated users can access data """
        if user.is_anonymous:
            return False
        else:
            return True

We just return False if a user is anonymous. We shall see how the user object should be created later on.

Since we wanted to let Students and Teachers update only their own info, we encapsulated authorization logic within a Person class.

class Person(Resource):

    class Schema:
        email = StringField(regex="[^@]+@[^@]+\.[^@]+", pk=True,
                            description="Addess to which the notifications shall be sent")
        first_name = StringField(description="Given name(s)")
        last_name = StringField(description="Family name(s)")

    def can_update(self, user, pk):
        """ Only a person himself can update his own information """
        return user.email == pk or user.is_admin

    def can_delete(self, user, pk):
        """ Only admins can delete people """
        return user.is_admin

And both Student and Teacher inherit from the Person:

class Student(Person):
    """ A pupil """

    class Schema(Person.Schema):
        birthday = DateTimeField()
class Teacher(Person):
    """ A lecturer """

    class Schema(Person.Schema):
        category = StringField(description="TQS Category",
                               choices=["four", "five", "five plus", "six"])

Notice, that we also extracted common bits of the schema into Person.Schema. Thus Student and Teacher schemas inherit from it.

Within the scope of our app it makes sense that teachers can create only courses for themselves and students can make comments only on their own behalf.

In order to enforce this behavior we introduced a PersonalLink:

class PersonalLink(Link):
    """ Users can link things to their accounts only """

    def can_update(self, user, pk, rel_pk, data):
        return user.email == rel_pk or user.is_admin

    def can_create(self, user, pk, rel_pk, data):
        return user.email == rel_pk or user.is_admin

And made Course‘s link to Teacher and Comment‘s and TeacherRating‘s link to Student inherit from the PersonalLink:

        class teacher(PersonalLink):
            """ The lecturer of the course """
        class student(PersonalLink):
            """ The pupil who made the comment """
        class student(PersonalLink):
            """ The pupil who gave the rating to the teacher """

One last bit of authorization detail required to understand how the implementation is done - a user object. It can be virtually anything. However, it is critical to note that this object is passed to all authorization methods as the first parameter by the framework.

Lets have a look at what the school app does with the user:

    def _get_user(self, data):
        if data is None:
            return User(None)
        else:
            return User(data.get("email"))

Where class User is defined the following way:

class User(object):

    def __init__(self, email=None):

        if email is None:
            self.is_anonymous = True
        else:
            self.is_anonymous = False

        if email == "admin@school.com":
            self.is_admin = True
        else:
            self.is_admin = False

        self.email = email

As simple as that.

11: object interface

Object interface provides a python way for traversing the resources.

In order to do the traversal on our school service we need to fetch the entry point.

entry_point = srv.entry_point({"email": "admin@school.com"})
student_root_collection = entry_point.get_resource(Student)
student_root_collection.create({"email": "student@school.com",
                                "first_name": "John",
                                "last_name": "Smith",
                                "birthday": "2000-09-25"})

Note

Please check object interface docs for more detailed information on how to use the direct Python API.

12. HTTP interface

In order to make HTTP interface work, service instance has to be passed to a WSGI application:

srv = ShelveService()
srv.register(Student, "school.Student")
srv.register(Teacher, "school.Teacher")
srv.register(Course, "school.Course")
srv.register(Comment, "school.Comment")
srv.register(TeacherRating, "school.TeacherRating")


from resource_api_http.http import Application
from werkzeug.serving import run_simple
app = Application(srv)
run_simple('localhost', 8080, app, use_reloader=True)

Full version of the file (which can be executed as a full featured app) can be found here.

Lets have a look at the most significant bits in the declaration.

First, notice how the resources are registered:

srv.register(ResourceClass, "namespace.ResourceName")

This is in general a good practice to register all entities under a specific name so that the API is not too tightly coupled with Python modules & class names. Module name and class name are used by default as a namespace and a resource name respectively.

Second, we removed setup() call. WSGI application does it internally anyways.

Third, the application is passed to Werkzeug‘s method. Werkzeug is a WSGI library powering Resource API HTTP component.

When the service is up and running it is possible to do HTTP requests with CURL:

Fetch service descriptor via OPTIONS request:

curl -X OPTIONS 127.0.0.1:8080 | python -m json.tool

Fetch a collection of students:

curl 127.0.0.1:8080/foo.Student

Oh. 403 status code. This is because we did not include authentication information required for authorization.

curl --header "email: admin@school.com" 127.0.0.1:8080/school.Student

Empty collection. Lets create a student.

curl -X POST --header "email: admin@school.com" --header "Content-Type: application/json" \
     -d '{"email":"foo@bar.com","first_name": "John", "last_name": "Smith", "birthday": "1987-02-21T22:22:22"}' \
      127.0.0.1:8080/school.Student

Lets fetch Student collection again:

curl --header "email: admin@school.com" 127.0.0.1:8080/school.Student

As you can see a new student appeared in the list.

Please check HTTP interface reference for more information.