Django Expense Manager 7 - Implementing the API to Read and Add Categories and Retailers

Sun, Jan 30, 2022
13-minute read

Today we’ll implement some endpoints to be able to read our Categories and to be able to add new ones. Then we’ll do the same for Retailers. In the next post we’ll get to Transactions (our model, not the database transactions). Although they all share more or less the same pattern, we’ll implement them differently at first in order to explore the Django REST Framework more.

Serialisation

What is a serialising? Serialising is the way we’re going to represent complex data (such as a queryset result) into datatypes that can then be rendered into JSON format (in this case). Deserialising is the opposite process. The way we are going to serialize our Category objects is by inheriting from DRF’s ModelSerializer class. Let’s have a look at our Category serializer code. Also, keep in mind that there are many many many many many ways of doing certain things. This is one of them.

class CategorySerializer(serializers.ModelSerializer):
    user = serializers.PrimaryKeyRelatedField(
        read_only=True, default=serializers.CurrentUserDefault())
    product_type = serializers.ChoiceField(
        choices=Category.PRODUCT_TYPES
    )

    class Meta:
        model = Category
        fields = ["name", "product_type", "user"]

Let’s dissect that. It inherits from the ModelSerializer class. What does that give us? Well it automatically generates fields that will be based on the model as well as validators for the serializer.

Remember that our Category had a “name”, “product_type” and “user”. in the Meta class, I explicitly specify which fields I want to include (I know it’s all of them) -> we could have used ‘all’ to specify. BUT what if later one we change the model and there’s some field we don’t want to be public…you would then have to remember to go back and exclude it. Instead, I prefer and it is recommended (I think) to explicitly write the field you’d like to include. So we include “name”, “product_type” and “user”.

Another reason for using a model serializer is that it generates validators for the serializer. Let’s see what that looks like. Let’s open a django shell session and do:

from expenses.api.serializers import CategorySerializer
serializer = CategorySerializer()
print(repr(serializer))

The output we get is:

CategorySerializer():
    name = CharField(max_length=255)
    product_type = ChoiceField(choices=(('P', 'Physical'), ('E', 'Electronic')))
    user = PrimaryKeyRelatedField(default=CurrentUserDefault(), read_only=True)

Ok so it tells us there is a name field (CharField) which was defined automatically for us, a product type with two choices and a user. What does the

PrimaryKeyRelatedField(default=CurrentUserDefault(), read_only=True)

do? The first part is the PrimaryKeyRelatedField. In the serialized representation, for example,

[
  {
    name: "Alcohol",
    product_type: "P",
    user: 2,
  },
  {
    name: "Test",
    product_type: "E",
    user: 2,
  },
];

the user is displayed in the output by its primary key, in this case the user id. There are other ways of representing it but for now that’s all I want and need.

Let’s quickly write a similar serializer for the Retailer but this time let’s not specify anything.

class RetailerSerializer(serializers.ModelSerializer):

    class Meta:
        model = Retailer
        fields = ["name", "online", "user"]

and its serialized representation is:

RetailerSerializer():
    name = CharField(max_length=255)
    online = BooleanField()
    user = PrimaryKeyRelatedField(queryset=User.objects.all())

Ooook that’s not great. In fact, when we do a POST request to this endpoint, we can specify any user we want. We don’t want that. So let’s change our user field to have the same settings as in the CategorySerializer. Do we need to specify what the other two fields should be? No, because DRF figured them out automatically for us. BUT explicit is better. So let’s add both the name and online fields with what the representation gave us (in fact, let’s also add the name to the CategorySerializer as well). You might be wondering: WHY???? Well, when I open the code I like being able to see right away what those fields are and what they’re doing. Later on, when we want to make some changes, it will also make it easier for us. So now we have:

class CategorySerializer(serializers.ModelSerializer):
    user = serializers.PrimaryKeyRelatedField(
        read_only=True, default=serializers.CurrentUserDefault())
    product_type = serializers.ChoiceField(
        choices=Category.PRODUCT_TYPES
    )
    class Meta:
        model = Category
        fields = ["name", "product_type", "user"]


class RetailerSerializer(serializers.ModelSerializer):
    name = serializers.CharField(max_length=255)
    online = serializers.BooleanField()
    user = serializers.PrimaryKeyRelatedField(
        read_only=True, default=serializers.CurrentUserDefault()
    )

    class Meta:
        model = Retailer
        fields = ["name", "online", "user"]

Now you’re gonna complain about me saying I tried these endpoints but how did I try them? Let’s think of how a “normal” Django app works.

You make a HTTP request to some URL. The urlpatterns list routes that URL to a view. The view returns a response.

It’s the same when using DRF. The DRF APIView class subclasses Django’s View class (and does a few extra things). We can actually write both class based views and function based views. TODAY, we will use some of the generic class based views but we might then change to using function based views (which are my preference, it does not make them better or worse).

We are going to do the same thing for both categories and retailers (referred to below as objects). What exactly? We want to have some endpoints where we can:

list ALL the OBJECTS created by the user making the request
get/update/create/delete a SPECIFIC OBJECT by the user making the request

(sorry for shouting above, just want to make sure the difference is very clear)

Looking through DRF’s Generic views page, it would make sense to use ListAPIView as it is for read-only endpoints to represents collections (i.e. lists) of model instances and it only has a get method handler. Ok, that’s exactly what we need. Since it wouldn’t be fun to just use it without explaining how and why it works, let’s also read the line that follows in the documentation:

Extends: GenericAPIView, ListModelMixin

What is the GenericAPIView? Let’s investigate that first and go down the rabbit stack (is that the expression?). I’m gonna go to the deepest level quite fast and then slowly climb up. GenericAPIView extends, and we’re back to, the API View (adding stuff on the way). (If you ever want to know everything about these class views, have a look here). The APIView class has a few class attributes that are of interest to us now (as I will use them shortly, or just talk about them):

authentication_classes. It’s worth mentioning that I have site-wide one set in the expenses_backend/settings.py file: dj_rest_auth.jwt_auth.JWTCookieAuthentication. If, however, you want to overwrite that, you would add a class attribute of authentication_classes = [DIFFERENTAUTHENTICATIONMETHOD] to whatever. I will add it regardless to make it clearer for myself what I’m using.
permission_classes -> The permission classes also has a default site wide setting in expenses_backend/settings.py file but I will set it on the class as well.

For example, if I had the following CBV (class based view):

class RandomClassBasedView(APIView):
    authentication_classes = [dj_rest_auth.jwt_auth.JWTCookieAuthentication]
    permission_classes = [permissions.IsAdminUser]

    def get(self, request):
        ...

    def post(self, request):
        ...

It says that in order to be able to “call” this view, one must be authenticated using a JWT cookie and must have admin permissions. Please note that the CBV above is not tied to any model or anything. We could do whatever we wanted (as long as we passed a JWT authentication token and were admin that is) if we used that view in URL route in urlpatterns. For example, in

path(
        "whatever/", RandomClassBasedView.as_view(), name="whatever_stupid_view"
    ),

what occurs is whatever we ask for in the get and post methods of the class. If we sent a get request or a post request, it would be handled by the class’ get or post method respectively.

What about the Generic API View? It “[adds] commonly required behaviour for standard list and detail views”. Nice! There are a few more class attributes we can set on it, two of which we will discuss now:

queryset - the queryset that is evaluated when calling this endpoint. We will actually not include this BUT have a get_queryset() method on the class, more on this shortly.
serializer_class - the serializer class that is used to validate input/deserialize input/serialize output. aha, this is where we will use/set the serializers we created earlier.

I mentioned we will have a get_queryset() method. That allows us to provide dynamic behaviour to the queryset that is returned: for example to only return the results from a specific user (like the one making the request).

Finally, let’s also look at the ListModelMixin which will return a serialized representation of the queryset (i.e. 0 or more objects) in the response we get. Furthermore, we could look at paginating this (will use when dealing with Transactions).

With all of these in mind, let’s have a look at our first two CBVs.

from expenses.api.serializers import CategorySerializer, RetailerSerializer
from expenses.models import Category, Retailer
from rest_framework import generics
from dj_rest_auth.jwt_auth import JWTCookieAuthentication
from rest_framework.permissions import IsAuthenticated


class CategoryList(generics.ListAPIView):
    authentication_classes = [JWTCookieAuthentication]
    permission_classes = [IsAuthenticated]
    serializer_class = CategorySerializer

    def get_queryset(self):
        """
        This ensures only the categories created by the user are returned.
        """
        user = self.request.user
        return Category.objects.filter(user=user)


class RetailerList(generics.ListAPIView):
    authentication_classes = [JWTCookieAuthentication]
    permission_classes = [IsAuthenticated]
    serializer_class = RetailerSerializer

    def get_queryset(self):
        """
        This ensures only the retailers created by the user are returned.
        """
        user = self.request.user
        return Retailer.objects.filter(user=user)

We have two CBVs inheriting from the ListAPIView, with JWTCookieAuthentication required, and being authenticated as required permission. They serialize either the CategorySerializer or RetailerSerializer and in both cases we overwrote the get_queryset method in order to only return the objects created by the user making the request.

How do we use these? We need to import them and route to them, just like you would a Django CBV. In our expenses/api/urls.py file:

from django.urls import path
from expenses.api.views import CategoryList, RetailerList

urlpatterns = [
    path(
        "categories/",
        CategoryList.as_view(),
        name="view_categories",
    ),
    path(
        "retailers/", RetailerList.as_view(), name="view_retailers"
    ),
]

With all of that, we can start testing these. Since they can’t be used to create any data, we’ll manually create some data before our tests that we can test with.

Testing Authentication of Endpoint

First let’s test whether we can access the endpoint without authentication.

@pytest.mark.django_db
def test_authentication_required(api_client):
    list_category_url: str = reverse("view_categories")
    list_response: Response = api_client.get(
        list_category_url,
    )
    assert list_response.status_code == status.HTTP_401_UNAUTHORIZED

We get the endpoint url and request it. We don’t pass in anything else. The status code is HTTP_401_UNAUTHORIZED, just as expected. Now let’s try logging in a user. For now, we will force_authenticate our testing client. At the end we will also add a proper “integration” test where we actually log in the user properly, get the token and add it to our headers before making the requests.

@pytest.mark.django_db
def test_get_categories_with_token(api_client: APIClient, user_factory):
    email: str = "bobby@email.com"
    password: str = "smith"
    user = user_factory(email=email, password=password)
    list_category_url: str = reverse("view_categories")
    api_client.force_authenticate(user=user)
    list_response: Response = api_client.get(
        list_category_url,
    )
    assert list_response.status_code == status.HTTP_200_OK

With an authenticated request, the status is 200 OK. Perfect! Ok now let’s get some actual categories. In order to do that, let’s create them.

@pytest.mark.django_db
def test_get_categories_with_token_actual_categories(
    api_client: APIClient, user_factory, category_factory
):
    email = "bobby@email.com"
    password = "smith"
    user = user_factory(email=email, password=password)
    _ = category_factory()
    _ = category_factory(user=user)
    list_category_url: str = reverse("view_categories")
    api_client.force_authenticate(user=user)
    list_response: Response = api_client.get(
        list_category_url,
    )
    # only 1 should be available since the other one was created by a
    # different user
    assert (len(list_response.data)) == 1
    assert list_response.status_code == status.HTTP_200_OK
    # let's create another category
    _ = category_factory(user=user)
    # now there should be two returened
    list_response: Response = api_client.get(
        list_category_url,
    )
    assert (len(list_response.data)) == 2
    assert list_response.status_code == status.HTTP_200_OK

What are we doing there? Well, we’re creating an user. Creating a category NOT FOR that user and a category FOR that user. That means when we request the endpoint we should only get 1 category in the response. Then we create another category and we should get two in our data. All good so far! I have done exactly the same thing for retailers but I’m not getting into all the details here. Feel free to have a look at the git commit for this post to see all the tests added.

Adding POST capabilities to our Categories/Retailers endpoints

For now, we can only request existing categories. There is no way for us to create the categories just yet. In the tests we did it using our category and retailer factory respectively. Remember that our ListAPIView only provided a get method handler that could return a collection of model instances. In order to give it a post method handler, we will change it to a ListCreateAPIView. On top of the GenericAPIView and ListModelMixin that the previous one extended, this one also has a CreateModelMixin which provides a .create(request, *args, **kwargs) that we can use to create and save model instances. Let’s change the CategoryList CBV to:


class CategoryList(generics.ListCreateAPIView):
    authentication_classes = [JWTCookieAuthentication]
    permission_classes = [IsAuthenticated]
    serializer_class = CategorySerializer

    def get_queryset(self):
        """
        This ensures only the categories created by the user are returned.
        """
        user = self.request.user
        return Category.objects.filter(user=user)

What do you think would happen if we POSTed to our endpoint now? This:

IntegrityError at /api/categories/

null value in column "user_id" of relation "expenses_category" violates not-null constraint
DETAIL:  Failing row contains (3, Groceries, P, null).

Uh oh, what’s that about? Well, we’re not providing a user id. Why are we not providing a user id in the POST request? Well, because we don’t want anyone to allow creating a record unless it’s by them and they’re logged in (i.e. we don’t want Bob to create a category called NFTs for Jane, she wouldn’t like that). if we have a look at the documentation, we will see that we can always pass in additional information to the serializer, e.g. serializer.save(owner=request.user). How would we do that? Well, we need to do something in the CBV. if we look at the source code for ListCreateAPIView, wee see that it does

    def post(self, request, *args, **kwargs):
        return self.create(request, *args, **kwargs)

where the self.create method comes from the “CreateModelMixin”. If we look in there:

    def create(self, request, *args, **kwargs):
        serializer = self.get_serializer(data=request.data)
        serializer.is_valid(raise_exception=True)
        self.perform_create(serializer)
        headers = self.get_success_headers(serializer.data)
        return Response(serializer.data, status=status.HTTP_201_CREATED, headers=headers)

    def perform_create(self, serializer):
        serializer.save()

We see that we can overwrite the perform_create method. Yay for OOP! Let’s change that to (and I do mean in OUR CBV, not in the source code):

    def perform_create(self, serializer):
        serializer.save(user=self.request.user)

Now, our CategoryList CBV is:

class CategoryList(generics.ListCreateAPIView):
    authentication_classes = [JWTCookieAuthentication]
    permission_classes = [IsAuthenticated]
    serializer_class = CategorySerializer

    def get_queryset(self):
        """
        This ensures only the categories created by the user are returned.
        """
        user = self.request.user
        return Category.objects.filter(user=user)

    def perform_create(self, serializer):
        serializer.save(user=self.request.user)

Now if we POST to it, everything will be fine! We do the same thing to the RetailerList CBV:

class RetailerList(generics.ListAPIView):
    authentication_classes = [JWTCookieAuthentication]
    permission_classes = [IsAuthenticated]
    serializer_class = RetailerSerializer

    def get_queryset(self):
        """
        This ensures only the retailers created by the user are returned.
        """
        user = self.request.user
        return Retailer.objects.filter(user=user)

    def perform_create(self, serializer):
        serializer.save(user=self.request.user)

But wait, now our urlpattern names:

urlpatterns = [
    path(
        "categories/",
        CategoryList.as_view(),
        name="view_categories",
    ),
    path(
        "retailers/", RetailerList.as_view(), name="view_retailers"
    ),
]

don’t make sense. Let’s change them to view_and_create_categoris and view_and_create_retailers. Keep in mind we also need to change them in our test_categories API file. and, we also want to have some TESTS obviously. One without authentcation and one with:

@pytest.mark.django_db
def test_post_category_without_authentication(api_client):
    list_category_url: str = reverse("view_and_create_categories")
    data = {"name": "Groceries", "product_type": "P"}
    list_response: Response = api_client.post(
        list_category_url, data=data, format="json"
    )
    assert list_response.status_code == status.HTTP_401_UNAUTHORIZED


@pytest.mark.django_db
def test_post_category_with_authentication(api_client, user_factory):
    initial_count: int = Category.objects.all().count()
    user = user_factory()
    api_client.force_authenticate(user=user)
    list_category_url: str = reverse("view_and_create_categories")

    data = {"name": "Groceries", "product_type": "P"}
    list_response: Response = api_client.post(
        list_category_url, data=data, format="json"
    )
    final_count: int = Category.objects.all().count()
    assert list_response.status_code == status.HTTP_201_CREATED
    assert final_count == initial_count + 1

And I think that’s where we will stop today. Now I need to review all the code I added, make sure I didn’t do anything stupid and maybe even proof read this post. In the next one we will look at implementing endpoints for individual categories/retailers and should start looking at expenses too.

The commit for this post is here

python django djangorestframework python finance postgresql django sql pytest api