lostinsoba
Posts Tags
lostinsoba

Contents

Kubernetes DAO

   3782 words   18 minutes 

This post covers a basic API implementation using Kubernetes informers

Limitations

  • etcd is the most vulnerable component of Kubernetes

We need to limit both the number of stored objects and the number of API requests.
That’s one of the reasons we use informers - to reduce load on the API server and, indirectly, on etcd.

  • Kubernetes does not use pessimistic locking

We rely on optimistic concurrency using resourceVersion:

Worker A reads object O with resourceVersion N → 
    applies changes → 
        sends an update request with version N → 
            the object is updated to version M.

Worker B reads object O with resourceVersion N → 
    applies changes → 
        sends an update request with version N → 
            receives a conflict error because the current version is no longer N.

This is similar to Grafana, where a user may receive an error indicating that a dashboard has been updated since it was opened, with an option to force an update.

In Kubernetes, a similar force update behavior can be implemented, but without field ownership, it will not be atomic. It requires re-reading the latest version of the object and re-applying changes on top of it.

Overview

There are multiple levels of abstraction to operate with custom resources in Kubernetes.
Depending on our needs we can use:

API Client

This is the lowest-level building block and all other abstractions rely on it.
It allows us to interact with built-in Kubernetes resources such as Pods, ReplicaSets and Deployments via direct API calls.
For custom resources, we typically need to manually marshal and unmarshal YAML or JSON into Go structs.
The client can also be configured with rate limiting to reduce the number of API requests and prevent overloading the API server.

Watchers

Watchers are a mechanism for receiving real-time updates from the Kubernetes API server about changes to resources.

They stream events:

  • ADD
  • UPDATE
  • DELETE

and can reduce API load by maintaining local state based on these events.

However, watchers alone are low-level and do not provide caching, indexing or type safety by themselves.

Informers

Informers are a higher-level abstraction built on top of watchers.

They provide:

  • Local caching of objects
  • Event handlers
  • Automatic resync mechanisms
  • Indexers for efficient lookups
  • Cache transform interface (minimizing memory consumption)
  • Reduced load on the API server

For custom resources, informers can be generated using code generation tools (client-gen, informers-gen or controller-gen depending on the stack).

Controllers

Controllers are the highest-level abstraction in this stack.

They implement reconciliation loops that continuously ensure the cluster state matches the desired state defined in Kubernetes objects.

Controllers typically:

  • Use informers for efficient state observation
  • Contain reconciliation logic (the control loop)
  • Expose minimal or no external API surface (mostly metrics endpoints)

Model

In our example, to avoid being distracted by the business domain, we’ll implement a simple library service that allows administrators to add books and tenants to borrow them.

Book

Field Type Description Source
ID string Book ID ObjectMeta.Name
Author string Book author CustomResourceDefinition.Spec
Title string Book title CustomResourceDefinition.Spec
Description string Book description CustomResourceDefinition.Spec
NamespaceID string Book namespace ObjectMeta.Namespace
ManagedBy string Book object manager ObjectMeta.Labels
CategoryID string Book category ObjectMeta.Labels
Field Label Type Description
ManagedBy app.kubernetes.io/managed-by string Recommended Labels
CategoryID books.bookshelf.lostinsoba.com/category string Book category

The ManagedBy label serves two purposes:

  • It identifies whether a book was created by the service or provisioned through an IaC workflow.
  • It enables us to prevent users from modifying resources managed by IaC.

The CategoryID label is used solely to demonstrate label-based indexing and resource lookup, since labels are being indexed while annotations are not.

Labels can be efficiently queried both through the API server and informer indexes.

However, labels should not be treated as arbitrary storage:

  • Labels are intended for filtering and grouping
  • Values should remain relatively small
  • High-cardinality labels may negatively affect performance

BookLoan

Field Type Description Source
ID string Book loan ID ObjectMeta.Name
BorrowedBy string Tenant ID CustomResourceDefinition.Spec

We could use a status subresource for this, but that approach may introduce additional complexity in access control and resource update flows.

Instead, we’ll use an event-driven approach with a small adjustment: a BookLoan object will use the same name as the corresponding Book object, simplifying state checks and lookups.

This approach can be changed later to preserve the full borrowing history and provide visibility into all events related to a particular book.

Development enviroment

It’s possible to test in any available Kubernetes cluster, but not obligatory.
Our example only requires having minikube installed on your laptop.

minikube start --driver=docker

CustomResourceDefinition

Book

Let’s create and apply custom resource definition for Book model:

kubectl apply -f - <<EOF
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: books.bookshelf.lostinsoba.com
spec:
  group: bookshelf.lostinsoba.com
  scope: Namespaced
  names:
    plural: books
    singular: book
    kind: Book
  preserveUnknownFields: false
  versions:
    - name: v1alpha1
      served: true
      storage: true
      additionalPrinterColumns:
        - name: Author
          type: string
          jsonPath: .spec.author
        - name: Title
          type: string
          jsonPath: .spec.title
        - name: Category
          type: string
          jsonPath: .metadata.labels['books\.bookshelf\.lostinsoba\.com/category']
        - name: Age
          type: date
          jsonPath: .metadata.creationTimestamp
      schema:
        openAPIV3Schema:
          type: object
          description: |
            Book            
          required:
            - spec
          properties:
            spec:
              type: object
              required:
                - author
                - title
              properties:
                author:
                  type: string
                  description: Book author
                title:
                  type: string
                  description: Book title
                description:
                  type: string
                  description: Book description
EOF

Create namespace:

kubectl create namespace science-library

Create example books:

kubectl apply -f - <<EOF
apiVersion: bookshelf.lostinsoba.com/v1alpha1
kind: Book
metadata:
  name: modenov-analytic-geometry
  namespace: science-library
  labels:
    books.bookshelf.lostinsoba.com/category: math
spec:
  author: "Modenov P.S."
  title: "Analytic Geometry"
EOF

kubectl apply -f - <<EOF
apiVersion: bookshelf.lostinsoba.com/v1alpha1
kind: Book
metadata:
  name: taylor-biological-science
  namespace: science-library
  labels:
    books.bookshelf.lostinsoba.com/category: biology
spec:
  author: "Taylor D.J., Green N.P.O., Stout G.W."
  title: "Biological Science"
EOF

List created books:

> kubectl get books -n science-library

NAME                        AUTHOR                        TITLE                CATEGORY   AGE
modenov-analytic-geometry   Modenov P.S.                  Analytic Geometry    math       25s
taylor-biological-science   Taylor D.J., Green N.P.O...   Biological Science   biology    14s

Labels are indexed, so we can query books by label:

> kubectl get books -l books.bookshelf.lostinsoba.com/category=biology -n science-library

NAME                        AUTHOR                        TITLE                CATEGORY   AGE
taylor-biological-science   Taylor D.J., Green N.P.O...   Biological Science   biology    38s

BookLoan

And the model for book loans:

kubectl apply -f - <<EOF
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: bookloans.bookshelf.lostinsoba.com
spec:
  group: bookshelf.lostinsoba.com
  scope: Namespaced
  names:
    plural: bookloans
    singular: bookloan
    kind: BookLoan
  preserveUnknownFields: false
  versions:
    - name: v1alpha1
      served: true
      storage: true
      additionalPrinterColumns:
        - name: Borrowed By
          type: string
          jsonPath: .spec.borrowedBy
        - name: Borrowed At
          type: string
          jsonPath: .metadata.creationTimestamp
      schema:
        openAPIV3Schema:
          type: object
          description: |
            Book Loan            
          required:
            - spec
          properties:
            spec:
              type: object
              required:
                - borrowedBy
              properties:
                borrowedBy:
                  type: string
                  description: Borrowed by
EOF

Codebase structure

We’ll need to clone kubernetes sample controller
We are only interested in the hack directory, we’ll copy this directory to our project directory
And in update-codegen.sh we need to change THIS_PKG variable to store our application name:

THIS_PKG="bookshelf"

kube::codegen::gen_client \
    --with-watch \
    --output-dir "${SCRIPT_ROOT}/pkg/generated" \
    --output-pkg "${THIS_PKG}/pkg/generated" \
    --boilerplate "${SCRIPT_ROOT}/hack/boilerplate.go.txt" \
    "${SCRIPT_ROOT}/pkg/apis"

So now we can create a Makefile with this directive:

generate:
    ./hack/update-codegen.sh

Then we need to create directory pkg and containing apis so our project now has the following directory structure:


  ├── Makefile
  ├── hack
  ├── pkg
  │    ├── apis.go
  │    └── apis/bookshelf/v1alpha1
  │         ├── doc.go
  │         ├── register.go
  │         └── types.go
  └── rest of the project code

We can also use the existing go.mod file to understand what modules are required.
We just need to ensure all k8s.io packages use the same version.

apis.go:

package apis

import (
	"k8s.io/apimachinery/pkg/runtime"

	bookshelfv1alpha1 "bookshelf/pkg/apis/bookshelf/v1alpha1"
)

func RegisterScheme() (*runtime.Scheme, error) {
	sch := runtime.NewScheme()
	err := bookshelfv1alpha1.AddToScheme(sch)
	if err != nil {
		return nil, err
	}
	return sch, nil
}

doc.go:

// +k8s:deepcopy-gen=package,register
// +groupName=bookshelf.lostinsoba.com

package v1alpha1

register.go:

package v1alpha1

import (
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/apimachinery/pkg/runtime/schema"
)

const (
	resourceGroup   = "bookshelf.lostinsoba.com"
	resourceVersion = "v1alpha1"
)

var SchemeGroupVersion = schema.GroupVersion{
	Group:   resourceGroup,
	Version: resourceVersion,
}

func Resource(resource string) schema.GroupResource {
	return SchemeGroupVersion.WithResource(resource).GroupResource()
}

var (
	SchemeBuilder = runtime.NewSchemeBuilder(addKnownTypes)
	AddToScheme   = SchemeBuilder.AddToScheme
)

func addKnownTypes(scheme *runtime.Scheme) error {
	scheme.AddKnownTypes(SchemeGroupVersion,
		&Book{},
		&BookList{},
		&BookLoan{},
		&BookLoanList{},
	)
	metav1.AddToGroupVersion(scheme, SchemeGroupVersion)
	return nil
}

types.go:

Book

package v1alpha1

import (
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/labels"
)

const (
	bookResourceKind       = "Book"
	bookResourceKindPlural = "books"
)

var BookGroupResourceVersion = SchemeGroupVersion.WithResource(bookResourceKindPlural)

var BookGroupVersionKind = SchemeGroupVersion.WithKind(bookResourceKind)

var BookTypeMeta = metav1.TypeMeta{
	APIVersion: BookGroupVersionKind.GroupVersion().String(),
	Kind:       BookGroupVersionKind.Kind,
}

const (
	labelCategory = "books.bookshelf.lostinsoba.com/category"
)

func (b *Book) GetCategory() string {
	return b.Labels[labelCategory]
}

func BuildCategoryLabelSet(categoryID string) labels.Set {
	return labels.Set{
		labelCategory: categoryID,
	}
}

const (
	labelManagedBy      = "app.kubernetes.io/managed-by"
	labelManagedByValue = "bookshelf"
)

func (b *Book) GetManagedBy() string {
	return b.Labels[labelManagedBy]
}

func (b *Book) IsManageable() bool {
  managedBy := b.Labels[labelManagedBy]
  return managedBy == "" || managedBy == labelManagedByValue
}

func BuildManagedByLabelSet() labels.Set {
	return labels.Set{
		labelManagedBy: labelManagedByValue,
	}
}

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
type BookList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata"`
	Items           []*Book `json:"items"`
}

// +genclient
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
type Book struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`
	Spec              BookSpec `json:"spec"`
}

type BookSpec struct {
	Author      string `json:"author"`
	Title       string `json:"title"`
	Description string `json:"description"`
}

BookLoan

const (
	bookLoanResourceKind       = "BookLoan"
	bookLoanResourceKindPlural = "bookloans"
)

var BookLoanGroupResourceVersion = SchemeGroupVersion.WithResource(bookLoanResourceKindPlural)

var BookLoanGroupVersionKind = SchemeGroupVersion.WithKind(bookLoanResourceKind)

var BookLoanTypeMeta = metav1.TypeMeta{
	APIVersion: BookLoanGroupVersionKind.GroupVersion().String(),
	Kind:       BookLoanGroupVersionKind.Kind,
}

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
type BookLoanList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata"`
	Items           []*BookLoan `json:"items"`
}

// +genclient
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
type BookLoan struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`
	Spec              BookLoanSpec `json:"spec"`
}

type BookLoanSpec struct {
	BorrowedBy string `json:"borrowedBy"`
}

Generate code for informers:

> make generate

./hack/update-codegen.sh
Generating deepcopy code for 1 targets
Generating client code for 1 targets
Generating lister code for 1 targets
Generating informer code for 1 targets

Docker image

Instead of using docker build, we’ll use minikube image build to build and deliver our image to minikube:

minikube image build -t bookshelf:develop .

Helm chart

Next, we need to write our helm chart.
We’ll use it in the following way:

helm upgrade --install bookshelf . \
  -n bookshelf \
  --create-namespace

Our helm chart must contain at least:

  • ServiceAccount
  • ClusterRole
  • ClusterRoleBinding
  • Deployment
  • Service

Our service will have to run under a ServiceAccount to be able to operate with our CRDs:

ServiceAccount

apiVersion: v1
kind: ServiceAccount
metadata:
  name: bookshelf

ClusterRole

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: bookshelf
rules:
- apiGroups: ["bookshelf.lostinsoba.com"]
  resources: ["books", "bookloans"]
  verbs: ["*"]

ClusterRoleBinding

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: bookshelf
subjects:
- kind: ServiceAccount
  name: bookshelf
  namespace: {{ .Release.Namespace }}
roleRef:
  kind: ClusterRole
  name: bookshelf
  apiGroup: rbac.authorization.k8s.io

Deployment

Deployment must contain field spec.template.spec.serviceAccountName equal to ServiceAccount Name.
After the image build, we’ll be restarting our deployment the following way:

kubectl rollout restart deployment/bookshelf -n bookshelf

Service

Service must expose our API port.
We’ll connect to our service using port-forwarding:

kubectl port-forward -n bookshelf svc/bookshelf 8080:8080

Service development

Model

package model

type Book struct {
	ID          string
	Author      string
	Title       string
	Description string
	CategoryID  string
	NamespaceID string
	ManagedBy   string
}

Storage

package storage

import (
	"log/slog"

	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/client-go/rest"

	"bookshelf/internal/storage/kubernetes"
)

type Storage struct {
	store  *kubernetes.Store
	logger *slog.Logger
}

func New(cfg *rest.Config, scheme *runtime.Scheme, logger *slog.Logger) (*Storage, error) {
	store, err := kubernetes.New(cfg, scheme)
	if err != nil {
		return nil, err
	}
	return &Storage{
		store:  store,
		logger: logger,
	}, nil
}

Connector

In our example, we use namespaced objects. However, this connector (as well as the informer) can be used for cluster scoped objects too. The namespace field will just contain an empty string.

package kubernetes

import (
  "context"

  "k8s.io/apimachinery/pkg/labels"
  "k8s.io/apimachinery/pkg/runtime"
  "k8s.io/client-go/rest"

  "sigs.k8s.io/controller-runtime/pkg/client"
)

type Store struct {
  typedClient client.Client
}

func New(cfg *rest.Config, scheme *runtime.Scheme) (*Store, error) {
  typedClient, err := client.New(cfg, client.Options{Scheme: scheme})
  if err != nil {
    return nil, err
  }
  return &Store{
    typedClient: typedClient,
  }, nil
}

func (s *Store) Get(ctx context.Context, namespace, name string, obj client.Object) error {
  key := client.ObjectKey{
    Namespace: namespace,
    Name:      name,
  }
  return s.typedClient.Get(ctx, key, obj)
}

func (s *Store) List(
  ctx context.Context,
  namespace string,
  labelSelector labels.Selector,
  list client.ObjectList,
) error {
  return s.typedClient.List(
    ctx,
    list,
    &client.ListOptions{
      Namespace:     namespace,
      LabelSelector: labelSelector,
    },
  )
}

func (s *Store) Create(ctx context.Context, obj client.Object) error {
  return s.typedClient.Create(ctx, obj)
}

func (s *Store) Update(ctx context.Context, obj client.Object) error {
  return s.typedClient.Update(ctx, obj)
}

func (s *Store) Patch(ctx context.Context, oldObj, newObj client.Object) error {
  return s.typedClient.Patch(ctx, newObj, client.MergeFrom(oldObj))
}

func (s *Store) Delete(ctx context.Context, obj client.Object) error {
  uid := obj.GetUID()
  return s.typedClient.Delete(ctx, obj, client.Preconditions{
    UID: &uid,
  })
}

Repository

Book
package storage

import (
  "context"
  "fmt"
  "log/slog"
  "strings"

  apierrors "k8s.io/apimachinery/pkg/api/errors"
  metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
  "k8s.io/apimachinery/pkg/labels"
  "k8s.io/apimachinery/pkg/runtime"
  "k8s.io/apimachinery/pkg/util/validation"
  "sigs.k8s.io/controller-runtime/pkg/client"

  "bookshelf/internal/model"

  bookshelfv1alpha1 "bookshelf/pkg/apis/bookshelf/v1alpha1"
)

func (s *Storage) ListBooks(
  ctx context.Context,
  query model.ListBooksQuery,
) ([]*model.Book, error) {
  var books bookshelfv1alpha1.BookList
  err := s.store.List(ctx, query.NamespaceID, buildLabelSelector(query), &books)
  if err != nil {
    return nil, err
  }
  list := make([]*model.Book, 0, len(books.Items))
  for _, item := range books.Items {
    listItem, convertErr := convertRuntimeObjectToBookModel(item)
    if convertErr != nil {
      s.logger.Error("failed to convert object to book, skipping...",
        slog.String("name", item.Name),
        slog.String("namespace id", item.Namespace),
        slog.Any("error", convertErr),
      )
    }
    list = append(list, listItem)
  }
  return list, nil
}

func (s *Storage) CreateBook(ctx context.Context, query model.CreateBookQuery) error {
  book, err := convertBookModelToClientObject(query.Book)
  if err != nil {
    return err
  }
  err = s.store.Create(ctx, book)
  if apierrors.IsAlreadyExists(err) {
    return model.ErrAlreadyExist
  }
  return err
}

func (s *Storage) UpdateBook(ctx context.Context, query model.UpdateBookQuery) error {
  var targetBook bookshelfv1alpha1.Book
  err := s.store.Get(ctx, query.Book.NamespaceID, query.Book.ID, &targetBook)
  if apierrors.IsNotFound(err) {
    return model.ErrNotFound
  }
  if err != nil {
    return err
  }
  if !targetBook.IsManageable() {
    return model.ErrNotManageable
  }
  applyBookUpdate(&targetBook, query.Book)
  err = s.store.Update(ctx, &targetBook)
  if apierrors.IsConflict(err) {
    return model.ErrUpdateConflict
  }
  return err
}

func (s *Storage) GetBookByID(
  ctx context.Context,
  query model.GetBookByIDQuery,
) (*model.Book, error) {
  book, err := s.getBookByID(ctx, query.NamespaceID, query.BookID)
  if err != nil {
    return nil, err
  }
  return convertRuntimeObjectToBookModel(&book)
}

func (s *Storage) SetCategoryID(ctx context.Context, query model.SetCategoryIDQuery) error {
  book, err := s.getBookByID(ctx, query.NamespaceID, query.BookID)
  if err != nil {
    return err
  }
  original := book.DeepCopy()
  categoryLabelSet := bookshelfv1alpha1.BuildCategoryLabelSet(query.CategoryID)
  book.Labels = labels.Merge(book.Labels, categoryLabelSet)
  err = s.store.Patch(ctx, original, &book)
  if apierrors.IsConflict(err) {
    return model.ErrUpdateConflict
  }
  return err
}

func (s *Storage) getBookByID(
  ctx context.Context,
  namespace string,
  id string,
) (bookshelfv1alpha1.Book, error) {
  var book bookshelfv1alpha1.Book
  err := s.store.Get(ctx, namespace, id, &book)
  switch {
  case apierrors.IsNotFound(err):
    return bookshelfv1alpha1.Book{}, model.ErrNotFound
  case err != nil:
    return bookshelfv1alpha1.Book{}, err
  default:
    return book, nil
  }
}

func (s *Storage) DeleteBookByID(ctx context.Context, query model.DeleteBookByIDQuery) error {
  book, err := s.getBookByID(ctx, query.NamespaceID, query.BookID)
  if err != nil {
    return err
  }
  if !book.IsManageable() {
    return model.ErrNotManageable
  }
  err = s.store.Delete(ctx, &book)
  return client.IgnoreNotFound(err)
}

func buildLabelSelector(query model.ListBooksQuery) labels.Selector {
  labelSet := make(labels.Set)
  if query.CategoryID != "" {
    bookCategoryLabelSet := bookshelfv1alpha1.BuildCategoryLabelSet(query.CategoryID)
    labelSet = labels.Merge(labelSet, bookCategoryLabelSet)
  }
  if len(labelSet) == 0 {
    return labels.Everything()
  }
  return labels.SelectorFromSet(labelSet)
}

func convertRuntimeObjectToBookModel(object runtime.Object) (*model.Book, error) {
  switch crd := object.(type) {
  case *bookshelfv1alpha1.Book:
    return &model.Book{
      ID:          crd.Name,
      Author:      crd.Spec.Author,
      Title:       crd.Spec.Title,
      Description: crd.Spec.Description,
      CategoryID:  crd.GetCategory(),
      NamespaceID: crd.GetNamespace(),
      ManagedBy:   crd.GetManagedBy(),
    }, nil
  default:
    return nil, fmt.Errorf("unsupported object type: %T", crd)
  }
}

func applyBookUpdate(book *bookshelfv1alpha1.Book, bookModel model.Book) {
  categoryLabelSet := bookshelfv1alpha1.BuildCategoryLabelSet(bookModel.CategoryID)
  manageableLabelSet := bookshelfv1alpha1.BuildManagedByLabelSet()
  labelSet := labels.Merge(categoryLabelSet, manageableLabelSet)

  book.Spec.Author = bookModel.Author
  book.Spec.Title = bookModel.Title
  book.Spec.Description = bookModel.Description

  book.Labels = labels.Merge(
    book.Labels,
    labelSet,
  )
}

func convertBookModelToClientObject(book model.Book) (client.Object, error) {
  if book.ID == "" {
    return nil, fmt.Errorf("missing book id")
  }
  if book.NamespaceID == "" {
    return nil, fmt.Errorf("missing namespace id")
  }
  if book.Author == "" {
    return nil, fmt.Errorf("missing book author")
  }
  if book.Title == "" {
    return nil, fmt.Errorf("missing book title")
  }
  if errs := validation.IsDNS1123Subdomain(book.ID); len(errs) > 0 {
    return nil, fmt.Errorf("invalid book id: %s", strings.Join(errs, ", "))
  }
  categoryLabelSet := bookshelfv1alpha1.BuildCategoryLabelSet(book.CategoryID)
  manageableLabelSet := bookshelfv1alpha1.BuildManagedByLabelSet()
  return &bookshelfv1alpha1.Book{
    TypeMeta: bookshelfv1alpha1.BookTypeMeta,
    ObjectMeta: metav1.ObjectMeta{
      Name:      book.ID,
      Namespace: book.NamespaceID,
      Labels:    labels.Merge(categoryLabelSet, manageableLabelSet),
    },
    Spec: bookshelfv1alpha1.BookSpec{
      Author:      book.Author,
      Title:       book.Title,
      Description: book.Description,
    },
  }, nil
}
BookLoan
package storage

import (
	"context"
	"fmt"

	apierrors "k8s.io/apimachinery/pkg/api/errors"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"sigs.k8s.io/controller-runtime/pkg/client"

	"bookshelf/internal/model"

	bookshelfv1alpha1 "bookshelf/pkg/apis/bookshelf/v1alpha1"
)

func (s *Storage) BorrowBookByID(ctx context.Context, query model.BorrowBookByIDQuery) error {
	bookLoan, err := convertBorrowBookByIDQueryToBookLoanClientObject(query)
	if err != nil {
		return err
	}
	err = s.store.Create(ctx, bookLoan)
	if apierrors.IsAlreadyExists(err) {
		return model.ErrAlreadyBorrowed
	}
	return err
}

func (s *Storage) ReturnBookByID(ctx context.Context, query model.ReturnBookByIDQuery) error {
	var bookLoan bookshelfv1alpha1.BookLoan
	err := s.store.Get(ctx, query.NamespaceID, query.BookID, &bookLoan)
	switch {
	case apierrors.IsNotFound(err):
		return nil
	case err != nil:
		return err
	default:
		if bookLoan.Spec.BorrowedBy != query.ReturnedBy {
			return model.ErrForbidden
		}
		return s.store.Delete(ctx, &bookLoan)
	}
}

func convertBorrowBookByIDQueryToBookLoanClientObject(
  query model.BorrowBookByIDQuery,
) (client.Object, error) {
  if query.BookID == "" {
    return nil, fmt.Errorf("missing book id")
  }
  if query.NamespaceID == "" {
    return nil, fmt.Errorf("missing namespace id")
  }
  if query.BorrowedBy == "" {
    return nil, fmt.Errorf("missing borrowed by")
  }
  return &bookshelfv1alpha1.BookLoan{
    TypeMeta: bookshelfv1alpha1.BookLoanTypeMeta,
    ObjectMeta: metav1.ObjectMeta{
      Name:      query.BookID,
      Namespace: query.NamespaceID,
    },
    Spec: bookshelfv1alpha1.BookLoanSpec{
      BorrowedBy: query.BorrowedBy,
    },
  }, nil
}

Cache

package cache

import (
  "context"
  "log/slog"
  "time"

  "github.com/go-logr/logr"

  "k8s.io/client-go/rest"
  "k8s.io/klog/v2"

  "sigs.k8s.io/controller-runtime/pkg/client"

  "bookshelf/internal/cache/kubernetes"
  "bookshelf/pkg/generated/clientset/versioned"
  "bookshelf/pkg/generated/informers/externalversions"

  bookshelfv1alpha1 "bookshelf/pkg/apis/bookshelf/v1alpha1"
)

type Cache struct {
  namespacedBooks     *kubernetes.Informer
  namespacedBookLoans *kubernetes.Informer
  logger              *slog.Logger
}

func New(cfg *rest.Config, resyncInterval time.Duration, logger *slog.Logger) (*Cache, error) {
  {
    handler := logger.Handler()
    kLogger := logr.FromSlogHandler(handler)
    klog.SetLogger(kLogger)
  }
  clientSet, err := versioned.NewForConfig(cfg)
  if err != nil {
    return nil, err
  }
  // books
  namespacedBooksCacheItemTransformFunc := func(obj interface{}) (interface{}, error) {
    // Informer caches keep full copies of all observed objects in memory.
    // Large text fields such as descriptions may significantly increase memory
    // consumption when thousands of resources are stored.
    // The transform function allows us to remove fields that are not required for
    // read operations served from cache.
    switch received := obj.(type) {
    case *bookshelfv1alpha1.Book:
      cached := received.DeepCopy()
      // cached.Spec.Description = ""
      return cached, nil
    }
    return obj, nil
  }
  namespacedBooksInformerBuilder := func() (kubernetes.GenericInformer, error) {
    factory := externalversions.NewSharedInformerFactoryWithOptions(
      clientSet,
      resyncInterval,
      externalversions.WithTransform(namespacedBooksCacheItemTransformFunc),
    )
    return factory.ForResource(bookshelfv1alpha1.BookGroupResourceVersion)
  }
  eventCallback := func(eventKind kubernetes.InformerEventKind, obj client.Object) {
    logger.Info("received informer event",
      slog.String("object", obj.GetName()),
      slog.String("object namespace", obj.GetNamespace()),
      slog.String("event kind", eventKind.String()),
    )
  }
  namespacedBooksInformer, err := kubernetes.New(
    namespacedBooksInformerBuilder,
    kubernetes.OptionWithEventCallback(eventCallback),
  )
  if err != nil {
    return nil, err
  }
  // book loans
  namespacedBookLoansInformerBuilder := func() (kubernetes.GenericInformer, error) {
    factory := externalversions.NewSharedInformerFactoryWithOptions(
      clientSet,
      resyncInterval,
    )
    return factory.ForResource(bookshelfv1alpha1.BookLoanGroupResourceVersion)
  }
  namespacedBookLoansInformer, err := kubernetes.New(namespacedBookLoansInformerBuilder)
  if err != nil {
    return nil, err
  }
  return &Cache{
    namespacedBooks:     namespacedBooksInformer,
    namespacedBookLoans: namespacedBookLoansInformer,
    logger:              logger,
  }, nil
}

func (c *Cache) SyncInBackground(ctx context.Context) error {
  err := c.namespacedBooks.Run(ctx)
  if err != nil {
    return err
  }
  err = c.namespacedBookLoans.Run(ctx)
  if err != nil {
    return err
  }
  return nil
}

Informer

package kubernetes

import (
  "context"
  "fmt"

  "k8s.io/apimachinery/pkg/api/meta"
  "k8s.io/apimachinery/pkg/labels"
  "k8s.io/apimachinery/pkg/runtime"
  "k8s.io/client-go/tools/cache"

  "sigs.k8s.io/controller-runtime/pkg/client"
)

type Informer struct {
  internalInformer GenericInformer
}

type InformerBuilder func() (GenericInformer, error)

type InformerEventKind int

const (
  InformerEventKindCreate InformerEventKind = iota
  InformerEventKindUpdate
  InformerEventKindDelete
)

func (ek InformerEventKind) String() string {
  switch ek {
  case InformerEventKindCreate:
    return "create"
  case InformerEventKindUpdate:
    return "update"
  case InformerEventKindDelete:
    return "delete"
  default:
    return "unknown"
  }
}

// This function signature is only for demonstration purposes.
// Later, it can be extended with oldObj to cover more cases related with update events.
// And there's one notable thing about delete events: 
// We cannot rely on DeletionTimestamp field of object, 
// because the time we receive the event, it will be empty
type InformerEventCallback func(eventKind InformerEventKind, obj client.Object)

type GenericInformer interface {
  Informer() cache.SharedIndexInformer
  Lister() cache.GenericLister
}

type Option func(GenericInformer)

func OptionWithEventCallback(eventCallback InformerEventCallback) Option {
  return func(informer GenericInformer) {
    eventHandler := cache.ResourceEventHandlerFuncs{
      AddFunc: func(obj interface{}) {
        clientObj, isClientObj := obj.(client.Object)
        if isClientObj {
          eventCallback(InformerEventKindCreate, clientObj)
        }
      },
      UpdateFunc: func(_, newObj interface{}) {
        clientObj, isClientObj := newObj.(client.Object)
        if isClientObj {
          eventCallback(InformerEventKindUpdate, clientObj)
        }
      },
      DeleteFunc: func(obj interface{}) {
        switch o := obj.(type) {
        case client.Object:
          eventCallback(InformerEventKindDelete, o)
        case cache.DeletedFinalStateUnknown:
          if clientObj, ok := o.Obj.(client.Object); ok {
            eventCallback(InformerEventKindDelete, clientObj)
          }
        }
      },
    }
    _, _ = informer.Informer().AddEventHandler(eventHandler)
  }
}

func New(informerBuilder InformerBuilder, opts ...Option) (*Informer, error) {
  internalInformer, err := informerBuilder()
  if err != nil {
    return nil, err
  }
  for _, opt := range opts {
    opt(internalInformer)
  }
  return &Informer{
    internalInformer: internalInformer,
  }, nil
}

func (i *Informer) Run(ctx context.Context) error {
  informer := i.internalInformer.Informer()
  go informer.Run(ctx.Done())
  if !cache.WaitForCacheSync(
    ctx.Done(),
    informer.HasSynced,
  ) {
    if err := context.Cause(ctx); err != nil {
      return err
    }
    return fmt.Errorf("failed to sync informer cache")
  }
  return nil
}

func (i *Informer) Get(ctx context.Context, namespace, name string) (runtime.Object, error) {
  return i.internalInformer.Lister().ByNamespace(namespace).Get(name)
}

func (i *Informer) List(
  ctx context.Context,
  namespace string,
  labelSelector labels.Selector,
  list client.ObjectList,
) error {
  runtimeObjects, err := i.internalInformer.Lister().
    ByNamespace(namespace).List(labelSelector)
  if err != nil {
    return err
  }
  return meta.SetList(list, runtimeObjects)
}

Repository

package cache

import (
	"context"
	"fmt"
	"log/slog"

	apierrors "k8s.io/apimachinery/pkg/api/errors"
	"k8s.io/apimachinery/pkg/labels"
	"k8s.io/apimachinery/pkg/runtime"

	"bookshelf/internal/model"
	bookshelfv1alpha1 "bookshelf/pkg/apis/bookshelf/v1alpha1"
)

func (c *Cache) ListBooks(
  ctx context.Context,
  query model.ListBooksQuery,
) ([]*model.Book, error) {
  var books bookshelfv1alpha1.BookList
  err := c.namespacedBooks.List(ctx, query.NamespaceID, buildLabelSelector(query), &books)
  if err != nil {
    return nil, err
  }
  list := make([]*model.Book, 0, len(books.Items))
  for _, item := range books.Items {
    listItem, convertErr := convertRuntimeObjectToBookModel(item)
    if convertErr != nil {
      c.logger.Error("failed to convert object to book, skipping...",
        slog.String("name", item.Name),
        slog.String("namespace id", item.Namespace),
        slog.Any("error", convertErr),
      )
    }
    list = append(list, listItem)
  }
  return list, nil
}

func (c *Cache) GetBookByID(
  ctx context.Context,
  query model.GetBookByIDQuery,
) (*model.Book, error) {
  obj, err := c.namespacedBooks.Get(ctx, query.NamespaceID, query.BookID)
  switch {
  case apierrors.IsNotFound(err):
    return nil, model.ErrNotFound
  case err != nil:
    return nil, err
  default:
    return convertRuntimeObjectToBookModel(obj)
  }
}

func buildLabelSelector(query model.ListBooksQuery) labels.Selector {
	labelSet := make(labels.Set)
	if query.CategoryID != "" {
		bookCategoryLabelSet := bookshelfv1alpha1.BuildCategoryLabelSet(query.CategoryID)
		labelSet = labels.Merge(labelSet, bookCategoryLabelSet)
	}
	if len(labelSet) == 0 {
		return labels.Everything()
	}
	return labels.SelectorFromSet(labelSet)
}

func convertRuntimeObjectToBookModel(object runtime.Object) (*model.Book, error) {
  switch crd := object.(type) {
  case *bookshelfv1alpha1.Book:
    return &model.Book{
      ID:          crd.Name,
      Author:      crd.Spec.Author,
      Title:       crd.Spec.Title,
      Description: crd.Spec.Description,
      CategoryID:  crd.GetCategory(),
      NamespaceID: crd.GetNamespace(),
      ManagedBy:   crd.GetManagedBy(),
    }, nil
  default:
    return nil, fmt.Errorf("unsupportyed object type: %T", crd)
  }
}

API

Admin API

package router

func (r *Router) Route() http.Handler {
  mux := http.NewServeMux()
  mux.Handle("GET /book/{namespaceID}", r.listBooks)
  mux.Handle("POST /book/{namespaceID}", r.createBook)
  mux.Handle("GET /book/{namespaceID}/{id}", r.getBookByID)
  mux.Handle("PUT /book/{namespaceID}/{id}", r.updateBook)
  mux.Handle("DELETE /book/{namespaceID}/{id}", r.deleteBookByID)
  mux.Handle("PATCH /book/{namespaceID}/{id}/category", r.setCategoryID)
  mux.Handle("GET /health", r.health())
  route := middleware.ExtractTenantID(mux)
  route = middleware.ExtractRequestID(route)
  return route
}
List books
curl http://localhost:8080/book/science-library | python3 -m json.tool

{
    "list": [
        {
            "id": "modenov-analytic-geometry",
            "author": "Modenov P.S.",
            "title": "Analytic Geometry",
            "category_id": "math",
            "namespace_id": "science-library"
        },
        {
            "id": "taylor-biological-science",
            "author": "Taylor D.J., Green N.P.O., Stout G.W.",
            "title": "Biological Science",
            "category_id": "biology",
            "namespace_id": "science-library"
        }
    ]
}

curl http://localhost:8080/book/science-library?category_id=math | python3 -m json.tool

{
    "list": [
        {
            "id": "modenov-analytic-geometry",
            "author": "Modenov P.S.",
            "title": "Analytic Geometry",
            "category_id": "math",
            "namespace_id": "science-library"
        }
    ]
}
Update book
curl -X PUT http://localhost:8080/book/science-library/modenov-analytic-geometry \
  -H "Content-Type: application/json" \
  -d '{
        "id": "modenov-analytic-geometry",
        "author": "Modenov P.S.",
        "title": "Analytic Geometry",
        "category_id": "math",
        "description": "The green book for 1st semester",
        "namespace_id": "science-library"
  }'
Set category ID
curl -X POST http://localhost:8080/book/science-library/modenov-analytic-geometry/category \
  -H "Content-Type: application/json" \
  -d '{
        "category_id": "geometry"
  }'
Get book by ID
curl "http://localhost:8080/book/science-library/modenov-analytic-geometry" |
  python3 -m json.tool

{
  "id": "modenov-analytic-geometry",
  "author": "Modenov P.S.",
  "title": "Analytic Geometry",
  "category_id": "geometry",
  "namespace_id": "science-library",
  "description": "The green book for 1st semester",
  "managed_by": "bookshelf"
}

User API

package router

func (r *Router) Route() http.Handler {
  mux := http.NewServeMux()
  mux.Handle("GET /book/{namespaceID}", r.listBooks)
  mux.Handle("GET /book/{namespaceID}/{id}", r.getBookByID)
  mux.Handle("POST /book/{namespaceID}/{id}/borrow", r.borrowBookByID)
  mux.Handle("POST /book/{namespaceID}/{id}/return", r.returnBookByID)
  mux.Handle("GET /health", r.health())
  route := middleware.ExtractTenantID(mux)
  route = middleware.ExtractRequestID(route)
  return route
}
BorrowBookByID
curl -X POST http://localhost:8080/book/science-library/modenov-analytic-geometry/borrow \
  -H "X-TENANT-ID: bob"

curl -X POST http://localhost:8080/book/science-library/modenov-analytic-geometry/borrow \
  -H "X-TENANT-ID: alice" | python3 -m json.tool

{
  "message":"already borrowed"
}

> kubectl get bookloans -n science-library

NAME                        BORROWED BY   BORROWED AT
modenov-analytic-geometry   bob           2026-06-18T15:38:51Z
Updated on 2026-06-18
 kubernetes, minikube, golang
Back | Home