Ghost's Blog

AI and a personal project

Sun, 01 Feb 2026 20:00:00 GMT

Yes, another AI post. I apologize, but I think this one will be worth it. I’m tired of posts about companies downsizing because of “AI” (when they really mean LLMs) too, or engineers bragging about workflows with 5 parallel agents (only to throw 95% of the generated PRs in the trash). I’m also tired of companies trying to taking away the joy we found in building. Definitely, we can’t have nice things.

The thing is, today I want to tell you about a project that AI has actually enabled me to do something I wouldn’t have done without these tools: the transcription of a handwritten diary my grandfather left behind. My grandfather wrote this in 1989. They are his memoirs of the Spanish Civil War. He passed away 20 years ago and, when I inherited this notebook, I tried to transcribe it. It’s full of spelling mistakes, common for people of his generation, and the handwriting is quite difficult to read. It took me a whole day just to poorly transcribe a couple of pages. I abandoned the project.

But the other day, I decided to feed the first two pages to ChatGPT. And “magic” happened: on its own, it managed to enhance the image, run it through OCR, and get most of the content right. It still makes plenty of mistakes, which is normal given the spelling and the handwriting, but it has allowed me to decipher the script and finish the sentences myself.

Your browser does not support the video tag.

I don’t know where this project will lead me. My boss told me that when you start worrying about your past, it’s a sign that you’ve officially grown old. And I couldn’t agree more :)

The value in execution

Wed, 05 Nov 2025 20:00:00 GMT

As you grow in your career, you tend to move away from execution roles. You increasingly spend more time doing management and less time executing. Sometimes, you even get promoted right into a position where you become totally useless.

Companies tend to push you into these management roles if you want to grow. The Individual Contributor (IC) path often has a shorter trajectory, while the management path is usually longer. Perhaps this makes sense at an organizational level (you manage more people), but in terms of impact, I have serious doubts. Every day, I see engineers who have far more cross-departmental impact than positions supposedly designed for it. The difference is that the engineers have the power of execution, while many managers actually don’t: others execute for them.

That’s why I believe it’s so crucial to remain capable of getting down to execution (or getting your hands dirty, as some dismissively call it). I’m not saying you have to be constantly executing (which many would label as micromanagement), but there are critical projects where you absolutely must support your team.

I see in my own teams that when managers share the actual workload, or are capable of supporting their team’s execution during tough times, the manager is better regarded and the team becomes more cohesive. Supporting execution doesn’t necessarily mean just writing code. It can mean simplifying a problem, helping to minimize a trade-off, or assisting with a complex technical decision.

I’m writing this today because I watched two independent teams collaborate incredibly fast on a common problem, and I believe it had a lot to do with the fact that the managers got our hands dirty quickly.

Cold calls and closed doors

Sun, 14 Sep 2025 20:00:00 GMT

Several salespeople have contacted me because of this blog. On one hand, I like that you try new ways to get our attention. On the other, it’s obvious you don’t care about what I write: it’s just an excuse for a hook.

For years I’ve been bombarded with calls, even to my personal number. At this point, if you’re not in my contacts, I don’t answer. That sometimes makes me miss important calls: medical or other urgent ones. So you’ll understand why I’m defensive when I get this type of call.

I’ve never been in sales, but I’ve always thought cold calls can’t really work: you’re trying to sell to someone who doesn’t want to buy. Not because your product isn’t interesting, but because we’re not in the mood to be interrupted anytime.

Another big annoyance: urgency. Tech leaders are usually very busy. It’s unrealistic to expect us to have time this same week to meet.

If you’re a salesperson reading this and want my honest advice: get introduced by someone. If you have no common contacts, try an email, but know that we already get dozens like yours. And please, don’t call us unless we’ve explicitly given you permission (spoiler: what your marketing tool says isn’t it, even if it’s legally covered).

Most likely, you’ll just close a door forever.

Starting to fail

Fri, 20 Jun 2025 22:00:00 GMT

Two things happened to me this month: I missed a meeting and I missed a train.

That had never happened to me in 20 years, and for most people, it would probably just be another anecdote. But I feel there may be something else. I may be hitting my ceiling.

I’ve been feeling for several months now that I’m trying to swallow more than I should. It has worked so far. I see how these two examples may seem pretty small, but I think they were a good warning: stop and delegate before it’s too late.

Thoughs on my way home

Sat, 19 Apr 2025 10:00:00 GMT

The 21st century really is amazing. I’m on an AVE train back to Madrid, coding on a laptop that weighs less than 1 kg, connected to the Internet. I barely type —an AI does the heavy lifting. I give it commands, review what it gives me, make corrections, and I don’t forget to say please and thank you.

Seven years ago, I was on another AVE headed to Valencia. Back then, my laptop was twice as heavy and had barely any Internet. I wrote the foundations of Playtomic’s payment flow in those 2.5 hours. All in Java + Spring Boot, hand-coded in IntelliJ. The DB was local. Today, I’m in Cursor with Next.js/TypeScript, and the DB is remote on NeonDB.

I used to prep offline work for train rides. Now? Nothing. If I don’t have a connection, it’s not worth working —better to wait until I arrive.

Makes me really think about how much our profession is going to change in the coming years.

Also, this is the first post I write in Spanish on this blog and I’m asking ChatGPT to translate it for me, in my style.

8 years at Playtomic.io

Fri, 14 Feb 2025 10:00:00 GMT

I made 8 years at Playtomic today. I feel old :)

I’ve been reading my post from last year. Well… this year has not been well-balanced at all 🥲.

I’ve been responsible for Tech since August. My predecessor left me with a lot of debt to pay, so it has been a year to surface it. We are better than ever in processes, resilience, security, and, especially, internal communication. Still work to do, but I see light at the end of the tunnel.

One of the things I enjoyed the most is the connections I’ve made with many other players in the industry. Many of our providers use Playtomic, and they don’t say it just to gain our favor: they really use it. It’s always an honor when some of our main providers (AWS, Google, MongoDB, Datadog) want to visit us directly in our offices.

What I enjoyed the least was the amount of spam I received on my personal phone number. If you are trying to reach me to sell me something and you read this, be warned: you will be automatically banned from working with me in the future. Yes, I am pretty mad about this.

For 2025, my expectation is to move from this reactive/internal work to a more proactive/external approach. Let’s see how it goes.

What I Learned in 2024

Wed, 01 Jan 2025 20:00:00 GMT

I don’t usually write year-end reviews, but I opened this blog and saw that I haven’t posted since February. The drop in time I spend writing is concerning, but it’s something I plan to change in 2025.

I’m rereading that post, and back in February, I mentioned having a good work-life balance and even getting back to my hobbies. Well, all of that went out the window in April for a good reason: I was promoted to Senior Director of Technology! I am really proud of leading the technical decisions at one of Spain’s most promising companies: Playtomic. But it came with a cost: it has taken all my energy, and all my free time has gone toward staying sane. Sun, exercise, enough sleep, and friends. Many would envy that, but I usually dedicate a lot of time to personal projects (don’t worry, they’re not coding-related).

The point is, I want to make a list of things I’ve learned or applied this year:

Work is probably the most important thing you’ll do daily, but your health is more important.
Outside of work, disconnect. Eight hours are more than enough, and you need to be at your best the next day. It’s a marathon, not a sprint.
You’ll be a better leader if you can show vulnerability: sometimes you won’t have the answer, and the best thing is to say it openly.
Share your doubts with your team.
I don’t buy into the “loneliness of leadership” idea. I’ve never felt more supported.
Everyone communicates worse than they think (especially managers).
Everyone understands what they want to some extent (especially employees).
Everyone is more sensitive than they would like to admit (especially… everyone).
Everyone should learn to put misunderstandings into perspective.
Want to be a role model? It’s not enough to be right, you have to convince others.
You don’t have to fight every battle.
Some battles are for others to learn (even if you think they will lose).
Quit Twitter/X. This should have been first.
Attend more tech events and get out of your cave, I mean, office.
Your email, phone, and LinkedIn will get bombarded. You own your time, and you don’t have to reply to everyone.
That said, respond to contacts you want to maintain, even if it is to say now isn’t the right time.
There are projects necessary for the company, and there are projects necessary for your team.
Learn to find time for the latter and learn to sell them to your company.
Photography made me a better engineer.
We don’t want to read texts written by AI. We want to hear the writer’s voice.
Learn to write in your own voice. In the coming years, everyone will sound the same. Like ChatGPT. This will be to your advantage.

I am sure I am forgetting some things, but 2025 looks promising, with exciting professional and personal projects that I hope I will finally have time to write about here. I definitely have to write about how having another passion has made me a better professional.

Happy 2025!

Kelly Sikkema https://unsplash.com/photos/white-spiral-notebook-on-brown-wooden-table-2q_frVRXWfQ

7 years at Playtomic.io

Wed, 14 Feb 2024 10:00:00 GMT

This year I haven’t written much. Time flew by; I blinked a few times, and suddenly, a whole year had passed. I’ve added only 2 posts to this blog. And Today also marks another year at Playtomic. That makes seven.

This was the year of AI, or more precisely, the year of generative AI. I still believe there is a lot of IA from the 90s and 00s that can be applied to many of our current challenges in the industry. With just a touch of linear regression, we could significantly simplify our lives :)

One of my previous employers told me that five years in enough in a company to learn most of you can get from that company. I somewhat agree with him, but this is the first year that I have felt that my role is well-balanced with my workload. I want to savor this moment. I still have ton of work, but now there are people managing the day-to-day. I can’t thank them enough.

For the first time in many years, I have found the energy to resume my other passion: exploring the world through the viewfinder of a camera. I might share some of my work here :) I just wanted to let you know that I am happy, doing well and looking forward for another year here.

This is Playtomic's Anemone

Mon, 25 Sep 2023 10:00:00 GMT

We, tech companies, don’t share a lot about our internals. But I think we should share more often, especially if we are proud of what we have built. So this a post about sharing (or bragging): let me introduce you to Anemone.

Anemone is the system that provides the data and operations to all Playtomic products. The Backend. It comprises 60 services, more than 150 instances, 3 different types of databases, 1 kafka cluster. All services share a common 1 core library (to rule them all) that provides common functionality as converters or security.

We wanted to build it so it is resilient, scalable, and easy to change. Sea Anemones inspired the name. They are capable of splitting completely on their own and keep living. Resilient beings. It’s a bit nerdy but beautiful.

I wondered how our anemone would look. So I asked DALL-E to draw it for me. And this is what it came up with. Again, a bit nerdy but beautiful.

How to add a listener port to Undertow in Spring Boot

Wed, 29 Mar 2023 10:00:00 GMT

Super short post today. I couldn’t believe none has tried this before. We wanted to add a management port to separate traffic and healthchecks ports. But during the migration, we wanted to have both ports listening to healthchecks so we make everything back-compatible.

Once you active the Spring Boot management port, /actuator endpoints are moved there. This is the easiest way to add an extra port to the Undertow server.

@ConditionalOnProperty(name = “server.extra-port”) @Configuration @Slf4j public class UndertowConfiguration implements WebServerFactoryCustomizer {

@Value("${server.extra-port:-1}")
private int httpPort;

@Override
public void customize(@Nonnull UndertowServletWebServerFactory factory) {
    factory.addBuilderCustomizers(builder -> {
        if (httpPort > 0) {
            log.info("Using extra port {} for the traffic Undertow", httpPort);
            builder.addHttpListener(httpPort, "0.0.0.0");
        }
    });
}

}


More info: https://docs.spring.io/spring-boot/docs/2.1.10.RELEASE/reference/html/howto-embedded-web-servers.html#howto-configure-webserver

[^1]: Matt Paul Catalano https://unsplash.com/photos/0QEG_xOoY7Y

6 years at Playtomic.io

Mon, 13 Feb 2023 10:00:00 GMT

I started this post 6 months ago but never finished it.

Today, I make 6 years in Playtomic. Everything goes so fast that it feels like 6 decades. It goes so fast that it took me 4 more years to gather the energy to write the sequel. Honestly, I didn’t think I would still be here. Neither that this blog would still be alive too.

It feels like it was 6 decades, but at the same time feels like I am at the same vital point. Crypto has exploded and decayed several times during these 4 years. I have seen NFTs artists rise and fall. AI Art and language models are shaking our world up.

And we are still here, trying to revolutionize the world of racket sports. Still trying. Feels so much more mundane compared to that previous paragraph.

Don’t get me wrong, I don’t want to diminish our progress. We have more than 800.000 players and 6.000 clubs. We are present in more than 30 countries. The growth has been insane, as our current mental is. We have suffered layoffs, as most tech companies based on venture capital. And several reorganizations.

I have changed and expanded my role so many times that I am trying to remember what I was supposed to do at the beginning. I have acted as Head of the Backend of a team of 22 people. During some period, I had to balance that role with the Engineer Manager (EM) role of a group of 5 developers. I had suffered outages while I was 2500 km away from home. I have fixed and optimized countless amount of problems in our systems. I have discussed functionality and roadmap worth for a lifetime.

Luckily, we have help now and can count on two of the best EMs I know. My responsibilities are mainly technical again, and it is a relief. Executing and managing for long periods is insane at this pace.

Where will I be in a year? Honestly, I don’t know. But I hope in a place where I can keep writing on this blog O:)

How to migrate a cluster in production

Fri, 10 Feb 2023 08:00:00 GMT

One of the most complex operations in tech is upgrades or migrations without disrupting the service. I always compare it to upgrading a plane when it is flying.

Last year we faced a problem: Docker Swarm (Community) seems stalled. Besides, autoscaling and other tools that other container clusters provide out of the box are very manual.

Our setup at that moment was:

Docker Swarm cluster
~50 Spring Boot services running on that cluster
Deployed on AWS EC2
API access via Application Load Balancers
Every service was mapped to a Target Group
No discovery service: We use hostnames to find the services internally.

There are many ways of deploying a microservice architecture. This is the simplest we found, and it worked like a solid rock for 5 years. This is what it looks like:

As you can see, Internet traffic comes from a Load Balancer. Traffic between services is internal. As part of the internal traffic, there is an event platform (Kafka).

Now that I think about it twice, it probably looked more like this:

For our new cluster, we wanted something standard and, preferably, managed. So we decided to move out to EKS (the managed Kubernetes service on AWS). The problem is not writing the new descriptors to deploy on Kubernetes. The hard part is how to migrate the platform without stopping the service.

We could define a maintenance window in the early morning for a few hours, do the switch, and continue. But that process is stressful and very error-prone. What happens if we find an error? Rollback? Go forward? So we decided to do it progressively. It would be longer but safer. And we would learn a lot.

In summary, we saw two options:

Set up the new architecture in the new cluster and do the switch: faster but harder to do without errors and stopping the service.
Migrate service step by step: slower but safer. No need to stop the service.

How to connect two clusters

There are two common points for all the services in our architecture: the gateway (the load balancer in our case) and the data stores (sql, mongodb, kafka, …).

If we start a service in the cluster, we need to consider that:

It might receive traffic from the Load Balancer.
It might receive events from Kafka.
It might call other services that… might not be in the same cluster.

If you have a discovery service, you could set up both clusters in reachable networks and register the services in the same discovery service. But we do not have such a piece.

The only way is to pass all the http traffic through the load balancer and split the traffic into both clusters. The load balancer will be our on/off switch. Drawbacks:

it is a bit slower compared to the internal traffic (although nothing critical on our setup)
more expensive, as AWS charges for the traffic handled by the load balancer.

So this is the setup during the migration:

One target group per service and cluster. The load balancer splits the traffic into those target groups (the dotted lines). Services call each other going through the load balancer (the green lines). That allows us to control what services are deployed on the new cluster.

When every service is running on the new cluster, then we can stop the traffic to the old one:

Again, as the traffic is controlled on the load balancer, we can stop services on our previous clusters while removing access from the load balancer.

And finally, we could delete all the components related to our previous cluster:

That is, we removed the mapping on the load balancer. That allowed us to delete the previous cluster completely. On the new cluster, we restored the internal traffic. Profit!

Conclusions

The plan was sound and easy to follow.

The only drawback that I see is the number of manual steps we needed to perform:

Move the internal traffic to the load balancer (one change in every service).
Deploy the service on the new cluster.
Create new target groups and map them on the load balancer.
Split the traffic to both target groups for every route mapped.
Check that the service behaves as expected.
Remove the previous load balancer.
Move the traffic back from the load balancer to the cluster.

Some of the steps were mostly automated on our CI+CD pipeline (deployments, creation of the target groups, etc.), but some required us to work with the AWS CLI / Web console.

Denormalizing the core of Playtomic

Sun, 25 Sep 2022 10:00:00 GMT

Yes, another post about optimization and MongoDB. I started this post in July, and I couldn’t find the time to finish this. I was optimizing something O:)

Our core product at Playtomic is what we call Matches. A match is an entity that represents when, where, and who plays a game. That is the date and time, the location, and the players.

That data structure is stored in a MongoDB collection. At the moment I am writing this, that collection stores 27M documents and takes 100GB of data, 32GB of disk, and 14GB of indexes. It’s one of the biggest collections that I have managed in a transactional system consumed online by thousands of users at the same time (we have 800k active users).

At Playtomic, there are two main types of users: players and clubs. Broadly speaking, players want to find people and book a court. Clubs want to manage their courts and occupancy.

Focusing on matches again, that collection is both consumed by players and clubs in several ways:

Players looking for matches to join.
Players checking their history.
Players uploading results.
Clubs checking their current schedule.
Clubs checking their history.

This is a screenshot of what you can find on Playtomic’s application:

As you can imagine, there are many access patterns to that collection. For example, I can search matches that I can join filtered by

club
location (coordinates)
my levels
number of players
court type (indoor vs outdoor, materials of the walls, …)
gender of the players

And if I cannot join due to some restrictions, I can request to join and wait for the approval of the rest of the players. I don’t want to bore you, but as I said, there are too many ways to access the same data.

There is only one way to make those queries efficient: indexes. But indexes come with a cost:

The more indexes you have, the slower the modifications are (insertion, update, or deletes).
Those indexes must be in memory to be efficient: more indexes, more memory needed.
As the collection grows, changing or creating indexes becomes a delicate matter: you must read the whole collection to create it. In a production system, that might entail problems if you are already overloaded.

Our product grew fast and in a slightly chaotic way. We had many indexes that covered the most common queries, but we could not have an index for every combination of parameters we allowed in our search. And thus, this happened:

In summary: we were dead. MongoDB had not enough resources to reply.

When we received many queries at the same that were not fully covered by our indexes… MongoDB suffered. Remind that our collection was 32GB on disk. If you want to read more about MongoDB Performance, I wrote a post about it in February.

We needed a fast solution that didn’t require completely reworking the API we were exposing. We realized something: the most complex patterns were always to find public matches in the near future. A public match means that everyone can see and search for those matches. Private matches are not searchable.

This was the idea: to denormalize that data into a new collection. The source of truth would still be the original collection. We would only have the data that we need to feed the indexes, the rest of the data is not there. We apologized to all the teachers that taught us the 3rd normal form at college and then we got our hands dirty.

Luckily all our platform was already event-driven, so we could update that denormalized collection via Kafka events. Data would expire using TTL indexes.

Our clubs generally allow opening public matches within 2-3 weeks. That means that we keep between 14 and 21 days of data. We allow players to upload their public matches, but their number is still below the number of matches that clubs manage.

To be sure that we are always returning the most recent data, we use the $lookup command to read at the final step the last version of the match in the original collection.

New collection: 7.7k documents, 5.30MB of data (!!), and 2.25MB on indexes.

I dont’t keep the screenshot of IOPS and Tickets Available after we made the change, or the reduced latency we experimented with in our API, but the improvement was huge.

Since then, this is how our Tickets Available chart looks: Still not 100% perfect, but much better than we were before!

How to peer two VPCs and access it using a VPN

Tue, 21 Jun 2022 10:00:00 GMT

Although I have some experience, I am not a sysadmin. So this might be obvious for a sysadmin or someone with an AWS certification but not for a developer O:)

We had 2 VPCs that are connected through the Internet. We wanted to peer them both so the traffic goes internally within the AWS region. Reasons?

Efficiency: it is faster to use the internal network.
Security: the traffic does not leave the data center.
Price: AWS charges per data transfer.

We access one of the VPCs using a VPN server (specifically an OpenVPN). The other VPC is managed by a provider.

This post applies to any VPCs, but our setup was:

AWS VPC with our docker cluster.
Atlas VPC with a MongoDB cluster.
Traffic goes through the Internet.

As we are connecting an Atlas VPC to an AWS VPC, I am going to use those names (although they both were in AWS of course).

Steps to connect them:

Peer the VPCs
Route the traffic from one VPC to another
Route your clients through the VPN

Peering the VPCs:

You have to start the process from one of the VPC. In our case, we started it from Atlas. You have to provide the other vpc id, the region, and VPC CIDR. For example vpc-1aa12345, eu-central-1, 192.16.8.0.0/16
You have to make sure that you have DNS hostnames enabled in the target VPC. That makes the DNS of your VPC resolve internally the hostname of your cluster.
Now in your target VPC under AWS > VPC > Peering connections you will find the peering to Atlas.
Now you may secure your Atlas cluster allowing access only to some target groups in your AWS VPC.

With this configuration, every host you have deployed in your AWS VPC will have access to the Atlas VPC (and vice versa). The magic happens at DNS level: If you resolve one of the hostnames from Atlas locally, you will get a public IP. If you do it from within VPC, you will get an internal IP.

Accessing the second VPC from the VPN

Our OpenVPN server was configured to access the AWS VPC. Our clients connect to that VPN server and then they route all the traffic to that VPC through the VPN server. If you are wondering how the OpenVPN client works, you should check your network interfaces :) You will find that the traffic to the internal IPs is routed through that interface.

For example, in macOS you can find those routes:

❯ netstat -rn |grep utun16
192.168     11.7.0.20          UGSc           utun16

So… how do we make our VPN route traffic to the new peered VPC? Just add the rule to /etc/openvpn/server.conf. You must have a previous rule there O:)

# This is AWS VPC
push "route 192.168.0.0 255.255.0.0"

# This is Atlas VPC
push "route 172.31.0.0 255.255.0.0"`

Now the key. How do we make our clients resolve the hostnames from the peered VPC? Just use the internal DNS from the VPC. This is where the hostnames are registered.

# Add AWS VPC DNS server so we can use internal mappings
push "dhcp-option DNS 192.168.0.2"

This overwrites our local /etc/resolv.conf to use that DNS server. If you want to add more DNS servers just in case the client is not able to restore back your configuration, you can add a fallback rule (for example, use the Google DNSs or any other public server)

push "dhcp-option DNS 192.168.0.2"
push "dhcp-option DNS 8.8.8.8"

Disclaimer: Those are not our real IPs. You should also check what’s your internal DNS, don’t assume it’s the IP that ends in 2.

Dockerized Multi-arch Github Actions Runner

Tue, 07 Jun 2022 10:00:00 GMT

Do you want to run the Github Action runner as a container? Do you want to run it on your Docker or Kubernetes cluster and scale the number of runners easily? This is your post :)

We are going to the point here, just a few explanations. If you want to know further, you have a few links at the bottom.

Dockerfile

#syntax=docker/dockerfile:1.3-labs
FROM ubuntu:20.04
ARG TARGETARCH

ENV RUNNER_VERSION=${RUNNER_VERSION:-2.291.1}

RUN apt-get update \
    && apt-get install -y --no-install-recommends \
    ca-certificates \
    curl \
    libicu-dev \
    netcat \
    jq

WORKDIR /actions-runner
RUN <<EOT 
    export ARCH=${TARGETARCH}
    # When the arch is amd64, the Github runner binary to use is x64.
    if [ "amd64" = "$TARGETARCH" ]; then
        export ARCH="x64"
    fi

    curl -o /actions-runner/actions-runner-linux.tar.gz -L \
        https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-${ARCH}-${RUNNER_VERSION}.tar.gz 
EOT

RUN tar xzf ./actions-runner-linux.tar.gz
RUN rm ./actions-runner-linux.tar.gz

COPY entrypoint.sh .
RUN chmod -R 7700 entrypoint.sh

RUN adduser --no-create-home github -uid 1001
RUN chown -R github:github /actions-runner

USER github

ENTRYPOINT ["/actions-runner/entrypoint.sh"]

The keys are the ARG TARGETARCH that the docker build command will provide and the checks when the architecture is amd64.

docker-compose.yml

version: '3.5'

services:
  github-action-runner:
    build: 
      context: .
      dockerfile: Dockerfile 
    image: ${DOCKER_REPOSITORY:-nexus.playtomic.io}/playtomic-github-action-runner:${VERSION:-latest}
    environment:
      - GH_ORGANIZATION=syltek
      - GH_ACCESS_TOKEN=${GH_ACCESS_TOKEN}
    logging:
      options:
        max-size: 10M
    deploy:
      placement:
        constraints: [node.role == worker]
      replicas: ${REPLICAS:-1}
      update_config:
        parallelism: 1
        failure_action: rollback

entrypoint.sh

#!/bin/bash
set -e

cleanup() {
  GH_RUNNER_TOKEN=$1
  echo "Removing runner with token ${GH_RUNNER_TOKEN}..."
  ./config.sh remove --token ${GH_RUNNER_TOKEN}
}

# GH_ACCESS_TOKEN must be a token with admin:org permissions https://github.com/settings/tokens/
if [ -z ${GH_ACCESS_TOKEN+x} ]; then
  echo "GH_ACCESS_TOKEN environment variable is not set"
  exit 1
fi

if [ -z ${GH_ORGANIZATION+x} ]; then
  echo "GH_ORGANIZATION environment variable is not set."
  exit 1
fi

# Generate a runner token from our Github Access Token. Every runner needs a new token.
GH_RUNNER_TOKEN=$(curl -sX POST -H "Authorization: token ${GH_ACCESS_TOKEN}" https://api.github.com/orgs/${GH_ORGANIZATION}/actions/runners/registration-token | jq .token --raw-output)
echo "Using the following token to register the runner: ${GH_RUNNER_TOKEN}"

# Traps are here to unregister the runners on docker stop, but they are not being triggered.
#trap 'cleanup ${GH_RUNNER_TOKEN}; exit 143' SIGTERM
#trap 'cleanup ${GH_RUNNER_TOKEN}; exit 2' SIGINT

cleanup ${GH_RUNNER_TOKEN}
/actions-runner/config.sh --runnergroup Cluster --url https://github.com/$GH_ORGANIZATION --token $GH_RUNNER_TOKEN --replace
/actions-runner/run.sh

docker buildx build --platform linux/amd64 .
docker buildx build --platform linux/arm64 .
GH_ACCESS_TOKEN=your-token docker-compose up

Inspiration:

Modernizing a CI+CD pipeline with Github Actions

Mon, 06 Jun 2022 10:00:00 GMT

Our CI+CD has been working for 5 years long. You know, if it ain’t broken, don’t fix it. But the company is not the same. It’s time to update it!

Let me wrap up a bit our current setup. I am going to be brief, I promise. We have a Jenkins cluster on-premise. That is, we manage (and maintain) a bunch of hosts that run Jenkins slaves and a host that runs the master. They are within our on-premise VPN, a reminiscence of our first hosting provider.

Our production backend runs a containerized system on top of a Docker Swarm cluster. Our container registry is Nexus, which allows us to deploy our services in the Swarm.

Both systems run within their own independent VPNs.

Our code repositories are in Github. Our Jenkins is listening for changes in Github. When we merge to develop/production, Jenkins pushes the image to the Nexus, connects via ssh to the managers of the cluster and runs the deployment command (docker stack deploy). We have a common Jenkins pipeline for all our services.

Pros:

Simple.
Security on our side (VPN certificates + ssh keys are in our servers).
Jenkins is commonly known.
Stable: we haven’t had to change it a lot all this time.

Cons:

We need to maintain and keep update the Jenkins machines.
As the team grows, the Jenkins cluster has to grow too to be able to run more jobs.
We have only configured Java 8 and 11.
Still on our old hosting provider.

We have been pretty happy with this setup so far. Maintenance is something that we always want to simplify at Playtomic. The fewer systems, the better. Besides, new versions of Java require new versions of the JDK, maven and thus… Jenkins. We have already been using Github Actions in other projects here, so that we know that it could be a fine replacement of Jenkins. The workflow defines the environment it is required to run (for example, the java version or the architecture), so it makes complete sense to use Github Actions.

Replacing Jenkins by Github Actions

We re-wrote the Jenkins pipeline as a Github Actions workflow. Our main concerns are two:

What about security? We are using ssh-keys to access the cluster. We can use Github secrets to store them, but we don’t like having such an important piece of the security stored in a third-party.
Cost might be a problem in the future. Github Actions are charged by the minute of computation and we have 50 services.

We discovered that Github Actions allow you to run self-hosted nodes… so that’s the solution to both problems. At this very moment, we can afford the minutes that we are spending, so we are adding a host just for deployments. We added a t3.nano, which is pretty cheap. ssh-keys are still 100% under our control, as they are installed in the machine.

Pros:

Still a common pipeline (workflow) but allows everyone to use new stuff (new java version, different languages, …) without installing more tools in Jenkins.
Security is still on our side.

Cons:

We need to monitor the cost.

What’s next?

Containerized Github Action runner

Worried about the cost? If you have a cluster, you can run as several copies as you want of the Github Action runner!

On-demand self-hosted runners

This would be a huge improvement to control the cost while still being able to scale the number of runners: https://github.com/machulav/ec2-github-runner

Separate the CI from the CD

To separate pipeline for the CI and CD is a good idea, as their lifecycles are pretty different. In our current setup, the workflow is responsible for running the deployment command. If that fails, then the whole pipeline fails, but the build was successful.

We have already tested ArgoCD in Kubernetes so that the CD is handled by the cluster itself. If you are running Kubernetes, you can already do that with FluxCD or ArgCD

Replace Nexus by Github Packages

Nexus requires a lot of space (as it stores all the containers, libraries, packages, … of your organization). We don’t want to maintain that space, be responsible of the backups, … so that we are considering to migrate to Github Packages too.

Performance and MongoDB

Tue, 15 Feb 2022 10:00:00 GMT

If you, a backend developer, had to describe your job, what would you say? We usually talk a lot about servers, clusters, layers, algorithms, software stacks, memory consumption, …

We put data into databases and we get it back as fast as we can. Databases are our cornerstone. Why don’t we talk more often about them? We are always relying on our ORMs.

My best advice? Simplify your queries. Simplify your data models. Simplify your access patterns.

Performance in MongoDB

Metrics

If you are using MongoDB, these two metrics will be your best friends:

Scanned Documents/Returned ratio
IOPS

Scanned docs/returned: it means how many documents you are reading from disk vs the number of documents you are actually returning in your find() or aggregate(). Ideally: this should be 1 (every document read is returned). The only way to get it? All your queries must be covered by indexes. Indexes are in memory (or they are if they fit), so MongoDB doesn´t have to read and filter them from disk.

IOPS: I/O operations per second. That is, operations on disk. It is correlated with scanned docs, as they are read from disk. But there are more sources of IOPS, for example writes. Your disk will give you a limit to your maximum IOPS. Ours is 3000.

Your goal is to keep IOPS below that threshold, and as low as possible. It is hard to know how many IOPS your query consumes, but it is easy to know the scanned docs/returned ratio.

Tools

How can you analyze why your database is behaving as it is?

MongoDB Profiler If you can afford to enable it, do it now. It’s the best source of info. We use the Atlas MongoDB Profiler and it is worth every penny.

Explain planner The planner is the key to understanding your access patterns. There are several ways of calling it, but you can start with explain() after your cursor:

db.our_collection.find(query).explain()

You can use add executionStats to get more data about the query.

db.our_collection.find(query).explain("executionStats")

There are many stages, but these ones are the most important:

COLLSCAN: the query is scanning the collection in disk. Pretty bad, as no index covered the search, so MongoDB has to read the whole collection.
IXSCAN: the query is using an index to filter. It doesn´t mean that all the query is covered by the index, but at least some part.
FETCH: the planner is reading the documents from the collection. If your query is returning documents, you will get a FETCH stage probably (unless your query is covered by the index).

This is an example of one of our queries:

winningPlan: 
      { stage: 'COLLSCAN',
        filter: 
         { '$and': 
            [ { '$or': 
                 [ { 'invitaed_user_id': { '$eq': '1' } },
                   { owner_id: { '$eq': '1' } },
                   { 'player_id': { '$eq': '1' } } ] },
              { is_canceled: { '$eq': false } },
             ] } },

This is not good, as it is scanning the whole collection.

Another one:

db.our_collection.find(query).explain("executionStats")
executionStats: 
   { executionSuccess: true,
     nReturned: 22,
     executionTimeMillis: 0,
     totalKeysExamined: 24,
     totalDocsExamined: 22,
     executionStages: 
      { stage: 'FETCH',
        nReturned: 22,
        docsExamined: 22,
        inputStage: 
         { stage: 'OR',
           nReturned: 22,
           inputStages: 
            [ { stage: 'IXSCAN',
                nReturned: 0,
                indexName: 'example-index-1',
                indexBounds: 
                 { owner_id: [ '["1", "1"]' ],
                   start_date: [ '[MaxKey, MinKey]' ] } },
              { stage: 'IXSCAN',
                nReturned: 22,
                indexName: 'example-index-2'
                indexBounds: 
                 { 'player_id': [ '["1", "1"]' ],
                   start_date: [ '[MaxKey, MinKey]' ] ] },
                keysExamined: 23,
                dupsDropped: 0 }] } } },

Here you can two see IXSCANs, merged by and OR. After that, the query is fetching the documents. I am reading the query inside-out. example-index-1 is used to resolve one part of the query, and example-index-2 for the other part.

Sometimes you will get a FETCH just after an IXSCAN: it means that the index covers only part of the filter. After that, the planner needs to read the documents from disk to finish the filter.

The complete list of stages? You need to check the code.

Simplify your queries

$ors

$ors are the devil. You, as a programmer, are used to thinking in $ors. You add a few $ors, your condition gets much more expressive. But guess what? Your query has gotten exponentially more complex. With every condition you add to the $or, you are adding one more combination of parameters.

How does the planner resolve all those combinations? It needs an index for each of them.

Do you remember the explain() above? The query was:

{
	$or: [{"owner_id: "1"}, {"player_id: "1"}].
}

The indexes used?

example-index-1: {"owner_id: "1", "start_date": 1}
example-index-2: {"player_id: "1", "start_date": 1}

And here is another tip: bigger indexes can cover smaller queries as long as the fields are at the beginning of the index. We are not using start_date to filter.

What would happen if I add an extra $or? { $or: [{“owner_id: “1”}, {“player_id: “1”}]. $or: [{“is_canceled”: true}, “start_date”: {$gt: ISODate(“2022-02-02T00:00:00)}] }

Then there would need 4 combinations:

{owner_id, is_canceled}, 
{owner_id, start_date}, 
{player_id, is_canceled], 
{player_id, start_date},

Denormalized fields

If you find yourself filtering by several fields within an $or, or sorting by several fields, consider adding a denormalized field (based on the others).

Yeah, my apologies to the third normal form.

Keep reading, we will give an example pretty soon using the so-called Summary Pattern.

nulls are always the minimum value

This is a minor trick, but still useful. Let’s say you have a nullable field, and you have to filter (or sort descending) by that field.

You will probably find yourself using a query similar to this:

{
 $or: [
        field: {$exists: false}, 
        field: {$lte: value}
      ]
}

That is, if the field is null, then it passes the filter. Otherwise, compare. We used this filter to check if a player could join a match given their level and the level restrictions of the match.

That’s an $or. We don’t like $ors. What would happen if we compare our value to null? Let’s check the sorting rules in MongoDB.

MinKey (internal type)
Null
Numbers (ints, longs, doubles, decimals)
Symbol, String
Object
...
MaxKey (internal type)

That is, null is always the lower value when comparing (except for the MinKey object, we will talk about it later).

Our query can be simplified by this:

{
  field: {$lte: value}
}

It works for $lt and $lte (lower than and lower or equal). It works too if you are sorting by descending order of field {field: -1} because the object with null will be at the end of the sort.

Counts are costly in MongoDB

Counting seems an easy operation, but it is not. Even if you have an index, MongoDB needs to traverse the index due to the way MongoDB builds B-trees: they don’t store the number of leaves that the sub-tree have. So they need to traverse the index until the end.

Again, if you are counting using a query with $or, it makes the counting even more complex: the query needs to take into account possible repeated documents.

For example, we used counts to compute the position of a player in a ranking (the original query was even more complex).

{ranking_id: ?0}, 
{
 $or: [
		{ value: {$gt: ?1}},
    { value: {$eq: ?1}, last_modified: {$gt: ?2} }
 ]
}

It required several indexes to count:

{ranking_id: 1, value: -1}
{ranking_id: 1, value: -1, last_modified: -1}

How can we avoid counting on several indexes? Add a single field that summarize the fields that you are filtering.

For example: weight = append(value, last_modified) With that new field, we only required one single index: Indexes:

{ranking_id: 1, weight: -1}

This is called the Summary Pattern.

How to build indexes

Ok, indexes are our best tool to keep MongoDB as performant as possible. So the next step, how do we know what indexes we should build?

Performance advisor

If you are in Atlas, use the Performance advisor.

At some point, you will know your system better than the Performance Advisor, but it is a good starting point.

Clone your production collections and explain() it in local

Test your indexes thoroughly before you put them in production: - it takes a lot of IOPS to build them.

once you built it, the planner is taking it into account too, even if it is not finally used.

Remember what we said before:

COLLSCAN: bad.
IXSCAN: good.
FETCH: good if it is the final step. Bad in between.

Others that you will see:

COUNT: ok.
MERGE: ok-ish, you probably could do better.
MERGE_COUNT: good.

By the way, there are blocking and non-blocking stages, meaning that a stage needs to wait for the previous one before it can start computing results.

ESR: Equal-Sort-Range

Have you ever wondered what fields should go first in an index? You need to follow this rule:

Fields filtered by equals ($eq, $in in some cases, …) go first.
Then fields used in the sort stage. Remember that the index has to be built in the same order as you are sorting.
Then fields filtered by a range ($lt, $lte, $gt, …).

ESR is the most useful rule you will find to build indexes. You should read as much as you can about it until you understand it.

This post by Alex Belilacqua is a gem.

Disambiguate equals

If you have two fields that are going to be filtered by $eq, what should go first?

The answer is that it doesn’t matter. You don’t need to worry about having a more balanced tree.

Just keep in mind the ESR rule. If one of them goes in a sort or a range, then it goes the latter.

Rollover process

Building an index is one of the most costly operations. Your IOPS will go nuts. If you need to do that in your production environment, and your collection is big enough, then we recommend you to use a rolling process. It starts the index build in a secondary, then promotes it to primary once the build is finished. You will be able to build any index even when your database load is high.

In Atlas, it’s just one click.

Remove / Hide indexes

The more indexes you have, the worst for the planner. The planner runs the query through the indexes it has and then takes the most promising.

Again, counting when you have several indexes is pretty bad.

Sometimes, you cannot just remove an index in production. You can check using your profiler if it can be removed but you might not be 100%. One not-frequent query might launch a COLLSCAN and then you would miss that index.

Luckily, since MongoDB 4.4 you can hide indexes. We use them to detect what indexes we can remove safely.

Limit your queries

Have you set a maximum limit of time to your queries? Why not? Do your clients have a request time out? Then it can be a healthy practice to avoid unexpected uses of your APIs.

db.our_collection.countDocuments(query, {maxTimeMS: 100})
MongoServerError: Error in $cursor stage :: caused by :: operation exceeded time limit

Do you see all these orange dots? No one was waiting for the backend to reply.

References

Flux vs ArgoCD

Thu, 28 Oct 2021 10:00:00 GMT

I have been trying to find any comparison between FluxCD and ArgoCD that could help me choose between them. I haven’t found any that truly help us at Playtomic (many were just a list of meaningless comments if you don’t know both systems). So… we tried both and made our own decision.

Goals

We are adapting the GitOps practices. After digging into the topic, our ideal setup is:

The cluster (infrastructure + applications) has to be described as code, not as commands.
That makes the cluster easy to replicate (eases the process to set up new environments or the disaster recovery of the existing ones).
As few steps to build a new cluster as possible (how many actions we have to perform manually before the cluster starts running). Spoiler: you can make it with 2 commands: one to build the K8s cluster, another to install all your infra and apps.
The cluster must monitor changes in our repositories and automatically sync them to the cluster.
One repository to describe the infrastructure (cluster repo).
One repository to every service/application we want to deploy in the cluster (applications repo).

These two last items are called multi-tenancy. The goal is that every team can be responsible for the lifecycle of their applications: they can write anything in their repositories, but they cannot alter the cluster infrastructure by themselves. That’s the responsibility of the owners of the cluster repo.

Long story short: you can do everything in that list with both Flux and ArgoCD but there are differences about how they attain that.

Flux vs ArgoCD

Concepts

Flux:

You have to define GitRepositories/HelmRepositories and make Flux monitor them (if you are using Git+Kustomize you will need a Kustomization)

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: service-one
  namespace: playtomic
spec:
  ref:
    branch: develop 
  interval: 1m
  url: https://github.com/your-org/your-repo
  secretRef:
    name: git-credentials
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: service-one
  namespace: playtomic
spec:
  interval: 1m
  sourceRef:
    kind: GitRepository
    name: service
  path: deploy/develop
  prune: true

ArgoCD

You have to install Applications. An Application is basically a repo to monitor. It supports Kustomize, Helm, …
An Application can contain references to more Applications.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: service-one
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default

  source:
    repoURL: https://github.com/your-org/your-repo
    targetRevision: develop
    path: deploy/develop

  destination:
    server: https://kubernetes.default.svc
    namespace: playtomic

  # Sync policy
  syncPolicy:
    automated: {}

Main differences

ArgoCD objects have to be installed in the namespace where ArgoCD is installed (argocd in this example). That is the Application object belongs to the ArgoCD namespace. The actual application can be deployed in any target namespace.
Flux objects belong to the namespace where the application will be actually deployed.
ArgoCD supports defining several projects with different permissions (what namespaces or Kubernetes objects the application in that project are allowed to use).

How to start the cluster

Flux:

You write your cluster using standard k8s objects and Flux objects to indicate what pieces it has to monitor. Dependencies are written as you would write the declarative files in a regular k8s setup.
You have to run flux bootstrap command, and everything is set up and monitored.

ArgoCD:

You have to install ArgoCD (using the regular k8s descriptors).
You have to make ArgoCD “self-aware” after that. That is, you have to install ArgoCD as an application in ArgoCD. You will need a secret to let ArgoCD access your repositories.
Or use ArgoCD Autopilot (which takes care of both steps)

Deploying new versions of your services

You probably have a CI already building your images. You can let that CI commit that version to your repository too (be aware of triggered builds afterward) or you can let Flux/ArgoCD check for new images and do that commit for you.

We wanted to stop our Jenkins CI to deploy actively our services. That’s the CD responsibility and besides, the version deployed was not written anywhere: we had to check our monitor tools or query directly the cluster. So… let’s see what Flux and ArgoCD have to say about this.

Flux

You have to add two extra components when bootstrapping Flux:

flux bootstrap github --owner=your-org --repository=your-cluster-repo --token-auth --path=/clusters/develop --components-extra=image-reflector-controller,image-automation-controller

In every repository where you are storing image versions, you have to add a ImageRepository, a ImagePolicy and a ImageUpdateAutomation.

apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageRepository
metadata:
  name: service-one
  namespace: playtomic
spec:
  image: your-docker-prefix/one-service
  interval: 1m0s
  secretRef:
    name: nexus-credentials
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImagePolicy
metadata:
  name: service-one
  namespace: playtomic
spec:
  imageRepositoryRef:
    name: configuration
  filterTags:
    pattern: '^develop-(?P<version>.*)'
    extract: '$version'
  policy:
     numerical:
      order: asc
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
  name: service-one
  namespace: playtomic
spec:
  interval: 1m0s
  sourceRef:
    kind: GitRepository
    name: one-service
  git:
    commit:
      author:
        email: fluxcdbot@users.noreply.github.com
        name: fluxcdbot
      messageTemplate: '{{range .Updated.Images}}{{println .}}{{end}}'
  update:
    path: /deploy/develop/
    strategy: Setters

ArgoCD

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: configuration-service
  namespace: argocd
  annotations:
    argocd-image-updater.argoproj.io/image-list: service-one=your-docker-prefix/service-one
    argocd-image-updater.argoproj.io/service-one.allow-tags: regexp:^develop-[0-9a-zA-Z\-]+$
    argocd-image-updater.argoproj.io/service-one.update-strategy: latest
    argocd-image-updater.argoproj.io/write-back-method: git:secret:argocd/github-credentials
    argocd-image-updater.argoproj.io/write-back-target: kustomization
    argocd-image-updater.argoproj.io/git-branch: develop
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default

  source:
    repoURL: https://github.com/your-org/your-service
    targetRevision: develop
    path: deploy/develop

  ...

You have several strategies, but here we are setting write-back-target: kustomization so it will write the image versions into the kustomization.yaml in deploy/develop directory.

Main differences

Flux needs two extra components when bootstrapping and then an image-updater per repository whose version is monitoring.
ArgoCD handles everything in the Application. Flux requires a Kustomization and a GitRepository.
Flux’s image updater is a component in your tenant repository. ArgoCD’s updater is configured via annotation in their Application object.
ArgoCD has a web console (with user management). Flux does not. We found that ArgoCD web UI is super useful for not experts to debug the errors in their projects.
Flux bootstrap command install and make the system monitor itself. ArgoCD does not by default, you will need the ArgoCD autopilot.

Dependencies control

Sometimes you need to install some components in Kubernetes before others. In ArgoCD they managed that dependency order as waves. In Flux, by default, it retries until everything is ready (at least in our experience)

Summary

In both systems, it feels like the documentation was written expecting that you have already mastered the topic.
Flux seems more k8s-ish: everything looks like a k8s object. More raw for my taste.
ArgoCD seems more refined: Applications seem better defined.
Flux Auto Update is tricky to configure (you have to configure it in two repositories).
ArgoCD Auto Update seems smoother to set up (it’s a property of the Application).
ArgoCD has a web console while Flux does not: it makes errors debugging much simpler.

How to decide

If you have time, try both and check what adapts best to your case. I know, you don’t have the time and that’s why you are here.

For us, the decisive points are:

The image updater configuration is simpler in ArgoCD.
The image updater in Flux was killing our Nexus with requests: last versions of the updater were unable to query by image name, they have to query everything and then filter the images. As we had several services, one image updater per service, it was too much.
The ArgoCD console lets our team migrate their services and be able to check their deployments easily.
Question of taste: ArgoCD descriptors seem more friendly to me.

My evolution in Playtomic in Github charts

Sun, 15 Aug 2021 10:00:00 GMT

I sometimes take a peek at the Github charts, I find interesting what they speak about.

I have never looked thoroughly at the activity overview. This is the 2-axis chart that shows how your contributions are split. My version for 2021 is surprisingly biased towards the Issues axis (87%). Omg, I have just turned into a manager.

Jokes aside, I have taken a quick look at the previous years. It’s fantastic that I see my evolution and the evolution of my team in Playtomic.

2017 - the only backend developer

This was my first year at Playtomic. The craziest year in my career. I was the only developer dedicated full time to the backend. So most of my work was… writing code.

2018 - we got a team

We started to hire and build our backend team. We were still few, so you can see tons of commit yet, but I had to review my teammates’ PRs as well. I started to act as the head of the backend, but I still had to code a lot.

2019 - the team grows

The exact proportion of commits but the number of reviews went above the number of pull requests opened.

The same number of commits but fewer PRs means that my contributions were either smaller or required fewer iterations to be merged.

2020 - sharing ownership

More people joined the team. We had some huge and complex developments (the new availability and prices engine), so it was too much for me. I had to delegate responsibility. I was involved only in the most profound changes, not every single line of code that went into the system.

That means fewer reviews, more commits.

2021 - the year of the management

This is a depressing year hahaha

87% of my contributions were related to… issues. I am a trouble maker now 😅

Now seriously. We have evolved into Feature Teams. We gave full ownership of the services to the teams, so my time to thoroughly review code decreased again. My main concern is design and architectural decisions that our teams have to implement by themselves.

We have been focused on SRE too, mandatory to let the teams be fully independent.

The path to observability

Tue, 25 May 2021 10:00:00 GMT

o11y = observability = logs, metrics, traces ¹

This post is a summary of the steps we unconsciously followed in Playtomic when digging into the world of SRE (Site Reliability Engineering).

Logs

Most of the people is already comfortable with logs. I have always been a fan of the 12-factors app design, and they have a chapter about logs. Just throw them to the stdout and treat them as a stream. As our services run in docker containers, we just read the logs from docker. With logstash+filebeat you can massage and forward them somewhere else (an elasticsearch, logz, datadog, …)

anemone_configuration.1.ip.eu-central-1.compute.internal    | 2021-05-25 08:44:44.080  INFO [configuration-service,65360f43df5797ee,65360f43df5797ee,3187176603182004136,7845016010104016623,767246] 1 --- [  XNIO-1 task-5] c.p.a.s.IgnorableRequestLoggingFilter    : After request [GET uri=/v2/status/version_control?app_name=playtomic&app_version=3.13.0&device_model=iPhone&os_version=14.5.1&platform=ios;user=one.user@gmail.com;agent=iOS 14.5.1;ms=1]
anemone_configuration.1.ip.eu-central-1.compute.internal    | 2021-05-25 08:44:44.087  INFO [configuration-service,5fe21e342fb8cac9,5fe21e342fb8cac9,4868272858418826990,131548911514375389,767246] 1 --- [  XNIO-1 task-6] c.p.a.s.IgnorableRequestLoggingFilter    : After request [GET uri=/v2/configuration;user=one.user@gmail.com;agent=iOS 14.5.1;ms=8]

Metrics

Metrics are also quite known in the community, thanks to prometheus and grafana. What is most unknown and super useful is to have them correlated somehow. Probably via tags/attributes to identify hosts, containers, environments, …

Traces

Traces are probably the newest in the trio.

In Playtomic, we followed the standard path: we started with in-house ELK (for logs) and prometheus (+ grafana). But we (developers) hate maintaining the infra: we run out of space, indexes get corrupted, … it is quite distracting. Therefore, we got rid of these in-house versions and moved those services to Datadog.

At that moment, we were already using distributed tracing to inject the trace and spain ids in our logs, so we can keep track of the requests among our microservices. So, with the Tracer already in place and the use of Datadog, the next natural step was using Trace to know more about our platform.

What’s next?

This SRE world is huge. Where are we heading to?

SLIs, SLOs, SLAs

Once you have your data (specially metrics), you can monitor your system. Welcome to the world of SLIs, SLOs, and SLAs.

SLI: Indicator (Service Level): the number (for the tech team). Example: error_rate = % of requests with code >= 500.
SLO: Objective (Service Level): the target (for the tech team). Example: keep error_rate < 99%
SLA: Agreement (Service Level): the agreement (with the customer). Example: keep error_rate < 95%

SLOs are great because it gives you a hint when to spend more time in stability (paying technical debt, investing time researching problems, …).

RUM: Real User Monitoring

One step further: what if you could connect traces in your clients with your traces in your server? That’s where we are at this moment.

Image: Ted Young https://twitter.com/tedsuo ↩

Money operations are not as easy as you think

Mon, 10 May 2021 10:00:00 GMT

I see this a lot in Stackoverflow and most of the replies always makes me wonder what kind of software is running over there. People ask about operations with currencies (that is, money) and they usually get replies about using doubles and then rounding.

Big no! Floats and doubles don’t represent exact numbers. They represent a number at certain precision. 101 in any coding course. If your application handles prices (and its probable if you are charging for some service), you can’t use float/double. Your language probably have a way to represent exact numbers. In Java we have BigDecimal and… Money!

Money price = Money.of(10.50, "EUR");
MonetaryOperator roundingOperator = MonetaryOperators.rounding(RoundingMode.HALF_UP);
Money newPrice = price.multiply(61).divide(3).with(roundingOperator);

You will be thinking: why so much trouble to do a simple ceil(10.50 * 61 / 3)?

What happens to decimals? What happens to cents? Do you know that there are currencies that don’t have the equivalent to cents (for example, the yen)?

There are a lot of subtleties you just don’t know when handling money (as happens with operations with time!). I don’t know you, but if some people have already taken their time figuring all those things out… I feel safer using their knowledge!

If your language does not have something similar, then… you’ll have to stick to integers and handle with care how many decimals your currencies need. For example, if you want to represent 15.53€, you can handle it as an integer 1553, but you will have to think what happens with rounding in your use cases.

PS: Of course, if you are performing simulations (for example, Montecarlo) or other mathematics stuff, although they represent money, you won’t probably need to apply any of the things I said before. You are already dealing with other precision problems! :)

Don't use default values in your database

Mon, 12 Apr 2021 10:00:00 GMT

This is a bad practice in my opinion if your database is owned by one application (or service). It is the usual scenario when using microservices: your service is the only owner of that database. Of course, it might not be your use case.

If several services are writing to the same database, then I can see some good examples where you would want default values. For example, if you don’t have control on the queries that are already writing to it.

Why shouldn’t you use default values in your schema?

Default values are usually defined in your code too. Now you would have these defaults spread in your code and your database.
Default values probably depend on your application flow. When the default is in the scheme, you are probably thinking of one single scenario.
It won’t help you to maintain your code.

When schemas evolve, it is super common to add a not null column. The common practice is to add the column with a default value, which will fill that new column with the default value. But the unexpected behaviour is that don’t have to write that value in your INSERTs anymore. And it is not probably what you wanted.

For example, let’s say we have a table for users and we are writing a functionality for a new type of user. We would probably do something like this in our database:

ALTER TABLE user ADD user_type ENUM('PLAYER', 'COACH') NOT NULL DEFAULT 'PLAYER';

It adds the user_type column and set all the old values to 'PLAYER'.

The robust way to do that is:

ALTER TABLE user ADD user_type ENUM('PLAYER', 'COACH') NOT NULL DEFAULT 'PLAYER';
UPDATE user SET user_type='PLAYER' where user_type is null;
ALTER TABLE user MODIFY user_type ENUM('PLAYER', 'COACH') NOT NULL;

That is, add the column as nullable, set the value, set the not null.

In case that you are altering your database while your code is running (for example, in a high availability environment), your code have to support writing the new column but probably reading null values until all the SQL updates are completed.

That forces you to do two deployments:

First deploy to add the nullable column. New code starts writing the default value but supports to read a null.
Second deploy to update the value in database. After that, the code can assume the value is not null anymore.

More work, I know, but this is the cost of making your database more manageable in the future!

PS: sorry for the typos in advance, I wrote this in 10 minutes!

Changing the log format in Spring Boot

Fri, 12 Mar 2021 10:00:00 GMT

Do you know that hours of trial and error can save you minutes of reading documentation? Well, that was not the case today because I wasn’t able to find that (although it is).

The Internet is full of (crappy) pages explaining how to change log format replacing the original files (logback-spring.xml, …) but we didn’t want to change and write ALL the configuration. We just wanted to add a field to the log.

You can find that param on the official documentation, but it is easily missed.

logging.pattern.level: <your marvelous pattern>

Do you know how I found that? I had to browse Sleuth code to know how they change it.

	@Override
	public void postProcessEnvironment(ConfigurableEnvironment environment, SpringApplication application) {
		Map<String, Object> map = new HashMap<String, Object>();
		// This doesn't work with all logging systems but it's a useful default so you see
		// traces in logs without having to configure it.
		if (Boolean.parseBoolean(environment.getProperty("spring.sleuth.enabled", "true"))) {
			map.put("logging.pattern.level",
					"%5p [${spring.zipkin.service.name:" + "${spring.application.name:}},%X{traceId:-},%X{spanId:-}]");
		}
		addOrReplace(environment.getPropertySources(), map);
	}

So easy once you know it.

Ours now is:

"%5p [${spring.application.name:},%X{traceId:-},%X{spanId:-},%X{user_id}]"

user_id is a variable set in the MDC on every request.

Optional dependencies in Rust

Sun, 22 Nov 2020 18:00:00 GMT

Yes, I’m learning a new language. A typed one =D I love C++. Some of my best code and all my research was done in C++. But I hate its dependency system. I don’t want to deal with them anymore. So here it is Rust, that feels as an appointed heir.

I don’t want to extend this post more than necessary, so I will get to the point. Rust can defined dependencies. Dependencies can be optional. Besides, we can optionally include dependencies or blocks of code using features.

To define a feature, you define it in your Cargo.toml

[dependencies]
rand = "^0.7.3"
ndarray = "^0.13.1"
log = "^0.4.11"
image = "^0.23.11"
imageproc = "^0.21.0"
sdl2 = {version = "^0.34.3", optional = true}

[features]
default = ["display-window"]
display-window = ["sdl2", "imageproc/display-window"]

We are here declaring a feature called display-window that includes the dependency sdl and the crate imageproc/display-window.

To activate that feature, you have to add a cfg macro in your .rs source.

#[cfg(feature = "display-window")]
use imageproc::window::display_image;

fn display_image() {
    let canvas = ...;
    let width = 2048;
    let height = 2048;
    imageproc::window::display_image("Title", canvas, width, height);
}

That code makes the feature display-window active, which enables us to use the imageproc::window space. We don’t see that in this code but imageproc::window requires sdl2 to work, and that’s the reason because we included the sdl2 crate in Cargo.toml.

Profit!

Teaching a computer to draw

Sun, 28 Jun 2020 18:00:00 GMT

There must be hundreds of artists trying to mix art and computing. I have always wanted to try to perform some experiments on this topic. It is interesting computationally speaking (there are tons of math and engineering involved), visually (computed generated art is usually impressive) and conceptually (is it actually art? how we define art?).

This @anastasiaopara’s tweet has inspired me to write this post. It is a project to make computer draw images. She has published her code so that we all can play around with it. I wanted to understand how it works and try my own ideas, so that I ended forking and refactoring her code https://github.com/sgmoratilla/genetic-drawing.

This is not an essay about genetic algorithms or function optimizations. There is a lot to know about these subjects. You can find several introductions to genetic algorithms on the Internet. If you want my advice, read some of them and jump to scholar papers. For example, this thesis of my former investigation partner Rubén (backup here). Part I is about Optimization, Section 2.3.2 about Genetic Algorithms.

Anyway, if you don’t understand something about what I wrote here, feel free to ask: @sgmoratilla

##What are genetic algorithms? To be brief, optimizations are search problems. You have a space of possible solutions, a function whose maximum (or minimum) you want to know. Sometimes you are lucky and your target function has nice properties as being differentiable and smooth, so that you can use some traditional approaches as gradient descent or Newton’s methods. The most interesting problems I’ve worked on weren’t differentiable or their solutions must have to fulfill some restrictions. That’s were genetic and evolutionary (they are not the same) algorithms shine.

##Can a computer paint?

So.. let’s say we want to solve the “problem of painting”.

What is a painting?
It is a succession of brush strokes on a canvas.

How can we model that into a computer?
Let’s say a stroke starts are some point of the canvas (x,y) and goes in some direction.
Painters are able to make a infinite number of strokes depending on how they spin their hands or how much pressure they apply. We could probably model that too, but keep it simple and let’s say that we give to the computer a list of brushes that it can use (brushes as in Photoshop). Painters mix different colors as well. We could model that too, but we are going to simplify assuming only black and white paintings.

In summary, we can model a stroke as:

2 integer numbers x,y that represents where the stroke starts.
1 integer number that represent which brush to use.
1 float number that represent the rotation of tha brush.
1 float number that represent the size of the brush in % of the original brush.
1 integer number that represent the color (0 = black, 255 = white, any number in between is a scale of grey).

We can call that set of 6 numbers a gene. A painting is just a collection of these genes applied in order. For example, let’s say we can draw a painting with 1000 strokes. A solution is a collection of 1000 genes, that is, a collection of 6000 numbers. This is called the genotype in Genetic Algorithms frameworks. Isn’t it magical how can we code the process of painting with these simple numbers?

How good is this solution of 6000 numbers? How can we measure it? We can apply these strokes on an empty canvas. This is our painting. The fenotype. Then we subtract the original image from this painting (pixel by pixel). The sum of the differences represents how different it is from the original one.

Our aim is to minimize that number. How can we do that using a genetic algorithm?

Generate a random population of possible solutions.
Create a children population by mixing those candidates.
Alter randomly some of the genes.
Evaluate how good these solutions are.
Discard the worse ones and keep the best ones.
Repeat several times.

At some point, we stop and choose the best solution. That’s our final painting.

This is how it works on one photo of my own:

##But… is it actually learning?

Sorry for the catchy headline. We are not actually teaching a computer to draw. This algorithm does not learn anything O:) If you think of the algorithm, we have actually programmed how to paint by trial and error given some image as inspiration.

There are thousands of possible optimizations. For example, use fewer strokes (faster iterations) and resetting the algorithm using the last image as seed of the next iteration (a greedy approach).
We are not discussing them here, you can see some of them in my repo.

I know there are several algorithms to transfer style. You can teach a neural network to copy the style of a painter (given a list of painting with the same style) and transfer that style to a image or photo. In my opinion and from an artistic/romantic point of view, I feel that is not painting but applying a filter to the image.

I am not promissing anything but I have plans to write more about this subject soon.

##Links Github Project
Anastasia’s Github

What I see in Github contributions

Mon, 15 Jun 2020 10:00:00 GMT

I have seen several people sharing their contribution graph in Github. Some of them, bragging about how dark and green their graph is. A few were very proud of them not having any blank square. I am not happy with that kind implications: more hours, no rest days, … They don’t mean you are working harder. Just mean that you are working more hours. Besides, more contributions to a repo don’t mean that they are better. Just that you have committed more times.

With that being said, I find this graph quite interesting. I can see my progression during the year: my holidays periods (blank columns), my weekends (blank rows), busier times at work (darker points). I could probably tell some of the functionality we launched seeing the color of the squares of that month.

I am not particulary fond of how green it is. I am specially dispised with the green dots on the first and last rows: they are commits on weekends. And that means hotfixes: that is, something that was wrong in production and needed to be fixed right now. I count 3 this year. It’s not a high number considered the pace of development and how many developers we are in Playtomic. But yikes.

That number has to be zero.

Cloud gaming

Wed, 20 May 2020 10:00:00 GMT

This is not a technical entry, but I still think it is worth to be written.

I used to be a hardcore gamer. Hundreds of titles, thousands (I’m not exaggerating) of hours in Team Fortress 2. I used to compete. But life is life, and my time in games was being reduced and I didn’t want to maintain a machine with Windows (for gaming and photography) and Linux (for everything else - work, study, …). So that, I jumped into Mac, said goodbye to video games and hello to Netflix.

But here we are, spending so many hours at home, getting bored of passive activities as watching TV shows or movies and missing gaming so much. But with neither PC nor video console.

So… what options do people like me have to enjoy games these days? These options are still valid for people with a PC but who don’t want to spend in upgrading it.

Google Stadia

The most known probably. Stadia has a subscription plan which gives early access to the platform and some complimentary games month after month. Apart from that, you have to buy your gamers in Stadia. This is the main handicap, because we usually already have a collection in some other platform (Valve’s Steam, Epic Games, …).

Current price: 10€/month

Geforce Now

This is my current winner. Geforce Now is partial compatible with Steam, Epic and Uplay. I say partial because no all their catalog is available. They take in and out games from Geforce Now every week without prior notice. So that it might be a bit annoying if you don’t find a game that your expected to be there..

Current price: ~5€/month

Shadow.tech

I’m looking forward to try this in Spain, but I think this will be the clear winner. They give you access to a cloud machine with a GPU. They have several levels of computing power being the basic one pretty competitive (they promise full hd gaming) while the advance one seems a bit overkilling (4K and Ray Tracing).

You can run anything on that machine: Steam, Epic, your own DRM-free games, …

Current price: 15€/month

Your own cloud

Shadow.tech gave me this idea. What about setting up your own compute in the cloud? Doable but a bit expensive.

I tried Paperspace and AWS. In both cases, you have to ask access to the GPU instances.

In the case of Paperspace, you have to pay a fixed price for the storage and the fixed public IP to access the computer from the Internet. This is a fixed price of 10€/month. Then you have to pay for the running instance in a hour basis, which is 0.5€/h for the basic GPU one. If you compare it to Shadow.tech, you have 10h/month of game for a similar price.

With AWS, it is much more expensive, but you have more options of instances and storage to combine. A similar machine to the Paperspace’s one is 3$/h. Not including the storage and network transfer.

Steam, Parsec, …

In case that you have remote access to a PC, you can always stream from that computer. Steam has a streaming mode (easily accessed via Big Picture mode), but you can use any other control system, such as Parsec.

Parsec is such a cool tool, I encourage you to try it.

PS Now and Xbox XCloud

I’m adding these options just for the sake of mentioning them, but I couldn’t test any of them. PS Now is only available for PC and Xbox XCloud is there only for mobile phones.

They look promising too. Besides, they give you access to the catalog of Playstation (which have some nice exclusives) and Xbox (fewer exclusives as they end in PC as well).

Conclusion

My bet: Parsec > Shadow.tech > Geforce Now > Stadia.

Choosing a documentation system

Tue, 19 May 2020 10:00:00 GMT

Documentation is always the victim when developing a system. We will do it tomorrow. And tomorrow…

I’m writing this because we had a deep discussion about this subject in our team, and I would like to share what options we have checked and why we liked or disliked them.

Our aim is to build a documentation nice to read (e.g. Stripe’s doc) but also nice to write (and this is why we are here).

From our previous experience, tools that automate the conversion from the code (such as Swagger or Javadoc) are easy to write, but frequently written carelessly. They turn out to be neither easy to read nor useful for the final reader.

Our requirements are:

We have several services, so we need an easy way to join all that documentation in our single site.
That single site must look like a single piece of documentation, not a glued bunch of several documents.
Easy to navigate and consume: we want to use it as an introduction to the APIs for the newcomers.

Third-Parties systems

At the beginning, we tried to use existing solutions.

Main advantage: you don’t have to invest your own time building the system. Major drawback: your documentation is locked in that system. Not very customizable.

Postman API / Collections:

Adding requests to collections is easy. Writing details in them is not so easy. Besides, it is a bit dark what part of the documentation you have to change after making changes to the service if you are not an expert on that service.

Postman API can be written in OpenApi format but there is no HTML view for them.

Static documentation for collections is nice and readable, but there is not a main documentation that acts as entry-point for the documentation.

No useful version control. Documentation locked in Postman.

Apiary:

It uses API Blueprint format, which is nice to write. From our experience, it’s a hell to maintain (no version control, what happens with multiple people editing the single document).

A bit pricey. Documentation locked in Apiary.

In-house / custom systems

So that, we gave up trying to use someone else’s system.

Slate (plain):

Generates static sites from markdown. Simple, a bit ugly by default, but quite tuneable. We can deploy the static site in Github pages (free!).

Main drawback: writing schemas in tables are a bit unreadable, because the description of some fields may be a bit long. And I don’t want to write tables/html, I want to write documentation.

Slate + Custom tunning (e.g. Monzo’s):

Prettier but it will be difficult to merge changes from the original Slate repo (although I’m not sure whether there will be changes often).

Main drawback: it is nicer but I feel they are “cheating” in the markdown, as they are including html tags to make it prettier.

For example, request/response parameters are tables of two columns, but the required keyword is in a span tag written in the first column of the markdown. That’s a bit trickier to write. If we don’t like that format in the future, we have to change all markdown files.

Slate + OpenApi converter:

What we liked about OpenApi is that it is a standard format and we didn’t have to care how it is displayed. On the contrary, we didn’t have a nice way to convert from openapi files to a static site.

Given that, our approach has been to write schemas and endpoints in OpenApi format (we already had a lot of this work done in Postman APIs), automatically transform them to markdown, then use slate to generate the documentation.

OpenApi-Generator and Widdershins allow us to use templates to transform the .yml into .md. You can choose to transform schemas into tables (like Monzo) or nested div (like Stripe). If you dislike that format in the future, we only have to change that template and rebuild the project.

Service description will be written in plain markdown.

For example, payments documentation might consist on two files:

payments-section.md: Manually written introduction to payments, their lifecycles, details or highlights, …
payments-api.yml: OpenAPI file describing schema and endpoints.

In order to automate the building of this site, every *-api.yml is transformed into a *-api.md after pushing. Then Slate site is built.

Pros:

Easy to write (markdown for descriptive text, openapi for schemas and requests).
Easy to change (templates to transform openapi to markdown, css and layout.rb to change Slate)
Easy to deploy (widdershins + slate + github page)

Cons:

Maybe too many steps that can fail (widdershins + slate)

Conclusions:

No free lunch. You have to spend your own time building your documentation site if you want happy API consumers and happy developers writting it.

You can check a simplified version of Playtomic’s solution at (https://github.com/sgmoratilla/openapi-slate-doc-example).

A way of testing code based on time in a Spring Boot Application

Fri, 20 Dec 2019 20:00:00 GMT

Every time I see a Instant.now() / new Date() / something that creates a date based on the current time I tremble. How do you expect to test that in a simple / coherent / easy to follow way?

When I see that code, I usually see tests using now() + duration to check that everything works. So that the test is not the same when run today than when run tomorrow.

Wouldn’t it better to be able to “fix” the time and test your code with exact values of times/periods as well?

So that I’ve fighting in my team in order that everyone uses a ClockProvider to get the “current” time. That way I can instance my own ClockProvider in unit tests and I can override the default ClockProvider of my application in integration tests.

The former is pretty easy to understand, so that I’m not writing an example of that. This is an example of the latter.

For instance, my application would look like this:

@SpringBootApplication
public class BookingApplication extends AbstractAnemoneApplication {

    @Bean
    @ConditionalOnMissingBean(ClockProvider.class) // to allow tests to overwrite it
    public ClockProvider getClockProvider() {
        return () -> Clock.systemDefaultZone();
    }

    // An example of method using ClockProvider
    boolean isItJanuary(@Nonnull ClockProvider clockProvider) {
        return clockProvider.getClock().instant().atZone(UTC).getMonth() == Month.JANUARY;
    }

And my IT tests:

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT, classes = {
    FixingClockConfiguration.class, BookingsApplication.class})
@ExtendWith(SpringExtension.class)
public class BookingsApplicationIT {

    @TestConfiguration
    static class FixingClockConfiguration {

        @Bean
        public ClockProvider getClockProvider() {
            ZonedDateTime fixedInstant = ZonedDateTime.of(2019, 04, 15, 14, 00, 00, 00, UTC);
            return () -> Clock.fixed(fixedInstant.toInstant(), fixedInstant.getZone());
        }
    }

// ... your tests based on that date
}

Concurrency in Spring's StreamListener and Kafka

Tue, 17 Dec 2019 10:00:00 GMT

Another too fast, too furious post. I have spent a few hours trying to make my event processor multi-threaded, and it’s so damn easy that I don’t want anyone to spend more than a few minutes in this. I couldn’t find it anywhere on Internet, so I share it here.

We are using the Spring Cloud Stream layer to configure our Kafka consumers.

For example, a configuration for a processor named ‘reservations-input’ connected to a Kafka topic ‘reservations-topic’ would be similar to this:

spring.cloud.stream:
  bindings:
    reservations-input:
      content-type: application/json
      destination: reservations-topic
      group: consumer-service-group

And your class to start processing those events:

@EnableBinding({MessagingConfiguration.ReservationTopic.class})
public class MessagingConfiguration {
    public interface ReservationTopic {

        String INPUT = "reservations-channel";

        @Input(INPUT)
        SubscribableChannel input();
    }
}

@Service
public class ReservationProcessor {
    @StreamListener(MessagingConfiguration.ReservationTopic.INPUT)
    public void handle(@Nonnull Message<ReservationEvent> reservationMessage) {
        // your stuff
    }

Easy peasy. Only problem here is concurrency.

If you have used Kafka before, you would know that the number of partitions in your topic limits the concurrency. Each partition have 1 single consumer.

I don’t know whether (and where) I read that, but I assumed that my application would generate as many threads/consumers as partitions my topic has. But I was wrong. By default, Spring’s only generates 1-threaded processor.

Solutions? Get more instances of your application or configure ConcurrentKafkaListenerContainerFactory to be able to throw more threads (see https://docs.spring.io/spring-kafka/docs/2.3.x/reference/html/#container-factory).

Option 1: create your own instance of ConcurrentKafkaListenerContainerFactory.

The only hint I found in the documentation or stackoverflow was to instance a bean of type ConcurrentKafkaListenerContainerFactory.

    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, Object> kafkaListenerContainerFactory(
        @Nonnull ConsumerFactory<String, Object> consumerFactory) {

        ConcurrentKafkaListenerContainerFactory<String, Object> factory = new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory);
        factory.setConcurrency(5);

        return factory;
    }

I am not very prone to creating my own beans to configure stuff that seems too obvious. It is easy to overwrite some Spring default values that I am already using, it is more code to maintain…

There has to be a way through configuration.

Option 2: use configuration

Getting back to configuration, what we write under spring.cloud.stream.bindings.channel-name.consumer ends in the configuration of Kafka. Therefore, I tried to configure the property concurrency. That is:

spring.cloud.stream:
  bindings:
    reservations-input:
      content-type: application/json
      consumer.concurrency: 3
      destination: reservations-topic
      group: consumer-service-group

Starting our application, we see that we have 3 binders.

    December 17th 2019, 14:22:57.274	2019-12-17 13:22:57.274  INFO [consumer-service,,,] 1 --- [container-1-C-1] o.s.c.s.b.k.KafkaMessageChannelBinder$1  : partitions assigned: [reservations-topic-1]
	December 17th 2019, 14:22:57.259	2019-12-17 13:22:57.259  INFO [consumer-service,,,] 1 --- [container-2-C-1] o.s.c.s.b.k.KafkaMessageChannelBinder$1  : partitions assigned: [reservations-topic-2]
	December 17th 2019, 14:22:57.256	2019-12-17 13:22:57.256  INFO [consumer-service,,,] 1 --- [container-3-C-1] o.s.c.s.b.k.KafkaMessageChannelBinder$1  : partitions assigned: [reservations-topic-3]

Profit!

BlockingQueue and ExecutorService

Mon, 16 Dec 2019 10:00:00 GMT

This is a quick and dirty post, but I promised I would publish everything I research at Playtomic.

Last week, we were having a discussion about how to limit how many tasks an ExecutorService can enqueue. We were trying to control how much memory a service can handle before it throws out of memory exceptions. This service accepts messages from both a Kafka topic and from an API. Those operations end in the same internal logic, which is threaded.

There is a kind of Queue, BlockingQueue, that can wait until a spot in the queue is free. It would seem that using an ExecutionService with a BlockingQueue would wait when submitting a task until that queue is not full. But it is not, the ExecutionService rejects the task.

You know that hours of trial and error can save you hours of reading the manual 😉. I’m proud to say that I have read the manual first this time.

This test shows what happens:

public class BlockingQueueExecutorServiceTest {

    @Test
    public void submitTest() {
        // Worst case scenario: accept only 1 thread in the queue.
        int nThreads = 1;


        ExecutorService exService = new ThreadPoolExecutor(
            nThreads,
            nThreads,
            0L,
            TimeUnit.MILLISECONDS,
            new LinkedBlockingQueue<>(nThreads));


        // Full this with tasks
        for (int i = 0; i < 10000; ++i) {
            WaitingTask t = new WaitingTask(i);
            exService.submit(t);

        }
    }

    private static class WaitingTask implements Runnable {

        int index;
        public WaitingTask(int index) {
            this.index = index;
        }

        @Override
        public void run() {

            try {
                log.info("Running task {}", index);
                Thread.sleep(1000);
            } catch (InterruptedException e) {

            }
        }
    }
}

java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@550dbc7a[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@4dbb42b7[Wrapped task = com.playtomic.anemone.matchmaker.service.BlockingQueueExecutorServiceTest$WaitingTask@66f57048]] rejected from java.util.concurrent.ThreadPoolExecutor@21282ed8[Running, pool size = 1, active threads = 1, queued tasks = 1, completed tasks = 0]

	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2055)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
	at com.playtomic.anemone.matchmaker.service.BlockingQueueExecutorServiceTest.submitTest(BlockingQueueExecutorServiceTest.java:28)
... more boring stacktrace

If you want to wait until queue is not full, you have to provide a RejectedExecutionHandler which does that. For example, Spring’s CallerBlocksPolicy.

public class BlockingQueueExecutorServiceTest {

    @Test
    public void submitTest() {
        // Worst case scenario: accept only 1 thread in the queue.
        int nThreads = 1;

        CallerBlocksPolicy policy = new CallerBlocksPolicy(10000); // 10secs
        ExecutorService exService = new ThreadPoolExecutor(
            nThreads,
            nThreads,
            0L,
            TimeUnit.MILLISECONDS,
            new LinkedBlockingQueue<>(nThreads), 
            policy);


        // Full this with tasks
        for (int i = 0; i < 10000; ++i) {
            WaitingTask t = new WaitingTask(i);
            exService.submit(t);

        }
    }

    private static class WaitingTask implements Runnable {

        int index;
        public WaitingTask(int index) {
            this.index = index;
        }

        @Override
        public void run() {

            try {
                log.info("Running task {}", index);
                Thread.sleep(1000);
            } catch (InterruptedException e) {

            }
        }
    }
}

And this time we get:

12:21:32.409 [pool-1-thread-1] INFO com.playtomic.anemone.matchmaker.service.BlockingQueueExecutorServiceTest - Running task 0
12:21:32.422 [main] DEBUG org.springframework.integration.util.CallerBlocksPolicy - Attempting to queue task execution for 10000 milliseconds
12:21:33.420 [pool-1-thread-1] INFO com.playtomic.anemone.matchmaker.service.BlockingQueueExecutorServiceTest - Running task 1
12:21:33.420 [main] DEBUG org.springframework.integration.util.CallerBlocksPolicy - Task execution queued
12:21:33.421 [main] DEBUG org.springframework.integration.util.CallerBlocksPolicy - Attempting to queue task execution for 10000 milliseconds
12:21:34.423 [pool-1-thread-1] INFO com.playtomic.anemone.matchmaker.service.BlockingQueueExecutorServiceTest - Running task 2

Amazed by the Soyuz

Thu, 10 Oct 2019 10:00:00 GMT

I never thought that one of the things that would impress me the most when visiting New York City, would be seeing a Russian Soyuz. It was at the Intrepid Museum. There was a Soyuz and a space shuttle, the Enterprise (wink wink to Star Trek fans).

The Enterprise is huge. Half the size of a Boing or Airbus. First class in spacecrafts. The workhorse of the NASA. I imagine myself working on that and I drool. The dream of every engineer. They built 5 of them, only 3 survive nowadays. But they are discontinued. Too complex, too expensive to keep them on duty.

Next to the Enterprise there was a tiny metal shell. Something that reminded me of a piece of a submarine that got loose and dropped. A Soyuz. It hardly fits 3 people in it. It can carry a tiny portion of what Enterprise could. But they work. And they are still in service.

That recalled me my work. Sometimes we engineers are so excited about technology that we get lost in the process and miss the final purpose: solve the problem. It’s cool to solve it with the last shinny piece of tech.

But what if simpler, rude solutions are the solution? What if those are the solutions that really last?

A way to unify two authorization methods - Part I

Mon, 15 Jul 2019 10:00:00 GMT

Time to talk about security today. Security is one of the most important aspects of a system, and probably one of the most boring to implement. It is usually divided in two steps: authentication and authorization. In short words, authentication gives access to a system. It is a yes/no question: do I know you? On the other hand, authorization controls permissions (what parts of the system you can access). In this post, I am going to talk about the latter.

As I said in this very blog, Playtomic evolves so fast (do you remember our microservices architecture?). So fast that we ended with two authorization systems: one based on roles and another one based on authorities. The main difference is that authorities define fine-grained permissions, while roles usually define sets of permissions based on the type of the user. For instance, an authority could be “read_users” while a role could be USER_ADMINISTRATOR. We like the authorities because each service is responsible of defining their own authorities. On the other hand, roles are useful because define types of users of the system. They are not defined by the services themselves, but by the whole system.

We could argue why we got there in the first time. However, I learned from two years in a startup that we don’t discuss guilt but solutions. And solutions must be fast to implement and don’t have to be perfect but nearly :)

So, this is our problem:

Two authorization schemes that we want to merge, or make compatible.
Authorization based on authorities simplifies the definition of the access rules to your resources, authorization based on roles eases the access configuration of your users.
There are several microservices already running in production, so that, the fewer the changes, the better. Some use roles, some authorities.
We use JSON Web Tokens (JWT) to move the authentication/authorization through the services. It can contain roles or authorities. So that, at this point we can access some services but not others depending on how the jwt was generated.

Our purpose is to define a “framework” that allows us to write permission rules in a simple way. It is common that your security logic ends scattered all over your code. By writing permissions using authorities in our controllers and services, we aim to separate the type (role) of the user that access from the actual permission it has. By contrast, we wanted to be able to assign roles to the users, because it is easier to understand what type of operations that user should be able to do.

Extra: changes have to be as minimal as possible. It would be preferable if services are independent. That is, they don’t have to call to another one to get the transformation from roles to authorities.

## How authorization was working at this point: Spring security.

When the user is logged, a jwt is generated. All requests will include that jwt as their credentials.
A security filter intercepts all requests, reads the jwt and gets the roles/authorities.
Controllers or services are protected via annotation (@PreAuthorize), via config (HttpSecurity in WebSecurityConfigurerAdapter).
Finer-grained permission (i.e can this user modify this precise resource?) where handled programmatically accessing the Authorization.

Example with authorities and PreAuthorize:

    @PreAuthorize("hasAnyAuthority('admin_venues', 'read_venues')")
    @RequestMapping(path = "/venue/{venue_id}", method = RequestMethod.GET)
    public Venue getVenue(@Nonnull String venueId) {
        return venueService.getVenue(venueId);
    }

Example with roles and HttpSecurity:

   @Configuration
    public class CustomWebSecurityConfigurerAdapter extends WebSecurityConfigurerAdapter {
        @Override
        protected void configure(HttpSecurity http) throws Exception {
            http
                .authorizeRequests()
                .antMatchers(HttpMethod.GET, "/venue/**").hasAnyRole("VENUE_ADMINISTRATOR")
                // ...
                // see JwtAuthorizationFilter below
                .addFilterBefore(jwtAuthorizationFilter, AnonymousAuthenticationFilter.class)
                // ...
        }

## How authorization is going to work from this point:

Each service defines:

A mapper from roles to authorities. Good: independent a easy to implement. Bad: if a new role is added, you have to modify this piece. On the other hand, we could add a service that

knows the mapping, but then we have to add authorities to a role each time a new service appears. Trade-off. Choose your one.

A checker that verifies your access.

/**
 * AbstractJwtAuthorizationFilter reads the jwt and handles the errors when 
 * it cannot be verified or misses some expected data.
 */
@Component
@Slf4j
public class JwtAuthorizationFilter extends AbstractJwtAuthorizationFilter {

    @Nonnull
    private RolesToAuthorities authoritiesProvider;

    public JwtAuthorizationFilter(@Nonnull ObjectProvider<RolesToAuthorities> authoritiesProvider) {
        this.authoritiesProvider = authoritiesProvider.getIfAvailable(() -> new DefaultAnemoneRolesToAuthorities());
    }

    /**
     * User contains user data from the jwt (such as, id) and roles/authorities.
     */
    @Override
    protected boolean handleInternalAuthorization(@Nonnull HttpServletResponse httpResponse, @Nullable User user, @Nonnull String bearer) {
        if (userContext == null) {
            httpResponse.setStatus(HttpStatus.UNAUTHORIZED.value());
            log.info("userContext == null, unauthorized.");
            return false;
        }

        // CurentUserDetails is our implementation of UserDetails
        // By using authoriesProvider, maps the roles/authorities on User to the final authorities.
        CurrentUserDetails currentUser = new CurrentUserDetails(userContext, authoritiesProvider);
        SessionAuthenticationToken token = new SessionAuthenticationToken(currentUser, bearer);
        SecurityContextHolder.getContext().setAuthentication(token);

        return true;
    }
}

public interface RolesToAuthorities {

    @Nonnull
    Collection<GrantedAuthority> toAuthorities(@Nonnull String role);
}

We have abstracted this checker into:

A class that gets the owner of the object being accessed (dependant of the object your are accessing).
A class that checks that your are the owner of the object (generic).

public interface PermissionChecker<T> {
    boolean isAllowed(@Nonnull String permission, @Nonnull Authentication auth, @Nullable T object);
}

@Getter
@AllArgsConstructor
public class ByResourceOwnershipPermissionChecker<T> implements PermissionChecker<T> {

    @Nonnull
    private String resourceType;

    @Nonnull
    private GetOwner<T> getOwner;

    protected boolean isAdmin(@Nonnull Authentication auth) {
        final GrantedAuthority adminAuthority = new SimpleGrantedAuthority("admin_" + resourceType);

        return auth.getAuthorities().contains(adminAuthority);
    }

    protected boolean canOperateOnOwned(@Nonnull String permission, @Nonnull Authentication auth) {
        final GrantedAuthority ownsAuthority = new SimpleGrantedAuthority(permission + "_own_" + resourceType);

        return auth.getAuthorities().contains(ownsAuthority);
    }


    final public boolean isAllowed(@Nonnull String permission, @Nonnull Authentication auth, @Nullable T object) {
        if (isAdmin(auth)) {
            return true;
        }

        if (!canOperateOnOwned(permission, auth)) {
            return false;
        }

        return ownedBy(object, auth);
    }

    boolean ownedBy(@Nullable T object, Authentication auth) {
        if (object == null) {
            return false;
        }

        Object p = auth.getPrincipal();
        if (p == null || !(p instanceof UserDetails)) {
            return false;
        }

        Object ownerId = getOwner.apply(object);
        if (ownerId == null) {
            return false;
        }

        String id = ((UserDetails)p).getUsername();
        // The anonymous case
        if (id == null) {
            return false;
        }

        return ownerId.equals(id);
    }

    public interface GetOwner<C> {
        @Nullable Object apply(@Nonnull C o);
    }
}

PermissionChecker checker =
        new ByResourceOwnershipPermissionChecker<TestObject>("objects", (t) -> t.getOwnerId());

@Component
public class VenueRolesToAuthorities extends DefaultAnemoneRolesToAuthorities {

    @Override
    @Nonnull
    public Collection<GrantedAuthority> toAuthorities(@Nonnull String role) {
        if ("VENUE_ADMINISTRATOR".equalsIgnoreCase(role)) {
            return Arrays.asList(ga("admin_venues"), ga("read_venues"));
        }

        return Arrays.asList(ga(role));
    }

    @Nonnull
    public static final GrantedAuthority ga(@Nonnull String authority) {
        return new SimpleGrantedAuthority(authority);
    }
}

I am going to stop here as this post is already pretty long. We have seen the skeleton of an authorization system that works with roles and authorities (actually, it works on authorities but allows defining permissions in term of roles). Following posts will show a real example with uses of ByResourceOwnershipPermissionChecker.

Nice to have:

Use custom claims in your jwt to specify what authorities/roles apply to what resources.

What I have learned from building microservices

Thu, 11 Jul 2019 10:00:00 GMT

Continuous Integration + Continuous Deployment.

Repeat with me: there is no agile without CI+CD.

Continuous deployments get you to the next level: iterations are shorter, your code is earlier in production, risk is lower. You will bold enough to push code to production on Fridays!

A common framework to build them eases the development

I know that microservices theorists encourage no coupling at all among services, either by data or by code. But a good base library and a good base Dockerfile for those services written in the same language will accelerate the deployment of new services.

Playtomic’s base library contains our typical configuration for Spring applications: security (including configuration and interceptors to call among services), converters, … and some very basic domain classes (such as users in the security context, location, addresses, …). More specific domain classes (such as players, venues, …) are not shared.

Eases the iteration and the update of libraries

Besides, if you release these base components in an ordered way, with a decent versioning, it eases the upgrading of dependencies of your services. For instance, we started with services in Spring Boot 1.x and Java 1.8. We experiment and upgrade libraries in the new services. Older microservices get upgraded when new functionality is added.

By iteratively upgrading versions in new services, differences are fewer, so that upgrading to Spring 2.0 first, then to 2.1, finally to Java 11 was piece of cake.

Sooner or later, you’ll have to denormalize

In a microservices architecture, you don’t have a common database. Efficiency will demand you to denormalize your data across services. For instance, your user profile may end stored in several databases. Maybe, they are slightly different versions depending on the needs of your service.

Learn to live with outdated data

This is a consequence of denormalization and efficiency: you won’t have all data updated in every service. You have to reconsider if you need the last version of that data to operate or you can survive with an outdated one while the fresh version arrives.

This entails eventual consistency, what we will talk about that in some other post.

Makes me design better

As sharing data is more complicated, it forces you to define cleaner concepts and design better APIs. At first, it seems so slow and overwhelming. Once you have a handful of services, pieces start to fit together easily, development speeds up and microservices start to pay off.

2 years at Playtomic.io

Mon, 27 May 2019 10:00:00 GMT

Last February I made 2 years in Playtomic. Everything goes so fast that it feels like 2 decades. It goes so fast that I started to write this post 2 months ago.

There have been very stressful moments, most than peaceful. We have iterated versions and versions of our product and built a huge amount of software in this time. It has been an exhausting learning period.

So that I am writing this post to share some of the most important stuff I learnt here. Rest assured, I have gained tons of technical knowledge but those are not the most relevant

What I truly learnt from two years in a startup: we don’t discuss guilt but solutions. And solutions must be:

Fast to implement (because we don’t have the resources/money to hold up weeks on development).
Solve a problem. And user’s problems are more imperative than developer’s problems (don’t go for that fancy-time-consuming solution if a simpler one will solve it as well).
Good enough to solve those problems (don’t leave trash behind).
Open enough to be iterated in future releases.

I will writing several examples in future posts of this blog. The best sample? Our authentication system.

How to mix Spring Data queries and MongoDB syntax

Fri, 24 May 2019 10:00:00 GMT

I love writing queries using Spring Data’s Criteria. Writing queries directly in SQL (@NativeQuery, @Query) feels a bit dirty. And I also think the official API using MongoCollection and Bson is quirk (at least for a Java developers).

I started using MongoDB a year ago, so I feel pretty lost at first. I started using Spring’s Criteria because it felt more usable and I felt more skilled. But as every abstraction, it is not perfect and does not cover all cases. So, what happens when Spring’s DSL is not powerful enough to describe some part of your query? I had to write all my stages using Bson documents because I didn’t know how to mix Spring’s and MongoDb’s objects.

For example, I was unable to write an addField operation using AggregationOperations. Besides, I had to write a pretty complex expression with operators precedence. I search stackoverflow, indeed, what didn’t give me the solution but gave me some hints O:)

Spring is usually well written and relatively easy to extend/tune to your case. So that… why don’t writing our custom AggregationOperation? Here you have a snippet of my proposal.

/**
 * This example shows how to mix Spring Data Criteria with stages written in MongoDB syntax.
 */
@Component
public class MongoFilterExample {

    @Nonnull
    private MongoTemplate mongoTemplate;

    public MongoFilterExample(@Nonnull MongoTemplate mongoTemplate) {
        this.mongoTemplate = mongoTemplate;
    }

    public AggregationResults<MatchDocument> example() {
        // Using Spring syntax
        Criteria match = new Criteria().where("id").is("document-id");
        Sort sort = Sort.by("creationDate");

        // Using Spring's AggregationOperation built with MongoDB syntax
        FieldsExposingAggregationOperation addFields = addFieldsOperation();
        return aggregate(match, addFields, sort);
    }
    /**
     * It receives standard Spring's Criteria and Sort, and a Spring's AggregationOperation to be use with the Spring's pipeline operation.
     */
    @Nonnull
    public AggregationResults<MatchDocument> aggregate(@Nonnull Criteria match, @Nonnull AggregationOperation addFieldsOperation, @Nonnull Sort sort) {
        MatchOperation matchOperation = Aggregation.match(match);
        SortOperation sortOperation = Aggregation.sort(sort);

        return aggregate(matchOperation, addFieldsOperation, sortOperation);
    }

    /**
     * Aggregation using Spring's MongoTemplate syntax.
     */
    @Nonnull
    public AggregationResults<FancyDocument> aggregate(@Nonnull AggregationOperation... pipeline) {
        TypedAggregation<FancyDocument> aggregation =
                Aggregation.newAggregation(FancyDocument.class, pipeline);

        return mongoTemplate.aggregate(aggregation, FancyDocument.class);
    }

    @Nonnull
    public FieldsExposingAggregationOperation addFieldsOperation() {
        return
                new FieldsExposingAggregationOperation() {

                    @Override
                    public Document toDocument(AggregationOperationContext context) {
                        // Writing MongoDB queries here, so that use the names of the fields as stored in MongoDB
                        Bson newField = sqr(minus("$anotherField", 1));

                        // Returning a MongoDB stage here: {"$addFields: {...}}
                        return new Document("$addFields",  new Document("newField", newField));
                    }

                    @Override
                    public ExposedFields getFields() {
                        Field f = Fields.field("newField");
                        return ExposedFields.synthetic(Fields.from(f));
                    }

                    @Override
                    public boolean inheritsFields() {
                        return true;
                    }
                };
    }

    /**
     * Helper methods that write operations in MongoDB syntax.
     */
    @Nonnull
    private Bson sqr(@Nonnull Bson o) {
        BasicDBList list = new BasicDBList();
        list.add(o);
        list.add(2);
        return new BasicDBObject("$pow", list);
    }

    @Nonnull
    private Bson minus(@Nonnull String field, double value) {
        BasicDBList minusArgs = new BasicDBList();

        minusArgs.add(field);
        minusArgs.add(-value);

        return new BasicDBObject("$add", minusArgs);

    }
}

Playtomic's Pipeline

Mon, 15 Apr 2019 10:00:00 GMT

Your continuous integration (CI) pipeline is one of the most important pieces of software you can have in your business: it runs every time a change in your code is made.

At the backend team of Playtomic, we are quite proud of our CI pipeline. At least, I am :) It not only manages the integration of dozens of branches every day, but the deployments to our development and productions environment. It is currently handling around 35 projects and it is totally unsupervised. Every change to develop branch goes directly to development environment. Every change to master goes directly to production.

Ruben told us how Playtomic’s stack looks like. In this post, I want to share some hints about what we were thinking when building our pipeline. It is not very exhaustive, but I think it is a good approximation.

# How does a pipeline must be? When building a pipeline there are several points you have to care about. It has to be:

Fast: Building, testing and deployment must be fast. How much is fast? 1 minute? 2 minutes? It depends on your requirements, but being below 2 minutes is enough for our teams.
Reliable: several simultaneous builds must not interfiere among each other (i.e.: random ports for testing).
Replicable: builds on the same commit must produce the same artefact.

Pushes to develop and master are always performed after code review (we use github’s pull-requests). More:

Continuos Deployment: a push to develop goes directly to develop environment. A push to master goes to production. No questions.
Reduces the drifting among develop and production: several minor changes are safer to deploy to pro than fewer major changes.
Easy to know which version is deployed where: if the build is green, then it is deployed. build includes commit hash. Important in a microservices architecture.

Software components:

These are the components that form our pipeline:

A git repo: github
A CI system: Jenkins
An artefact store: nexus
An application container: docker swarm

Pipeline steps:

Download code (obviously!)
Validation (master: does not accept snapshots)
Build + unit tests
Integration tests
Docker build
Docker push
Docker deploy
Clean (be nice)

Some snippets. I am skipping the content of the config class and sendNotificationOk(), but I think it is still clear.

Download code (java node):

stage('Download code') {
    deleteDir()
    checkout scm
    env.COMMIT_HASH = lastLocalCommitHash()
    env.POM_VERSION = mavenVersion()
}

Validation (java node):

if (config.isMasterBranch(env.BRANCH_NAME) || (env.CHANGE_ID && config.isMasterBranch(pullRequest.base))) {
    stage('Validating project') {
        sh "mvn validate -Denforcer.skip=false"
    }
}

Build + unit tests (java node):

stage('Build and test code') {
    sh "mvn clean package -DskipITs"

    sendNotificationOk(config, 'Build and unit tests: OK')
}

Integration tests (java node):

stage('Integration tests') {
    sh "mvn verify"
}
sendNotificationOk(config, 'Integration tests: OK')

Docker build (docker node):

stage('Docker Build') {
    // docker-compose.yml uses DOCKER_REPOSITORY and SERVICE_VERSION to tag the image of the service.
    sh "DOCKER_REPOSITORY=${dockerRepository} SERVICE_VERSION=${version} docker-compose -f docker-deploy/docker-compose.yml build"
}

Docker deploy (docker node):

stage("Deployment to AWS - Docker Swarm ${environment}") {
    DOMAIN = config.getPublicDomain()

    // no suffix for production
    if (environment != "pro") {
        DOMAIN = DOMAIN + "-${environment}"
    }

    // default for develop and features environments
    REPLICAS = 1

    // This is a ElasticIP
    DOCKER_HOST = config.getDevelopDockerHost()
    DOCKER_CERT_PATH = config.getDevelopDockerCert()

    profile = environment
    switch(environment) {
        case "develop":
            break
        case "pro":
            profile = "amazon-pro"
            REPLICAS = 2
            DOCKER_HOST = config.getProDockerHost()
            DOCKER_CERT_PATH = config.getProDockerCert()
            break
    }

    DOCKER_REPOSITORY = dockerRepository
    SERVICE_VERSION = version

    // Remember that:
    // 1. We need docker login to access docker repository
    // 2. We need --with-auth-registry to pass these docker credentials to the docker host (in order that it can access to docker repository)
    // 3. If we launch docker deploy in the remote machine, how does it access to the docker repo?
    // 4. Conclusion: we are launching a ssh tunnel, and launching the docker deploy in this machine.

    // See: https://unix.stackexchange.com/questions/83806/how-to-kill-ssh-session-that-was-started-with-the-f-option-run-in-background
    // -L opens a ssh tunnel. -f goes to the background.
    // This command opens the tunnel for 5 secs, but tunnel is not killed while it is used.
    // So that, immediately after that we launch docker deploy which uses
    sh "ssh -i ${DOCKER_CERT_PATH}/cert.pem -f -L localhost:2374:/var/run/docker.sock docker@${DOCKER_HOST} sleep 5; DOCKER_REPOSITORY=${dockerRepository} SERVICE_VERSION=${version} SPRING_PROFILES_ACTIVE=${profile} JAVA_ENABLE_DEBUG=${debug} REPLICAS=${REPLICAS} docker -Hlocalhost:2374 stack deploy ${DOMAIN} --with-registry-auth -c docker-deploy/docker-compose.yml"

    sendNotificationOk(config, "Deployed to ${DOMAIN}")
}

Clean (docker node):

dir('docker-deploy') {
    sh "rm -rf target/"
}

Out of this post:

Different pipelines for libraries and services.
libraries: develop = SNAPSHOT; master = RELEASE.
Auth: How to auth jenkins with github, nexus and docker swarm. Auth between jenkins and swarm is outlined in deploy step.

Tips:

Make your commands explicits, that is, specify all environment variables with the command. This way you can replay them in your machine if something goes wrong by looking for the command in the log.

Next steps:

Some stuff that we want to incorporate or improve:

System tests: set up a test environment in docker swarm.
Quality gates: best implemented as pr-checks than in pipeline. If implemented in pipeline, don’t do in master (build would be broken after merge).
Multistage docker as pipeline (pros: no specific jenkins nodes required. cons: slower: dependencies have to be downloaded every build.

Yet another static blog

Sun, 07 Apr 2019 10:00:00 GMT

Hello there! Yes, another blog written in a static page generator. Just what the Internet needed.

Seriously. Why using a custom template and framework to do something that could be easily done at medium.com or dev.to?

Ownership. Being locked in a platform (such as dev.to, medium.com, wordpress.com, squarespace.com) has many drawbacks. Changes in the terms of use are the most annoying to me: price increases or changes in the features of your tier service (i.e squarespace.com), limiting the number of reads (i.e. medium.com), … Besides, I’m tired of losing content every time I change of platform (or struggling in the migration process).

Having a repo with your code and posts make that contents truly yours.

Yes, it is slower to setup and you will have to invest some time in maintenance occasionally. But it pays completely in terms of control of your content.

So here I am. With a new shiny Gatsby site, a template written in TypeScript, whose code is hosted in GitHub and whose build is constructed and deployed by Netlify.

PD: This is gonna be full of typos. Embrace mistakes!