Migration of an on-premise kubernetes deployment to Google Cloud Run.

Tools used: GCP, Google Cloud Run, Terraform, Kubernetes, Vault, Python

At Adaptive, I helped to benchmark the Aeron messaging platform on Amazon Web Services (AWS) and Google Cloud Platform (GCP).

During the course of the project, I:

• built the infrastructure provisioning harness required to run the tests on two cloud providers with different scenarios and configurations
  • instance types
  • network topology and configuration
  • transport security (IPSec, Wireguard, etc.)
  • storage types
• built an automation for rendering results (latency graphs, experiment parameters, etc.)
• built an automation for provisioning infrastructure, running benchmarks in various configurations (peer-to-peer, cluster) and rendering results
• assisted with the running of benchmarks, analyzing results and investigating performance issues

See the press release from AWS on the topic and the performance testing results are publicly available.

See also the article on achieving sub-20 microsecond performance on Google Cloud.

Tools used: AWS, GCP, Linux, Terraform, Bash, Performance Engineering, Python, Java

• self-study of the Rust programming language (books, online courses)
• self-study of machine learning (books, online courses)
• building prototypes to understand the concepts and tools
• reading papers on distributed systems that I intended to read for a long time

After watching the drone footage of San Francisco in burning orange skies due to wildfires, with the Blade Runner soundtrack playing on top of it, I decided to look for an engagement with a company helping to do something to address climate change. This is how I learned about Leap Energy which provides solutions for connecting distributed energy resources to wholesale energy markets in order to make the electric grid more resilient.

In the role of Technical Lead I focused on projects in the following two areas:

• infrastructure modernization
  • migration of the self-hosted customized VPN server to a cloud offering (vendor selection, user migration)
  • migration of the self-hosted Kafka cluster to Confluent Cloud (vendor selection, data migration, integration with the monitoring system, service migration and coordinated roll-out)
  • migration of the self-hosted elasticsearch cluster to elastic cloud (vendor selection, data migration, service migration, coordinated roll-out)
• microservice architecture modernization
  • introduction of ADRs in order to document architecture decisions
  • introduction of GRPC as a means for service-to-service communication (replacing direct database access across services and an experimental service-to-service communication via Kafka topics) between Spring Boot services
  • development of a service protocol registry and distribution mechanism (based on protocol buffers) to allow contract-first design and development of the various services
  • introduction of Gradle with Kotlin support as a build tool as a replacement to maven and porting multiple services
  • introduction of jOOQ as a replacement to Hibernate for database access

Tools used: Kotlin, Java, React, Typescript, Go, Python, Spring Boot, Microservices, Bash, Gradle, Maven, GRPC, Kafka, SQL, elasticsearch, logstash, filebeat, Terraform, AWS

Ongoing support of the engineering team with the implementation of Scala- and Akka-based microservices:

• architecture and design reviews
• code reviews
• technical questions
• performance engineering

Topics & tools: Reactive design patterns, Microservices, Distributed systems, Akka, Scala, Kubernetes, Helm

• review of the technical design of a single-node Akka application against the business objectives
• support of the project team with regard to questions and issues arising during the project with regard to the usage of Akka and Scala

Topics & tools: Actor Systems, Reactive design patterns, Akka, Scala

Architecture evaluation of an event-sourced system based on Apache Kafka and Kubernetes

Topics & tools: Event Sourcing, CQRS, Kafka, Kubernetes

Troubleshooting performance issues with an Order Management System (based on Akka Cluster, Kafka and Kubernetes) and setting up a large-scale load test for the system.

Topics & tools: Performance Engineering, Akka Cluster, Kafka, Kubernetes

Houston Ross

CTO - Paidy Inc.

Paidy’s business is rapidly growing, and with each new customer, we face exciting new challenges and targets to strive towards.

The recent on-boarding of a significant new customer with a newly agreed-upon Service Level Agreement meant we had to review the latency requirements of our core payment service.

Manuel, who is incredibly experienced with Akka Cluster, helped us better understand the background behind the elevated latencies in our core services and improve the issues. We are now on the way to having predictable latency at the scale we will need to support our growth, including during update and failover scenarios. Manuel’s work was invaluable in helping us achieve this milestone for Paidy.

The onboarding of a new strategic client on the Paidy platform brought with it stricter latency requirements (in order to meet the agreed-upon Service Level Agreement). The backend application built on top of Akka Cluster with cluster sharding did not deliver consistent latency in redeployment and failover scenarios. I helped the engineering team find and understand the root cause of the latency fluctuations and guided them throughout the process of implementing a solution.

Topics & tools: Performance Engineering, Akka Cluster, Akka Cluster Sharding, AWS ECS

Matías Pacelli

VP Engineering - Delivery Hero AG

As the world’s largest food delivery network, logistics are a vital part of our business and it’s important that we’re always operational and avoid outages at all costs. A critical service of our logistics infrastructure uses Akka Cluster, so we reached out to Manuel to get input from a reactive systems expert.

Manuel conducted a thorough reactive audit, reviewing the architecture of one of our core applications, and then discussed his findings with us in a one-day workshop. He helped us improve our application and optimise our infrastructure so we can ensure maximum uptime. I’m happy to say that our logistics are now running even more smoothly for our brands all around the world.

A core application based on Akka Cluster had ongoing memory issues, requiring to run virtual machines with a lot of memory (64+ GB) as a workaround. I held a performance engineering workshop to show the team how to use the adequate tooling for performance profiling and memory pressure analysis. A key result of the workshop was a tenfold decrease of the JVM memory required to run the project.

Topics & tools: Performance Engineering, Async Profiler, JVM memory management

Development of a template project for deploying Akka Cluster projects with Kubernetes

Topics & tools: Kubernetes, Akka Cluster

Ralf Sigmund

CTO - MOIA GmbH

We started MOIA because we wanted to create a ride-sharing service that makes transport more environmentally friendly and more affordable. In order to meet our Hamburg launch target, we needed some external help to speed up the development process, and this is where Manuel came in.

Manuel initially consulted with us on evaluating and setting up split-brain resolution for Akka Cluster in the context of our Kubernetes deployment.

Then, to offer a competitive app, our payment service needed to be rearchitected so as to support more payment methods. With time until our Hamburg launch running out, we asked Manuel to support our payment team with the overhaul.

He joined the payment team where he helped design the new version of the payment service and directly contributed to its development.

It was great to see how well Manuel integrated with our team despite not technically being a part of the company. He truly cared about our service and what we were creating and was essential to us hitting our target launch date.

I joined the Payment Services team at MOIA to help with developing the second version of the payment system based on Scala and Akka. The system needed to enable the payment of MOIA trips and needed to integrate with multiple Payment Service Providers while providing strong consistency as well as being highly available.

Tools used: Scala, Akka, Akka Cluster, Akka HTTP, Microservices, SBT, DynamoDB, AWS Kinesis, PostgreSQL, AWS

Evaluation of the deployment of a split-brain resolution solution for Akka Cluster deployed with Kubernetes on Amazon Web Services (AWS). At this time, there was no official support for Kubernetes deployments of Akka and therefore I proceeded to analyze the deployment model with regard to different edge-cases that could occur in a split-brain scenario.

Topics & tools: Distributed Systems, Akka Cluster, AWS, Kubernetes, Helm

Adrien Treccani

CEO - Metaco SA

Security is paramount when it comes to cryptocurrencies and banking. So when these two interact, security becomes an even bigger deal. Our solution SILO, which launched in 2018, finally makes it possible for banks to safely store their clients’ crypto-assets.

We got Manuel on board right away when we started building SILO because of his outstanding expertise in Akka Cluster. He helped us kickstart the project and accompanied us during the first few months of development. By putting best practices in place, Manuel made sure SILO was off to a good start. This has allowed us to build a successful product that is secure, reliable and integrates seamlessly with the infrastructure of financial institutions.

I supported the engineering team during the initial development of the SILO platform (based on Akka Cluster and Scala) by:

• reviewing the system architecture and technical feature designs
• providing guidance and best practices with regard to the use of Akka Cluster, Scala and Kubernetes
• developing the initial project scaffold and core application layer

Topics & tools: Reactive design principles, Actor systems, Akka, Akka Cluster, Scala, Kubernetes, Helm

Architecture review and code review of an Akka Cluster based application

Topics & tools: Reactive design principles, Akka Cluster

• reading 50+ papers on the topic of distributed systems
• basis for the talk Is there anybody out there?
• basis for my personal research work in the distributed systems field

• architecture of the new microservice-based software platform for an existing real-time communication platform
• design and code reviews
• hands-on development of the core feature set of the platform (based on Akka, Scala and Scala.js)
• Cloud DevOps infrastructure on Azure

Topics & tools: Scala, Scala.js, Typescript, Akka, Microservices, Kafka, SBT, Terraform, Azure

Jessica Millar

CTO - VCharge Inc.

VCharge has a complex and naturally reactive technology which is working in production but needs to rapidly scale by two or three orders of magnitude. We hired Manuel to help with this. He is perceptive and practical, detail-oriented and abstract, hard-working and laid-back, smart and humble. He is fun to work with and learn from, and a great teammate and all-around strategist / thinker in addition to having deep reactive / Akka-based experience.

Overall, Manuel is fantastic, and he comes with my strong recommendation if you have an ambitious, distributed, high-speed technology that needs to flourish and scale.

This engagement was a direct result of my work on deploying Akka on the Raspberry PI (see also the initial article).

VCharge developed an embedded system and software solution to help address the issue of maintaining the stability of the electric grid in the presence of increasing renewable generation.

During this engagement I supported VCharge in:

• building a new version of the low-latency backend service (based on Akka) responsible for aggregating telemetry as well as actuating the IoT devices at the core of the VCharge Grid network.
• further development of the firmware driving the embedded IoT devices

Tools used: C, Buildroot, Bash, Scala, Python, Akka, elasticsearch, logstash, Kibana, Puppet, AWS

Gabriela Schatz

VP PSP Services - Wirecard Acceptance Technologies GmbH

Wirecard has been growing substantially over the past couple of years. Consequently, we needed to rethink our architectural paradigm to support this sustained growth going forward.

We found that reactive microservices were the perfect solution, allowing us to scale up in terms of transaction volume. The microservices architecture also lets us fully leverage our international developer teams as it allows them to work independently without requiring constant communication, which is crucial for a company that operates on a global scale.

Manuel has been accompanying us on this journey since 2016, training our teams on Akka and Akka Cluster. He also performs regular architectural reviews, which helps us stay on track as the project evolves. Furthermore, Manuel has been giving us great advice when it comes to advanced technical topics, such as performance tuning.

It’s incredibly helpful to have a distributed systems expert at hand to guide our teams so we can ensure our core systems are capable of supporting our business growth.

Ongoing support of the engineering teams working on the next generation of the PSP (Payment Services Provider) core platform:

• architecture reviews
• design and code reviews
• performance engineering workshops

Topics & tools: Reactive design principles, Actor systems, Event Sourcing, CQRS, Performance Engineering, Akka, Akka Cluster

Robert Lux

VP IT & Development - Dimoco Europe GmbH

We had to replace a key component in our system and brought Manuel in to introduce us to Akka. We had already been looking closely at a different solution, but Manuel’s prototype quickly convinced us that Akka would be the best choice for us.

Manuel used three connected Raspberry Pi computers to demonstrate how lightweight and fast the Akka toolkit is. He then got our team up-to-speed on the technology and supported us while we developed and successfully implemented our new system with Akka. In this critical process, his expertise with distributed systems was invaluable to us.

DIMOCO initially approached me in order to develop a prototype for an architecture evaluation with Akka Cluster to help make a decision of which technology to use (the other contending technology was VertX). I developed a software prototype deployed on multiple Raspberry PIs to demonstrate the resilience and failover capability of Akka Cluster (unplugging a network cable while transactions were being executed and showing that the processing continues and no transaction is failing makes for a good demonstration).

Akka Cluster was subsequently selected as the technology to use for developing the new version of DIMOCO’s SMS delivery platform. I supported the engineering team in the early stages of the development of the platform by:

• reviewing the architecture and the technical design
• regularly reviewing the code
• providing guidance and best practices with regard to the use of Akka and Akka Cluster

Topics & tools: Distributed systems, Reactive design principles, Akka Cluster, Kubernetes

Daniel Lang

CTO - Talenthouse Inc.

When we wanted to switch our technical platform to a reactive architecture, Manuel supported and coached us through the entire process. He helped us rewrite our platform from scratch and migrate the massive Talenthouse databases.

We only had three months to pull it off and wouldn’t have been able to do it without Manuel. With enthusiasm and creativity, he set us up with a brand new and stable platform that can now grow with our company.

I joined the engineering team at TalentHouse in order to build the new version of the company site using the Play Framework and Scala.

Tools used: Scala, Akka, Play Framework, Javascript, elasticsearch, RabbitMQ, PostgreSQL, Heroku

Michael Briem

CEO - Nexxchange AG

I consulted Manuel during my recent venture into software as a service, and he has been a great resource. His input has helped me obtain the technical solutions for the Nexxchange platform that I’d been envisioning.

I value his know-how in both technical and business matters and how diligently he translated technical content for me.

Nexxchange provides an all-in-one platform for managing Golf resorts and hotels.

I designed and implemented the online portal that aggregates the tee times of the multiple golf clubs, making them searchable and bookable.

See also the Reactive Golf article series part 1, part 2 and part 3.

Tools used: Scala, Akka, Play Framework, Groovy, Javascript, elasticsearch, MongoDB

Delving provided a SaaS offering for cultural heritage organizations (museums, libraries, archives) allowing them to bring their digital content online and make it searchable.

Tools used: Scala, Akka, Play Framework, Javascript, XML, SOLR, elasticsearch, MongoDB

Oxiras provided a SaaS solution for manual testing, akin to HP Quality Centre

Tools used: Scala, Play Framework, Javascript, Chef, SQL

As part of the Digital Preservation team at the Austrian Institute of Technology I took part in building software tools aiming at making digital assets accessible in the long term, such as:

• building workflow engines with automated plans for data format migration
• building tools to annotate digital media
• etc.

Tools used: Java, Scala, Adobe Flex, CSS, Javascript, SQL

As part of the eBusiness & CRM team at T-Mobile Austria I worked on the maintenance and development of the Order Management System for mobile phone tariffs as well as on a project to build the next generation order management system (Oracle Siebel, IBM Master Data Management)

Tools used: Java, SQL, Linux, Oracle Siebel

As part of the team at the HCI lab at KTH I worked on software solutions for collaborative programming (see this paper)

Tools used: Java, Tomcat, Hibernate, ANTLR, SQL, Linux

Contributor to the Makumba open-source web framework for Rapid Application Development

Tools used: Java, Tomcat, ANTLR, Spring, SQL, Hibernate, Maven, Ant, Linux