Europace started more than 20 years ago and is Germany’s biggest platform for real estate financing, housing-saving and credits with currently more than 35k of such transactions per month. With the success and therefore constantly adding new products and features using new technologies, Europace built up some technical debt as well as a zoo of technologies over the years.
The Europace technology zoo already included different technologies for operating services on production. And because all these technologies needed to be maintained, we were very conservative when it came to adding more. Hence our approach on building an internal developer platform is to focus on reducing technical debt as well as our technology zoo. Therefore we develop our platform as a product and solve actual problems of developers by running experiments. We are basically shaping our platform and shared services while cleaning up our closets.
Everything started with our decision to move out of our data centers and migrate to AWS.
Running Europace on AWS
Our way to an internal development platform was anything but a straight line. In the beginning, all we knew was that we wanted to migrate our services to AWS. At this point we didn’t even know whether we wanted to have something like an internal developer platform or just use plain AWS.
So we started by doing small experiments with Jenkins X, AWS EKS, AWS ECS, Docker Swarm and plain Docker. Our learnings led to bigger experiments. Teams started to use AWS EKS, AWS ECS and Docker on EC2 as well as AWS Lambda to run their production workloads on AWS. We learned that plain AWS with EC2 and ECS works well for completely autonomous teams, but for running all Europace services on AWS we needed to consolidate our efforts and support our product teams by removing the complexity of handling AWS on their own.
We saw that the knowledge and expertise necessary to manage AWS accounts and operate AWS resources in a secure, compliant and stable way should not be underestimated. This is something we cannot expect to be built up in every product team. We want our product teams to really understand the domain of their customers in order to solve their customers’ problems.
Furthermore, we realized that having different technology stacks for operating services made it really hard to grow as an organization, i.e. changing purposes and accountabilities of teams and therefore ownership of services.
As a result we founded a platform team which has the purpose of enabling developers in our product teams to focus on their actual challenges by providing services and tools that are easy to use for them.
As a first step our platform team brought in experts from an external consultancy to plan and implement a multi-account AWS setup with an optional Kubernetes setup based on AWS EKS that is ready-to use. Afterwards our product teams started to migrate their workloads to this setup with the support of the platform team.
Cleaning up
Now that we were able to replace our infrastructure in our data centers with AWS, we realized that there is more to do. Our experimentation mindset has led us to a zoo of technologies for observability, data persistence and so on over the years. This zoo has to be maintained and eventually consolidated. This is where we—the platform team—come into play.
For every technology that we “clean up” we decide on whether we discontinue it, migrate it or make a platform or shared service out of it. In order to be able to make such a decision we have the following guiding principles:
- Managed service over self-hosted
- Self service & automation over individual solutions
- Long term success over short-term band aid
- Stop starting, start finishing
- Eat your own dogfood
This way our internal developer platform is nothing that we planned and are implementing step by step, but rather a growing product, which we develop problem by problem.
Platform as a product
We develop our internal developer platform (PaaS) like any other product at Europace. We start with an actual customer problem. Customers of our internal developer platform are developers in product teams who are looking for support on their developer journey from writing code over delivering it to production until maintaining it.
For each customer problem, we check whether the platform already provides a feature that can be used for solving this issue. If not and a solution could be useful for the other product teams as well, we might use this opportunity to enhance the developer platform. If this is a challenge particular to this product team, we try to support the team with internal or external expertise.
Solving customer problems is what we call “product work” while running our platform is “maintenance work”. Therefore we are tracking the ratio between those two kinds of work and try to keep roughly around 60% for product work and 40% for maintenance, so that we have a focus on solving customer problems while providing a stable platform they can rely on.
Not only do we treat our internal platform as a product, we also organize ourselves and work as a product team.
To Scrum or not to Scrum
We work in iterations, but don’t follow Scrum. This way we can deliver product improvements in iterations and deal with unplanned maintenance work on short notice.
For our product work, i.e. solving customer problems, we run small experiments in order to
- understand the problem,
- find a solution that is “good enough for now” and “save enough to try”,
- implement and verify this solution.
In order to do so, we decide on focus topics, i.e. problems, we want to tackle in an iteration. We don’t plan the actual tasks we are going to do, since those will come up during our experimentation and learning process.
But how do we decide on those focus topics? We have regular grooming meetings in which we discuss customer problems and things we learned during our experiments. We also look at our OKRs and challenge possible solutions against our product and technical visions as well as against our strategy and principles. So we do have regular planning, but we don’t prioritize tasks. We rather ensure that we are aligned and on the same page.
Another thing that doesn’t work well with Scrum is that our maintenance tasks are usually unplanned things that come up on short notice. A decision on what to do about it usually needs to be done immediately or in the next daily standup.
We do have regular reviews and retrospectives, though. So maybe from a bird’s eye view we might actually do Scrum.
Conclusion
Building an internal developer platform was not our goal, but rather a result from making Europace more resilient and scalable for future challenges. Our platform team works like any other product team, but for internal customers. We run experiments in order to solve actual customer problems and to reduce the technical debt.
We still have a lot of things to “clean up”, but every cleanup is an opportunity to improve our internal platform and to make the life of developers in our product teams easier and therefore fulfill our purpose as a platform team.