Platform and Ecosystem

Monitoring team

Suppose we decide to put monitoring into the platform. There’ll surely be a monitoring team within the platform team. Would we expect that team to respond to application alerts? Definitely not! Instead, that team should provide the capability that others then use. In other words, the monitoring team doesn’t do the monitoring, it provides the ability for others to do their own monitoring. This is a mental shift from ownership of the domain to offering a service to customers.

Seems like an easy enough heuristic, but it leads immediately to a change in the way we view responsibilities. For example, it used to be common for the monitoring team to implement all the specific monitors, triggers, alerts, and thresholds. That puts them right in the middle of the change loop. It means they have to create a “request for monitoring” form for development teams to fill out (whether paper or online). It means that tweaks and changes to monitoring have to go through a queue in the form of the other teams’ inboxes.

Building the infrastructure for alerts

If we respect the customer-centric model, then the monitoring team should not implement the actual monitors. Team members should work one level removed: they implement the tools that let their customers implement their own monitors. In other words, the monitoring team may need to build infrastructure to receive alerts, deployment tools that push their monitoring agents out (if applicable), or scripting tools that let developers provide a JSON description of the monitors they need.

This begins to look like creating interfaces in an object-oriented application. The monitoring team offers up an interface that development teams can use. The details of implementation are owned by the monitoring team and can change as long as they continue to support their contract.

Database administrator

What about database administrators? It’s a shame that the acronym DBA can mean both “database administrator” and “database architect.” The lines of responsibility have gotten blurred over the years. The administrator should ideally be concerned with creating a high-performance, stable platform on which development teams can build any kind of database. Sadly, technology constraints in days past led us to have DBAs that were responsible for both the health of the database server and the data model used by the applications. This caused a lot of tension when the data model was contorted to make the server happy instead of vice versa. A lot of the energy behind the NoSQL movement was really about refactoring those responsibilities.

NoSQL

With NoSQL and post relational databases, we see a different split in the roles. The platform team includes database administrators who keep the database running and healthy. They ensure there’s enough capacity but the data model is up to the application.

SQL-based RDBMSs

The picture is harder with SQL-based RDBMSs. It’s too easy for one application to make a harmful schema change that affects other consumers. This leads us to decree a separate physical database for each service. It’s not very resource-efficient, but it does unfreeze development teams to move independently, without a queue for DBA attention.

Is it possible to create a platform that allows safe, autonomous delivery into a shared SQL database? Yes, but it requires accommodation from both developers and DBAs. In particular, the difficulty of parsing SQL to do automated sanity checking is too high. Developers and DBAs have to agree on a simpler, machine-readable format that can be scripted against. Many migration frameworks offer XML, JSON, or YAML formats that suffice.

Keep in mind that the goal for the platform team is to enable their customers. The team should be trying to take themselves out of the loop on every day-to-day process and focus on building safety and performance into the platform itself. If you find that your technology choices or architecture make this really difficult, it’s a good argument to change the technology!