Go for DevOps/

...

/

Using Policies to Restrict Tools

Using Policies to Restrict Tools

Understand how to enforce SOW with policies to be applied to the workflows.

We'll cover the following...

Rate limiting is great for preventing a bad tool run from wiping out a service when all items of work are equal. But not all items of work are equal, as some machine services are more important and fragile than others (such as our service’s database systems). Also, machines or services may need to be put into logical groupings that can only happen in a limited amount. These could be broken up by sites, geographical areas, and so on.

This logic is generally specific to some set of work items. This bundling, which we will call a SOWScope of Work (SOW) is a document providing details including list of features and outcomes of a work or software., can be quite complex.

To safely do work, we must understand our scope. This might be how we can safely update database schemas for a particular service or how many route reflectors in a network region can be modified at a time.

To implement safety around a SOW, we will introduce the idea of policies. Policies will be used to check a set of work that is entering into the system for compliance. If it is not compliant, it will be rejected.

As an example, we will look at handling disk erasures similar to Google's disk-erase case study. Here are some protections we will add:

  • Only allow a single satellite disk erasure to happen every hour.

  • Rate limit so that we can only erase five machines at a time.

  • Must pause for 1 minute after each five-machine erasure.

To be able to make a policy engine, we must have a common way to define what kind of work will be executed, in what order, and with what concurrency.

We also want the tool engineers to only define the work to be done and submit it to a separate service that executes it. This allows for the centralization of control.

Let's define the service that could do that in gRPC.

Defining a gRPC workflow service

Let's look at the arguments to make calls to see what our clients will send the workflow service, as follows:

Press + to interact
message WorkReq {
     string name = 1;
     string desc = 2;
     repeated Block blocks = 3;
}
message WorkResp {
     string id = 1;
}
message Block {
     string desc = 1;
     int32 rate_limit = 2;
     repeated Job jobs = 3;
}
message Job {
     string name = 1;
     map<string, string> args = 2;
}

These messages are used to define the work that a client wants the server to execute and contain the following attributes:

    ...