Mastering Serverless Computing for Data Science/

...

/

Multiple Lambda Functions---Configuring Layers and Deployment

Multiple Lambda Functions---Configuring Layers and Deployment

Learn the syntax and purpose of specific sections in the serverless.yml file for writing multiple Lambda functions using layers.

We'll cover the following...

The custom section

We use the custom section to:

  • Define our custom objects, such as name prefixes, tags, or config file references, which we can later reference in serverless.yml .

  • Configure particular plugins used in the service.

Press + to interact
custom:
  config: ${file(./config.${self:provider.stage}.yml)} # This allows us to define a more complicated configuration if needed (apart from using 'params') 
  # It is a good idea to create curated roles separately which you would then reference when creating these resources
  # We can also create such roles through the 'resources' section, otherwise, Serverless will create a basic role for us
  # roles:
  #   lambda: 
  service_stage: ${self:service}-${self:provider.stage}
  defaultTags:
    App: ${self:app}
    Service: ${self:service}
    Environment: ${self:provider.stage}
  pythonRequirements:
    pythonBin: python3
    dockerizePip: false # We might need to set it to true when compiling non-pure-Python modules or fetching their manylinux wheels is supported on non-linux OSs
    zip: true # Keeping the deployment bundle as small as possible
    slim: true # Keeping the deployment bundle as small as possible
    useDownloadCache: false # Not necessarily needed
    useStaticCache: false

Let’s put things into perspective based on the above example:

  • config: Uses the aforementioned file reference. This means that writing ${self:custom.config.<OBJECT>} will return objects specified in either of the configuration files.

  • service_stage: A handy prefix shortcut that we use when naming functions in code that is built from joining service and provider.stage objects.

  • defaultTags: This is what ${self:custom.defaultTags} was referencing earlier in the provider section. This set of key-value pairs is expanded and injected as a set of tags that are applied to all resources in the service.

  • pythonRequirements: A custom configuration object that we can modify when using the serverless-python-requirements plugin.

Managing resource usage using configuration files

The examples on the right indicate two configuration files for the dev and prod environments, respectively. Although these are optional in this case, we are showing their use case in the following example because they are very important when deploying larger applications.

#
# Configuration for development deployment
#
job_definition_generic:
  vcpus: 0.25
  memory: 1024
job_definition_train:
  vcpus: 0.50
  memory: 2048
Configuration files for dev and prod

We are not applying this configuration now because we do not have a lot of parameters, and the ones we do have can be squeezed nicely in the params section, which we will cover later. However, suppose we have more configurations to include, such as AWS Batch job definitions. In such a case, we might want to include them in separate configuration files to avoid cluttering our serverless.yml file and to improve visibility.

Coming back to the examples, they indicate different values of vcpus of our imaginary AWS Batch job definitions. That’s a pretty typical pattern: limit the usage of resources in the development environment to save costs and use actual required values in the production environment. Referencing the value of vcpus equal to “1” would involve the following syntax in the serverless.yml file: ${self:custom.config.job_definition_train.vcpus}.

Configuration for Serverless Python requirements package

The Serverless Python requirements package is an absolute must-have plugin for ...