GitOps your Elastic Beanstalk environment properties

In today's blog, I am going to cover how to setup environment variables for Elastic Beanstalk, with focus on a GitOps approach. AWS prefers to call these as environment properties.

Enterprise applications depend on multiple configuration values that change across different environments. In the very pragmatic approach of  Build Once Deploy Many an application bundle is created once and deployed to multiple environments. Such configuration is strictly stored outside of application code, as per Twelve Factor Principles.

For an Elastic Beanstalk application, there are different ways to provide this configuration :

1. The most crude way to setup these configurations is to login to EB console and manually add them. This is well explained in AWS Documentation and I am not going to cover that here. Given that this is a manual approach, it is error prone and not scalable.
2. A far better approach is to commit such configuration in version control using configuration files. This requires using the super powerful ebextensions and is explained in AWS Documentation with an example of .ebextensions/options.config. If you're new to setting these, I'd suggest staying away from detailed documentation, which can be pretty confusing. Few things to note here 
1. The config file must be a valid YAML. I learned it the hard way, and now use my favorite linter, http://www.yamllint.com/ to validate. Given the structure of this config file, its easy to make mistake in formatting, and the documentation does not mention that this is a YAML.
2. The weird-looking option_settings is a way to define environment variables and is explained in a not so intuitive way in AWS Documentation
3. Namespace is important when defining anything in this config file. For environment variables, the namespace is aws:elasticbeanstalk:application:environment. 
4. This config file can reference AWS CloudFormation! This is a big deal. This implies that you can reference any other resources created earlier in your CFN stack. For example, when your database got created, it might have added credentials to secrets manager, and that can be referenced through this config file. It is possible to reference Secrets Manager through code as well, and frameworks such as Spring make it a breeze, but what such frameworks cannot do is reference your CFN stack.
5. The config file can be a full fledged CFN file. For example look at this sample provided by AWS. Apparently CFN references can either be pseudo references, or the ones created by this config file itself. You could always reference CFN params from other stacks, but you'd probably need to know the stack ID and devoid of pre-defined naming convention, it could be a pain to solve. We are not going to do that today.
6. Finally note the back-tick when referencing CFN. You miss it and it would start failing!

But, what if you wanted to access some custom parameters that are not an output of CloudFormation? How would you access such custom parameters? 

Lets take a step back. Where would you even keep your custom parameters?

AWS provides a valuable resource, AWS SSM Parameter Store, to store such custom parameters. Once defined, they may be consistently referenced by diverse applications in your account, such as Lambda and Elastic Beanstalk. We version control these and deploy them to our account.

So, how do we reference parameters stored in Parameter Store, in our application? Turns out, Elastic Beanstalk does not have a well documented way to define an environment variable referencing to SSM Parameters. You could try to set it using ebextension hooks, but that is a rabbit hole which burned over two days of mine, and still did not work. A couple of folks have exported variables using these hooks and used them in EB application. That did not work for me. 

What does work, however, is our good old friend, Cloudformation. CFN allows referencing SSM values. With that in mind, our config file can be easily modified to reference SSM parameters, such as in the snippet below.

Let me explain this tiny snippet a little more. At line 3, we are defining the custom parameter, with a name of "CUSTOM_SSM_PARAM" that refers to a SSM parameter of same name -- and a unique version number. Note that, as of writing this blog, CFN does not support using LATEST version (duh!) of parameter, but relies on a specific version to be specified. If you fail to specify the version number, an error will be thrown. Also note that while String type of SSM parameters can be referenced, Secure String parameters cannot be accessed. This limitation is documented here.

A final note, if you look for this environment variable after setting it up through ebextension, you'd see that the value on UI console still shows up '{{resolve:ssm:CUSTOM_SSM_PARAM:VERSION}}' and not the value for it. However, in the application perspective, this would be resolved just fine, implying this is not a one time static binding, but rather a truly dynamic reference to SSM.