Fun with Feature Flags
Feature flags are a simple concept. A feature flag is a conditional block of code that allows you to toggle specific functionality on or off. This allows for separation of code deployment and feature deployment.
A simple example:
if ( flagService.IsFlagActive(FeatureaFlags.SuperCoolNewFeatureFlag) )
Release toggles will often have a broad scope. They will wrap an entire feature set, possibly disabling both UI components and business logic. Doing so allows you to deploy code with the flag off, so it’s not accessible to the user. When development of the feature is complete and code deployed, the flag can be flipped on to enable the functionality.
Experimental toggles are more about gathering useful data from users. By toggling visual functionality components for specific users, you are able to log interactions and determine what your users like and use most. This is referred to as A/B testing.
Ops toggles are used to toggle operational aspects of a system. For example, when switching to a new payment system, a feature flag can be used to determine if you use the new or old payment system. This allows you to quickly switch back to the old payment system without having to re-deploy if an issue arises.
Permission toggles allow you to toggle specific functionality for specific groups of users. For instance, you could allow privileged/premium users into beta functionality to give it a test run before going live to a larger user base. Conversely you can ensure that a given set of users resistant to change, are the last to get new functionality.
The many advantages to using feature flags
Infrastructure or lack thereof
Projects seeking to go for a serverless architecture, such as mobile app development, can utilize companies that offer software as a service for feature flags. You are able to enjoy the benefit of having no infrastructure to maintain, while being able to toggle functionality and systems for your entire user base, without users having to update the app.
For projects that do utilize infrastructure there are still tremendous advantages. With the use of feature flags practices like continuous deployment become much more attainable. Developers can code a feature with the flag turned off, continuously deploy their code to production, and turn the flag on when the feature is complete. This separates code deployments from feature releases.
This allows the team to form an always forward, never backward mentality. Barring catastrophic failure, rollbacks can become a thing of the past. Rolling back takes time to do and can carry its own negative impacts; especially when database changes are involved. Rather than going through the hassle of rolling back a deploy if an issue is discovered, a flag can be turned off to disable certain functionality and the change takes effect immediately. This means developers can get right to fixing the problem, and not waste time trying to stop it. This lowers both the time a user sees an issue, and the turn around time for a resolution.
While they don’t go away entirely, even merge conflicts get better with feature flags. Since code can be deployed while off, Pull Requests can be made for smaller chunks of code more easily. This means code reviews are less complex, faster and easier to do, and code is merged to master more frequently.
As mentioned in Experimental Flags, A/B testing is a very powerful tool for determining what your users want and use. You get real insight into user actions that you can act on. For instance, if you create a new feature and a group of users never use it, you may want to re-evaluate if you roll it out to your entire user base. You can make tweaks and see if a different user group increases usage.
Feature Flags can even aid in debugging a production environment. If a developer needs to investigate a high priority production issue, time is of the essence. But pointing a local dev build straight at production is dangerous and getting a local database copy of production can take lots of time. A great strategy is to include sections of targeted verbose logging behind an Ops Flag. This allows you to only turn on the verbose logging when you want it, so that your production logs don’t get cluttered up. With more verbose logging enabled, the developer may get hints that allow them to quickly resolve the issue.
Feature flags don’t make everything better
Feature flags are often an immediate source technical debt. If you have old code living beside new code, you have technical debt. Allowing the old code to sit there as a backup means you don’t fully trust what you have built, which isn’t ideal. Further, it can be very easy for a developer to miss an execution path and not update legacy code. This means if you have a problem with your new code, and attempt to turn the flag off, the legacy code may experience issues as well.
As you create more and more flags your code will experience cascading code complexity. Creating nested feature flags means you are creating additional execution paths. Depending on the number of logical layers in your application and the number of nested flags this can quickly become very complex and difficult to debug or test. When testing a feature, developers and QA will need to know every flag involved and test every flag in every on-off combination to truly pass every test case. This can quickly become unreasonable even for the most likely combinations.
There is an overhead cost to implementing feature flags. Just in response time, each flag you check is an additional database or web call. On high frequency operations, this could cause a delay. There are additional code paths that may need to be implemented as well. For example, imagine a system that stored an order shipment price in tbl_shipment. Let’s say we have a new feature to store the shipment price per item in tbl_shipment_item. Switching straight to the new feature means we can change all out our queries to use the new column. But if we want to have a feature flag to switch between the two, even when the flag is set to use tbl_shipment_item, we need to still update tbl_shipment. This is to make sure that if the flag is ever switched off, the data is still correct, and not stale. These additional steps may not always be obvious and are easily forgotten.
When Considering Feature Flags
When planning the project work for the new functionality discuss the need for feature flags. If you think a feature flag is warranted make sure to create a task specifically for the creation of the feature flag which ensures the developer has capacity to guarantee all routes and logic are covered and not overlooked. In addition, the feature flag should be included in the acceptance criteria for the story.
Most feature flags should be short lived. Once you are sure a new feature is solid you should begin removing the flag and old logic. It may be useful to create a future PBI to remove the flag right when your first make the flag. Set the PBI to be 3-4 sprints in the future to ensure proper testing time in a live environment. Consider checking your flags for the last time they were toggled and removing those that haven’t changed in some time.
Use unique and descriptive names for your feature flags. Never re-use old names. You want to be sure you know what is being toggled and where when you change flags, and similar names can cause confusion. Additionally, in code, it’s wise to use an enum or some concrete class for identifying your feature flags; never use just plain strings. When you go to remove a flag, having compile time context about flags is much better than searching through code for a specific string
Given the complexity feature flags can introduce, always make sure to test in a staging or dev environment before toggling a flag. Logic may have changed or been missed since the flag was created.
Look around for related flags that could affect the logic outcome. Nested flags can create many unexpected logic routes that need to be tested. Just because one environment is set one way, doesn’t mean a different environment is.
While feature flags are handy, they are not a solution for every problem. Do not attempt to put everything behind a flag. This will drastically increase the complexity of your codebase and can quickly become unmanageable.
While it may be tempting, most feature flags shouldn’t be kept around forever. Make a plan for removing them. Each flag adds alternate execution paths that your developers will need to account for. This can quickly increase the scope of new functionality and easily ruin initial estimates for work.
Feature flags are a power tool with many uses but can lead to a messy codebase. Using them should be carefully thought about in each scenario and only used when appropriate. Feature flags should be considered more as a development aide than a go-to pattern. Feature flags allow a developer to achieve continuous deployment and clients see great value in the speed at which issues can be resolved.
- Bird, J. (2014, August 8). Feature Toggles are one of the worst kinds of Technical Debt. Retrieved from https://www.javacodegeeks.com/2014/08/feature-toggles-are-one-of-the-worst-kinds-of-technical-debt.html
- Connolly, J. (2019, April 11). Introducing Feature Management in Full Stack: Launch new features with confidence. Retrieved from https://blog.optimizely.com/2017/10/18/feature-management/
- Contributor. (2017, May 22). Feature Branching vs. Feature Flags: What’s the Right Tool for the Job? Retrieved from https://devops.com/feature-branching-vs-feature-flags-whats-right-tool-job/
- Engdahl, S. (2008). Blogs. Retrieved from https://aws.amazon.com/blogs/startups/continuous-delivery-and-effective-feature-flagging-with-launchdarkly/
- Erik. (2018, January 9). Your Feature Flag Management Needs to Include Retirement. Retrieved from https://rollout.io/blog/feature-flag-retirement/
- Feature Flags. (n.d.). Retrieved from https://www.optimizely.com/optimization-glossary/feature-flags/
- Meléndez. (2018, June 7). The Top 5 Use Cases for Feature Flags. Retrieved from https://rollout.io/blog/top-5-use-cases-feature-flags/
- Paul-Hammant. (n.d.). Trunk Based Development. Retrieved from https://trunkbaseddevelopment.com/feature-flags/
- Rusovsky, E. (2017, February 20). Every Change is a Feature. Every Feature is an Experiment. Retrieved from https://rollout.io/blog/continuous-feature-deployment-for-mobile-apps/
- Samuele Resca. (2017, February 11). Continuos Delivery using feature toggle. Retrieved from https://samueleresca.net/2017/01/continuos-delivery-using-feature-toggle/