GitHub Copilot at MercuryWorks

GitHub Copilot is here to become “your AI pair programmer”. With that promised value, we could hardly resist a test spin through GitHub’s new AI offering.

While initially released in late 2021 as a technical preview, the licensed version that made it business-viable (more on that later) was announced in fall 2023. At that point, MercuryWorks engineering leadership decided to pilot Copilot (pun intended), keeping a skeptical view of a new AI tool. Our engineers received the gift of GitHub Copilot (Business license) for Christmas 2023 and from there, we saw the quickest new tool adoption we’ve seen in the firm’s 25-year history.

Any sufficiently advanced technology is indistinguishable from magic.

Arthur C. Clarke

To frame our approach to GitHub Copilot, I should note that MercuryWorks leadership generally sees AI tools as a boost for knowledge workers rather than replacements for knowledge workers. As a result, we think about GitHub Copilot less as a way to generate entirely new and novel software code and more as an aid to our professional engineers, a productivity boost and a quality aid.

We’re not dealing with magic yet but even this skeptical author has to admit – we’re getting close. Keep reading to get a feel for the magic-ish outcomes we’ve seen so far from GitHub Copilot.

Mercury’s Current Copilot Use Cases

As of the time of this writing, MercuryWorks engineers have been using GitHub Copilot for three months. While it is still certainly early days in our Copilot journey, there are several clear use cases and attendant benefits that our engineering staff have observed. We initially assumed that GitHub Copilot would not be an option for our engineers since we traditionally keep our code bases in Git repos within Azure DevOps – we soon found out that GitHub Copilot will work with code bases in any Git repo, regardless of location. Game on!

Most MercuryWorks Copilot chat has been with Copilot Chat embedded within our engineers’ IDEs (either VS Code or Visual Studio proper) to function as the literal coding sidekick that GitHub’s marketing promises. While I hesitate to claim that we have yet reached full feature use of Copilot, the following use cases are significant application areas our engineers are leveraging.

Autocomplete

This is our single most used feature in GitHub Copilot and I would expect that is the case within our industry.  Copilot Autocomplete empowers our developers to be more productive – it streamlines repetitive tasks and suggests best practices.  “Isn’t this just IntelliSense, Don?”, you may be asking. It’s a lot more.

IntelliSense is embedded into Microsoft Integrated Development Environments (IDEs) to help developers with very remedial tasks, like single statement syntax and line completion. Copilot’s Autocomplete will do that as well but it goes much must further – Copilot Autocomplete will provide the logic and code for an entire function.  In other words, it completes entire chunks of code rather than single lines or statements. 

Example Use

Let’s say I am working on some .NET code and want to add a new method called UpdateUser using RestSharp.  I may choose to start out with boilerplate logic from the CreateUser method and at that point, Copilot kicks up and provides Autocomplete for the entire method logic (the grey text in the first snapshot is Autocomplete, not me). You can intuit what my objective was from the name of the method (Copilot sure did).

Hitting tab, and some minor cleanup of my initial logic, produced this working method. The end results took under a minute as compared to 10-15 minutes of “busy work” I would have had to carry out myself:

Day-to-Day Autocomplete Benefits

Net-net we have seen Copilot Autocomplete realistically deliver the following benefits:

• Reduced Development Time: by reliably suggesting relevant code completions in real-time, developers can code faster and focus on logic and value-add functionality; this also results in less time wasted remembering/looking up syntax 
• Fewer Errors: Typos and syntax errors are greatly reduced with Copilot since Autocomplete suggests correct code constructs as you type
• Enhanced Code Discovery: Need a specific function but can’t recall the exact syntax or source libraries?  Autocomplete will seed code snippets or descriptions to engineers to aid in the discovery of relevant functions and libraries.
• Improved Code Consistency: Maintain consistent coding style and patterns throughout your code base.  Copilot learns from the existing code structure and developer coding habits to suggest completions that align with your established style.
• Exploration of Best Practices: Copilot can introduce developers to new coding patterns and best practices they might not be familiar with, fostering continuous learning

All of the above frees up valuable engineer time to spend on innovation rather than “blocking and tackling” coding time.  This has led to subtly faster development cycles and fewer first-time coding errors.

---

Automated Unit Test Creation

Copilot’s unit test generation capabilities help our engineers streamline testing processes and improve code quality (test-covered code imparts higher quality and resiliency).  Utilizing Copilot’s automated test creation capability brings higher test coverage therefore long-term code health and reliability.  We have also seen Copilot help catch potential edge cases while suggesting remaining tests that should be created.

Example Use

Using the same query seen in the Refactoring Section I asked Copilot to generate unit tests:

Copilot generated a basic unit test with mock objects:

As with any AI tool, engineers need to make sure that what is being tested makes sense; engineers need to ensure Copilot-generated tests accurately reflect the intended behavior and use appropriate assertions.

Day-to-Day Test Creation Benefits

• Faster Test Creation: Basic unit tests for your code’s functionality, significantly reducing effort required for testing and improving the odds your application will actually get some reasonable test coverage
• Early Bug Detection: Having unit tests written alongside code allows for immediate identification of potential issues during development
• Simplified Code Maintenance: Comprehensive unit tests generated by Copilot can significantly improve code maintainability. When modifications are made to the code, existing tests rerun (during automated build and release) to verify functionality hasn’t been broken unintentionally.

---

Code Explanation

I have yet to meet an engineer that 1) enjoys authoring technical documentations, 2) is good at it and 3) actually writes voluminous tech docs. As a result, our profession is awash in software applications with minimal documentation and a large uphill effort for new engineers to grok a new code base.  Using Copilot “explain” feature, we can now leave behind detailed explanations of code sequences but let the machines take care of the work.

Example Use

Let’s say I come across this code block that I am unfamiliar with and does not make sense to me:

I can ask Copilot to explain the highlighted code in question (right inside Visual Studio or VSCode) and receive the following synopsis of the code logic:

Day-to-Day Copilot Explain Benefits

There are many benefits to using this feature:

• Improved Code Comprehension: Copilot can analyze and explain the purpose and functionality of existing code; this is especially helpful for onboarding new team members and working with complex codebases
• Enhanced Code Documentation: Copilot can generate explanations that act as inline documentation within the code itself. This improves code maintainability for everyone on the team and reduces the need for separate documentation efforts.
• Improved Debugging: By explaining the logic behind different code sections, Copilot can aid developers in pinpointing the root cause of bugs. This can streamline the debugging process and help resolve issues more efficiently.

---

Code Refactoring

Reduce code complexity and improve code readability and maintainability using Copilot’s suggestions for renaming variables, extracting methods or restructuring code blocks.

Example Use

I asked Copilot to refactor a section of code to reduce its complexity:

Copilot will generate a suggestion:

With a little bit of help, the logic is now in its own method, the prior inline code references it and now I have a generic helper method I can use consistently throughout the applicationhj:

Day-to-Day Copilot Refactoring Benefits

• Effortless Code Improvement: Identify and address code smells (areas of code that can be improved). Copilot can analyze your code and suggest refactoring techniques, such as extracting functions, renaming variables, or improving code structure; this allows engineers to focus on overall application design.
• Enhanced Code Readability and Maintainability: Streamline your code for clarity and long-term maintainability; Copilot’s refactoring suggestions can help make code easier to understand, reducing time spent deciphering complex logic
• Reduced Risk of Regressions: Refactoring can be risky in the form of breaking pre-existing working code (aka “a regression”) but Copilot can help mitigate that risk. Its suggestions typically adhere to solid coding principles to make the introduction of unintended bugs less frequent than often executed by carbon-based coders.
• Exploration of Alternative Code Structures: Copilot has presented alternative code structure techniques that are novel to our engineers. This has periodically helped engineers experiment with elegant approaches in a low-risk environment

---

Code Creation

Copilot can be a powerful tool to streamline and enhance the process of implementing new features. While Copilot provides valuable suggestions, it’s crucial for developers to review and adapt the generated code to fit the specific requirements of the feature. I see this is the widest new horizon for Copilot to spread across.

Example Use

In this example, I used Copilot to add an additional filter parameter to an existing query:

Day-to-Day Copilot Code Creation Benefits

• Increased Development Speed: Describe the desired feature using comments or code snippets within your IDE and and Copilot will analyze the context and suggest potential code implementations
• Improved Focus on Functionality: By suggesting relevant code blocks and functionalities, Copilot allows developers to focus their energy on the core logic and problem-solving aspects of feature implementation
• Reduced Risk of Errors: Human error is inevitable, but Copilot can help mitigate it. By suggesting code that adheres to best practices and syntax rules, Copilot can minimize errors during the initial development stages.
• Enhanced Collaboration: Improve communication within your development team. Utilize Copilot’s suggestions as a starting point for discussions and code reviews to foster a shared understanding of the feature’s implementation.

---

Help Fixing Bugs

Copilot is a valuable tool in your bug-fighting arsenal.  The standard IDE IntelliSense has the ability to report on hard errors, like build failures and warnings around common issues like null reference checks.  However, all it does is the issues – IntelliSense does not address them for you.  Copilot will.

It’s important to remember that Copilot’s suggestions are not magic bullets – they require review and human expertise. However, by leveraging its capabilities, you can streamline the debugging process, identify new solutions, and ultimately deliver more robust and reliable code.

Example

The following logic can throw a null and divide by zero exception. If we run the “fix” command, Copilot will give us help:

Day-to-Day Bug Fix Benefits

• Identify Potential Issues: Stuck staring at a bug but unsure where to start? Copilot can point you in the right direction and help you narrow down the root cause of the bug more efficiently.
• Explore Alternative Solutions: Sometimes, fixing a bug requires a new perspective. Copilot can offer alternative code snippets/functionalities to spark new ideas and help you find a more elegant solution.
• Test-Driven Bug Fixes: Copilot can help write unit tests (see above) alongside its suggested bug fixes to verify that the fix actually resolves the issue, doesn’t introduce unintended side effects and lays down insurance against future breaks
• Boost Debugging Confidence: Even experienced developers can benefit from a second opinion. Copilot’s suggestions can provide validation for your debugging approach or offer different viewpoints, ultimately increasing your confidence in the bug fix.

---

Generate Code Documentation

I get to use my quote twice in one blog post: “I have yet to meet an engineer that 1) enjoys authoring technical documentations, 2) is good at it and 3) actually writes voluminous tech docs.” Copilot’s ability to generate code documentation comes in handy here.

Example Use

Day-to-Day Copilot Documentation Benefits

• Improved Code Clarity: Copilot can analyze your code and automatically generate clear and concise explanations of its functionality (see above). This improves code understanding for everyone working on the project, especially new team members or those unfamiliar with specific sections.
• Reduced Documentation Effort: Free up valuable development time. Copilot automates a significant portion of this process, allowing developers to focus on core development activities.
• Enhanced Code Maintainability: Well-documented code is easier to maintain and modify. Copilot-generated documentation serves as a living reference, ensuring everyone on the team understands the code’s purpose and functionality.
• Consistent Documentation Style: Maintain a uniform documentation style throughout your code base. Copilot can learn from your documentation practices and suggest explanations that align with your established format.

---

Engineering Management Topics

Results We’re Seeing So Far

MercuryWorks engineering leaders conservatively report a 5-10% productivity improvement between accelerated development and increased code coverage. We expect that productivity to increase as Copilot becomes second nature to our engineers and its use becomes a true habit. Every engineer we bought a license for (which is all of our engineers) had used it within the first month so we have already cleared the first hurdle in adopting a new coding lifestyle.

Just by integrating generative AI into the editor, GitHub Copilot has quickly defined a new age of software development, resulting in clear gains of developer productivity and happiness.

Thomas Dohmke, CEO of GitHub

What Others Are Reporting

• A six-month study of 450 Accenture developers shows that 90 percent of developers reported writing better code with Copilot
• GitHub research shows that developers accept, on average, 30% of suggestions received from Copilot
• GitHub research further shows that over 80% of developers report conservation of mental effort by delegating repetitive tasks to Copilot

How Reliable Is GitHub Copilot?

Pretty reliable, actually. So far, we have yet to observe a “false positives” along the lines of “change your code like this” and then result in a broken application. Engineers do sometimes get responses like “I don’t know what you mean” or “I don’t have anything to suggest” but those have been admittedly very rare – maybe 1-2% of the time.

---

Serious Business Topics

Legal Indemnity

The seminal event in Mercury’s adoption (and we expect all corporate adoption) was the creation of the Business license that includes legal indemnity. In September 2023, GitHub parent Microsoft said it would indemnify customers accused of copyright infringement as a result of using Copilot and commemorated this into a formal commitment with the issuance of the new Business license. If you retain nothing else from this post, take this point: keep your organization out of legal trouble by adopting the Business license (Enterprise will do it, too).

Business vs Enterprise Licenses

Copilot Enterprise provides several features not available to the Business tier that MercuryWorks is using (currently that is). The cost differential between Business and Enterprise is currently $20/month/developer and while we are not currently utilizing Enterprise, we expect that once we have maximized Business use cases we will jump to Enterprise. We expect to extract another unit increase in productivity benefits when we adopt Enterprise.

The Enterprise version of Copilot imparts repository-based semantic search, making it easier for engineers to scour an entire organization’s code repos. Additionally, Enterprise will analyze pull request (PR) diffs (code differences before and after work on a piece of functionality) to generate PR summaries. This will make it easier for engineers receiving the PR to understand the proposed changes and likely save IMMENSE time (and/or the odds of PR comments along the lines of “I changed some code”). An important point to call out here is that these capabilities require your code repos to be in GitHub.

Finally, Enterprise can access internal company knowledge bases which makes its suggestions more on-point to how your organization works. There has been some talk that in the future Copilot Enterprise may enable organizations to support model fine-tuning; we think this will enable our engineers to train a private OpenAI helper with the ins and outs of OUR own codebases. If this materializes, GitHub Copilot would provide domain expertise about our apps and not generic coding best practices.

---

Wrapping Up

I hope that this post has done a good job of covering how MercuryWorks is using GitHub Copilot and gives you some ideas on how your organization could benefit from it. We’ve been very impressed with it thus far and have even higher expectations of GitHub Copilot (and other Microsoft Copilots) going forward.

We plan on authoring another post that will cover what we think will be coming to GitHub Copilot next and provide an update on our further use of Copilot (we’ve extracted big benefits so far but after all, it’s only been 2 months).