In This Blog
- Prerequisites
- Whatʼs Copilot
- How About Other Chatbots?
- Copilot Test One: Python
- Copilot Test Two: Kubernetes
- Where Chatbots Fall Short
- Final Thoughts
- Frequently Asked Questions
New technologies often come with mixed reactions. Some people are early adopters, others are more skeptical, and many wait until innovation becomes the new normal. Generative AI is following that same pattern, but its influence is spreading quickly. It’s already reshaping how we work, communicate, and solve problems.
For software developers, the impact is especially significant.
In this blog post, we’ll explore how to get the most out of GitHub Copilot and how it can support your development workflows.
Prerequisites
- To follow along with this blog post, you should have the following:
VS Code installed, which you can find here. - The GitHub Copilot extension installed via VS Code, which you can find here.
Whatʼs Copilot
Copilot is Microsoft’s generative AI assistant, powered by large language models (LLMs). Unlike traditional tools with narrow functions, Copilot is designed to work across a wide range of scenarios. It’s integrated throughout the Microsoft ecosystem, from Azure Kubernetes Service (AKS), where it can assist with troubleshooting cluster deployments, to productivity tools like Microsoft 365, developer environments like Visual Studio Code, and data platforms such as Microsoft Fabric.
Copilot is available in both free and paid tiers, with the premium version priced at $10 USD per month. As with most LLM-based tools, the value often comes down to how specific and well-structured your inputs are. The more precise your queries, the better the output.
How About Other Chatbots?
Before we get into testing scenarios, it’s important to take a step back and consider the range of tools available. There are many chatbots that can help with coding, but not all are created for the same purpose.
The most important thing to understand is what each chatbot was trained to do. For example, Kubiya is built specifically as a DevOps assistant. It has been trained on data related to DevOps, cloud infrastructure, and platform engineering. You wouldn’t use Kubiya to look up a recipe for gluten-free pizza, but it’s a strong choice if you need guidance on deploying Terraform code in Microsoft Azure.
While many of these tools share similar training data, the models behind them can vary. Each chatbot may be optimized for different types of tasks, which impacts the quality and relevance of the responses.
Here are some of the top chatbots for programming and cloud-related work:
ChatGPT is great as well, but not as accurate as the ones mentioned above.
Copilot Test One: Python
Now that we’ve covered the basics of coding chatbots and introduced GitHub Copilot, it’s time to see how it performs in real scenarios.
We’ll start with a test case focused on Python.
First, create a new file called [main.py] and when you open it up, youʼll see an icon that sort of looks like two stars. If you click it, youʼll see that Copilots already looking to help out.
If you type in a simple [main] function, Copilot will prompt for some logic to go within your function.
It can, however, make some mistakes. For example, notice the indentation on the statement below. That should be outside of the function scope.
You may also notice that Copilot doesn’t always suggest exactly what you’re expecting. That’s not a flaw. It simply can’t anticipate your intent without the right guidance. This is where effective prompt engineering becomes essential.
To test our Github Copilots prompt engineering capabilities, letʼs look at something a bit more advanced.
Below is an example using CrewAI, which is a popular Python framework to create AI Agents. Notice that thereʼs an Ask Copilot search bar below. To use that search bar, type COMMAND + I on Mac or ALT + I on Windows.
You can then search for what youʼre looking for and when you search for a particular outcome, Copilot will write the code for you.
Not a bad output! This one was pretty spot on, but the thing to keep in mind here is the below was based on what was already in the script. For example, the [tools] list that you see on line 33 is the same exact list in the previous Agent that was already created.
Keep in mind that Copilot won’t get everything right. You’ll still need to review the code and confirm it works as expected. A strong architecture and thoughtful design are key to getting the most out of any chatbot. Writing the code is often the easy part.
Copilot Test Two: Kubernetes
In the previous section, you saw how Copilot works with a general-purpose programming language. Letʼs now see how it works with a configuration language.
Create a new file and name it deployment.yaml.
Within that file, open Copilot and ask it to create a Kubernetes Manifest like the one below.
In this scenario, the prompts were intentionally left vague to evaluate how much guidance Copilot requires to produce useful output.
Not bad! The results were fairly strong. However, it is important to remember that Copilot generates responses based on training data sourced from the public internet. As a result, it may not always reflect best practices. For example, in Kubernetes, best practice often discourages setting CPU limits and recommends setting CPU requests instead. Copilot is unlikely to catch that distinction unless it has been fine-tuned for the environment or paired with a retrieval-augmented generation (RAG) approach.
One thing to notice is with the Copilot prompt, you can choose from different Models as you can see in the screenshot below.
For our final test, letʼs ask GitHub Copilot something intentionally outside its intended use case.
Now, of course, we are not going to get what we want from this prompt because it falls outside of what GitHub Copilot is designed to do. Copilot is not an autonomous agent that can execute actions on your behalf beyond what you are already working on in the terminal, especially when using it in VS Code.
That said, there are now Copilot Agents that can be trained to perform more specific tasks. It is still unclear whether those agents can handle actions like deploying to a cluster, but the functionality is evolving quickly.
This is a good reminder that while AI can be powerful, it is important to understand the specific type of generative AI you are working with, what it is built for, and the kind of outcomes you can realistically expect.
Congrats! You have now completed your first experience using GitHub Copilot in VS Code, testing it with two different tasks to evaluate how well it supports each use case.
Where Chatbots Fall Short
The quality of the code generated by tools like Copilot depends entirely on the data available within the model. These models are trained on massive volumes of data from across the internet, and the training process itself can cost millions of dollars. However, that does not mean the data is always accurate or complete.
For example, while working on a recent project involving a policy enforcement tool built with Policy-as-Code principles, we noticed a pattern. Because this particular tool does not have widespread examples available online, many of the AI-generated suggestions were incorrect. This was consistent across multiple platforms, including ChatGPT.
On the other hand, tools and languages with well-established histories—like Python—are more likely to yield accurate responses due to the abundance of reliable documentation. Still, the sheer volume of content online also increases the risk of encountering inaccurate or outdated guidance.
The key is to understand both the strengths and limitations of generative AI tools. They are powerful, but only as reliable as the data they are built on.
Final Thoughts
As generative AI continues to shape how developers write, test, and maintain code, tools like GitHub Copilot offer both exciting opportunities and important tradeoffs. They can enhance productivity, streamline repetitive tasks, and support learning new frameworks. At the same time, they require careful oversight, strong architectural thinking, and a clear understanding of their limitations. Copilot is not a replacement for technical expertise. It is a powerful companion that works best when paired with experience, curiosity, and sound engineering judgment.
If you’re exploring how to integrate AI tools into your development workflows, get in touch with our team to start the conversation. We’re here to help you build smarter, more resilient solutions.
Frequently Asked Questions
What is GitHub Copilot, and how does it work?
GitHub Copilot is an AI-powered code assistant developed by GitHub and Microsoft. It uses large language models (LLMs) trained on public code and text to suggest lines or blocks of code as you type. It integrates directly into environments like Visual Studio Code and helps speed up repetitive tasks, explore new libraries, and troubleshoot issues.
Does Copilot write perfect code?
Not always. Copilot generates suggestions based on training data, so it may miss context or follow outdated patterns. It is a helpful assistant, but all output should be reviewed, tested, and aligned with your project’s architecture and coding standards.
How does prompt engineering affect Copilot’s output?
Prompt engineering plays a major role in the quality of Copilot’s responses. The clearer and more specific your prompt, the better the results. Providing relevant context within the file or in comments can significantly improve the accuracy of its suggestions.
Can I use Copilot for infrastructure tools like Kubernetes or Terraform?
Yes, but with caution. Copilot can generate YAML manifests or Terraform scripts, but it may not always follow platform best practices. Always validate infrastructure code and cross-reference with official documentation or community standards.
What is the difference between Copilot and other chatbots like Kubiya or Cursor?
Each tool is built for a different purpose. Copilot is embedded directly into your editor and provides code suggestions in real time. Tools like Kubiya focus on DevOps workflows, while Cursor adds collaborative AI features with broader code navigation and versioning support.
Can Copilot perform actions like deploying to a cloud environment?
No, Copilot is not an autonomous agent. It can assist with writing code or scripts, but it cannot execute commands or deploy environments on your behalf. That functionality would require a separate automation or orchestration tool.
How can Emergent Software help with AI integration in development workflows?
Emergent Software helps teams thoughtfully adopt AI tools like GitHub Copilot, ensuring they are implemented in a secure, scalable, and effective way. We provide hands-on guidance, custom development support, and training to help your team get real value from AI-enhanced workflows. Reach out to us to learn more.
