My experience with GitHub Copilot: can AI speed up the work of a VR developer?

The first time I heard about GitHub Copilot I was like WOW. It was far before the hype for artificial intelligence, and it was already promising an AI-powered system to speed up the work of developers. The teaser video showed a system that was able to predict what you wanted to write and write it down even before you could think about it. I was excited, but the system was not available yet, plus it was said it was compatible in the beginning only with web-related programming languages.

Fast-forward to today, all the world is excited and scared at the same time about artificial intelligence, and GitHub Copilot is offered as a complete solution that is also compatible with C# (hence Unity). I decided so to finally give it a try to see if it could help me and my fellow developers in going faster in our daily job of developing the VRROOM platform for virtual reality concerts. There are a lot of articles out there that say that AI can make us go 10x the speed of our usual work pace… is it true in this case?

Setting up Copilot

First of all, I and the IT of the company had to choose which version of Copilot to use. There is a personal and a business version, and since we are a company the choice to take was of course the second one. I could have decided to try the personal one for a few days at a personal level, before approving the use of the product for the whole company, but that would have been hard, because GitHub Copilot installs as a plugin to Visual Studio (or Rider). This means that whatever project your open, be it personal or professional, it does its job. Theoretically, you can enable and disable it, but practically no one would ever remember to do that for every project that is going to open.

The risk would so have been to use a personal license on the professional product. Apart from this being an infringement of the license (which anyway for just a few times of test drives wouldn’t have been a big drama), it could have been a problem of IP. If you are using the personal version, Copilot can use your code to train its AI model, while if you are paying for the Business license, after your code is sent to the cloud to have the suggestion of what to write next, it is discarded. This means that if you are working in a professional facility, and you don’t want to gift all that you have written to GitHub AI systems, you must go for the Business version from day 1. This is the same thing that holds also for OpenAI: if you are a professional user, you shouldn’t use ChatGPT, because all that you write there is “stolen” by OpenAI, while you should use the OpenAI APIs, which are a premium service and so your data is not taken.

As for the pricing, the personal version costs $10/month, with also the possibility of having a free trial, while the professional one is at $20/month per seat.

Doing the setup of Copilot has been pretty straightforward: I have associated Copilot with our Github organization, then I could select to which contributors of the organization I wanted to assign a Copilot seat. I appreciated the fact that I wasn’t forced to assign it to the whole people that are in the organization, so we could do a test of its capabilities without spending too much. After I assigned it to me and a few colleagues, I received a Welcome email containing a few links to the documentation of Copilot, teaching me how to install it in Visual Studio and start to use it.

I installed it in Visual Studio via the Plugins and Extensions menu, then the system generated a unique passcode I had to enter on the GitHub website to authenticate the plugin with my GitHub account (which had an associated valid Copilot seat). After I did that step, I was done and I could start using Copilot.

I had a few difficulties during the setup of Copilot in Visual Studio, and I tell them to you so you can avoid them:

• When you are looking for the Copilot plugin in the search window of Visual Studio, try to write its name with the right capitalization, because to some people happened that the Visual Studio repository couldn’t find it if written all lowercase
• If you can’t find the plugin in the list even with the right capitalization, update your Visual Studio version. I wasn’t able to find it, but after the update, it appeared in the search results
• When the Copilot plugin generates the passcode for you to insert on the GitHub website, you had better know that the code is generated in the Visual Studio output window (that thing that looks like a console) and it is hard to notice if someone doesn’t tell you.

Anyway, apart from a few hurdles, the setup was quite quick and straightforward. Then it was time to code!

The opensource debate

GitHub has been heavily criticized for Copilot because the AI has been trained with many open-source repositories that are available on GitHub. So basically GitHub has “stolen” their IP to produce a paid service, ignoring completely the spirit of open-source software.

The debate about IP and artificial intelligence is complex and ongoing. What I can say about Copilot is that after this backlash, GitHub is giving you the choice to select if you want the system to use for you suggestions from code that has been scraped from opensource repositories or not. Choosing “no” also helps your company in having less risk of IP infringement and license violations with the code.

How to use Copilot

The whole point of GitHub Copilot is having an AI assistant suggest to you what to write. You don’t invoke it, it just works. There is the upcoming Copilot X that will have integrated ChatGPT functionalities and much more, but the current version is “just” a helper.

The name Copilot is exactly about this. In “pair programming” there are two people working on the same computer: the pilot, the developer actually having the responsibility of writing the code and the copilot, that is there to suggest, advise, and check that everything is being developed correctly. GitHub Copilot is envisioned as your AI Copilot, so instead of a human, you can have an AI and feel forever alone. For once that we developers had a way of having a bit of humanity in our caves, GitHub has decided to remove also that :D.

Anyway, while you are typing, Copilot keeps sending your code to GitHub servers, and when it has a suggestion, it provides it to you writing it in light gray. If you like what it is proposing, you can press TAB to approve it, or you can keep writing to ignore it, waiting for the next suggestion.

How it does perform

I had an initial rollercoaster of emotions about Copilot, but after some time of use, I can say I have quite a solid opinion about it.

If there is a thing that Copilot does very well is understanding the context you are working on: so if you are working with REST APIs, it provides suggestions in line with REST development, if you are working with physics, the suggestion will be about physics, and so on. It almost never fails to understand what “more or less” you want to do. But it fails to understand what you exactly want to write a lot of times.

The problem is that development is not about “more or less”: computers want exact instructions. And it is very hard that an automated system can predict the exact words you wanted to write next.

Sometimes, it happens, and when Copilot suggests like the next 10 lines of code and they are all correct, it feels like black magic. It really makes you spare a lot of time. This usually happens when you are writing something very standard, like doing a copy between textures (e.g. a copy from a Texture2D to a RenderTexture), or conversion between data types (e.g. string to byte array). These are operations that all the developers of the world write a bazillion times, but all of them always forget how to do and so look on StackOverflow for some code to copy-paste. In this case, Copilot becomes your StackOverflow automatic copy-paster and gives you almost perfect code to write.

Anyway, having a lot of lines written without any need for editing is not something that happens frequently. Most of the time the suggestions are correct-ish, and you have to approve them but then edit them to fix them. Sometimes they are even totally wrong, and in this case, you just keep writing and ignore them. I can’t give you exact percentages, but let’s say that maybe the perfect ones happen 15% of the time, the ok ones 65% and the totally wrong ones 20%. Keep in mind that not all the suggestions are 10 lines, most of the time they are just a few words to help you complete the current line of code, so even the right suggestions are not going to write full big chunks of code for you automatically. It’s also strange that sometimes it gives you suggestions as big chunks of code, while other times, it writes them line by line while you type.

Most of the perfect suggestions are about very common programming problems and patterns. The more you are doing “standard stuff”, the more Copilot will give you good suggestions. The more you do very peculiar and standard problem solving, the more the suggestions will be useless. It’s interesting to notice anyway that it can help you with whatever library you are using. Two days ago, I bought a plugin on the Unity Asset Store, not even a common one, but one with a very specific and nerdy purpose for a VR research I’m doing, and Copilot was able to provide me suggestions also on how to initialize the classes of that plugin. That was amazing. It’s also crazy how it knows your code. If you have a method ConvertFromAToB and then you have another method ConvertFromBToC, the moment you have to write a method called ConvertFromAToC, Copilot will suggest you its body as a call to ConvertFromAToB followed by ConvertFromBToC.

Regarding the wrong suggestions, I have two interesting things to say. The first one is that the system learns and copies from you. So let’s say that you are writing two functions: CopyFromString and CopyFromInt, and while you are writing CopyFromString, Copilot gives you a completely wrong suggestion, so you keep writing and finish the function alone by yourself. Then when you start writing CopyFromInt, most probably the system will learn from how you have written the previous method and provide you with a perfect suggestion. This has happened to me more than once.

Another fun thing is that sometimes it “invents” data types. There’s no check that the suggested code compiles and sometimes the system has given me a great suggestion, but some data types were totally wrong. For instance, yesterday I was operating with a plugin about scripts, and there was a type called ScriptType, but the generated code was suggesting to use a class called ScriptObject, which didn’t exist at all. I have no idea where the AI took that name, maybe from some similar code it had been trained with. I so had to accept the suggestion and fix the code.

Since I like to comment a lot my code, I also appreciated that Copilot helps me to auto-complete the comments, which is also a way of speeding up my work. It understands what the code does and usually suggests a comment of the <summary> that is quite on point.

I found it also very interesting to give me suggestions and ideas. For instance, while I was writing a class related to REST APIs, Copilot suggested I make it inherit from IDisposable. It was not absolutely in my plans, but after I thought about it, I realized that this was correct. So it helped me to realize a feature was needed, which was great.

Sometimes I’m really scared about where it takes the suggestion from. Talking again about the REST API code, it suggested commenting a method with something like

//calls the API at a specific endpoint (e.g. https://vrroom.buzz/api)

The fact is that our website is https://vrroom.world, and we don’t use the website https://vrroom.buzz since ages (it was the news website handled by our CEO Louis Cacciuttolo, but he closed it months ago). So I have no clue where the hell the plugin took that web address. My invisible AI copilot is rather scary. It’s like having Terminator as your buddy programmer…

A new way of working

I think that using a software like GitHub Copilot changes the way we developers work. Our usual workflow is to think about what we want the application to do and then we write it in programming instructions, occasionally going to StackOverflow to copy-paste some stuff (well, honestly, a lot of stuff). With Copilot this changes.

First of all, it is not a continuous flow anymore. I’ve noticed that usually now I make a little pause to see what Copilot is going to suggest to me, then I read it and then decide what to do. So from a continuous writing flow, it is more like a write, pause, read, accept/confirm, fix the code it has written, write again, and so on. It is really a bit like working with someone else, for the good and the bad.

Since the suggestions of Copilot are useful and it’s good to have them, I started to also write in a way that facilitates its work: the suggestions are for instance better if you give classes meaningful names (which would be a good coding practice anyway). So if you call a function Texture2D FromRenderTextureToTexture2D(RenderTexture rt), most probably just by the name of the function and the parameters, Copilot is going to suggest you the entire body of the function. If you call the same function PotatoPotaton, of course, it has no idea what to suggest to you. Also, comments help in this sense: if you call your function PotatoPotaton, but the first line of code is a comment like “//converts from RenderTexture to Texture2D”, then Copilot understands what it has to do and gives you the right suggestion. You have so to adapt a bit your coding style, giving hints and suggestions to Copilot to work better, so it can give you better suggestions and you can write less code.

Having good suggestions means also staying more time inside Visual Studio. I forget a lot of time all the conversion routines (e.g. converting from byte array to string), so everytime I have to use one, I usually look for it on Google. Since Copilot suggests all of this to me directly when I’m writing, I’m spending less time doing this context switch, and I’m staying more time in the code editor window. This is time that I spare while doing my work, so it’s good.

As a matter of fact, the job of Copilot is the one of freeing you from the heavy lift of writing obvious code. While I’m using it, I’m noticing that when it works, Copilot de facto writes my code. My job becomes the one of having in mind the architecture of what I want to write, and making sure that the written code respects that. Theoretically, all the rest is handled by Copilot. I “tell it” what I want to do, it writes the code, I adjust its code, and we are done. This is good and bad at the same time.

It is good because the real intellectual work of us developers is to think about solutions to problems. Writing the actual code is dull work, the most satisfying job that we have is to solve problems in elegant ways. If there is an automated tool that can do the dull part, that’s even better, so we can concentrate on the real intellectual work that is our job. On the other side, it’s bad because you really feel like a machine is “stealing your job”. I can envision myself in a few years just being there watching a Copilot writing my code, and I just having the job of supervising what it is doing. This job of “AI sitter” sounds really boring… writing code is fun, it’s still “creative work”, but reviewing the code of someone else is never fun. ChatGPT already writes Unity scripts without any problems, and it showed us that this will be the medium-term future of us developers. The long-term is that our job won’t exist anymore: people will just write a prompt, and the code will write itself. Honestly, I don’t give the job of the developer as it is now today more than 15 years of life. It’s scary.

Don’t sleep on the AI

As I’ve told my colleagues, it’s important that when using Copilot, you don’t just use it and trust it. Like a Tesla car always tells you to be careful of what is happening on the street to be able to intervene with manual override, Copilot is there to suggest but not to write for you, at least not today. As I’ve said, some suggestions are wrong, or not totally correct, and you must always review the code. And review it not only for correctness, but also to make it adhere to the coding standards of your organization.

Plus don’t lose the curiosity of learning more. If Copilot suggests to you some class or function you didn’t know, take your time to look online about it. If it suggests some code, reason if it is optimized enough. Don’t just lean on its suggestions. While not going to StackOverflow is a lot of time spared, it is also a lot of knowledge wasted. When I go to StackOverflow, I love learning by reading the debates of people discussing what is the best strategy to obtain something. I learn a lot by reading what pro developers are arguing about. Having just a pre-cooked answer teaches me nothing.

And at the end of the day, even if the AI is going to substitute us in writing the code, it will take more time to substitute us in making higher-level reasonings (like architecture design) and problem solving. So if we keep ourselves always curious, and we understand and learn the optimization strategies of the pros, we can stay also more afloat of the AI tide. So my suggestion is to still invest your time into looking at how to do stuff in the old-school way by doing proper research on Google.

Final review

At the end of the day, I’m quite satisfied with the use of GitHub Copilot. It doesn’t do miracles, but for sure it is speeding up my work as a developer. Most of the suggestions it provides require some fixing, but also most of them are correct-ish, meaning that it is saving me some time in writing code and writing comments. I see that I’m a bit more relaxed when coding because it is like I have an invisible hand that is pushing me a bit faster than usual. Also having the ability to have some standard coding snippets written automatically is a big plus.

I’ve noticed that GitHub Copilot has slightly changed my way of working, and now I’m adapting my workflow to it, writing the code in a more self-explanatory way so that the AI can give me better suggestions. I also have changed from pure code writer to code writer + code reviewer of what the AI is writing for me. In the long term, I see myself writing always less code, and reviewing AI-written code always more. Copilot has given me a view of what the job of the developer will become over time.

Regarding the advantages I had, I can tell you that for now, we are nowhere close to 10x. Maybe we are around 10-15% faster development for me, which is still an impressive result, but not as big as advertised by some “AI influencers” (previously known as “metaverse influencers” on Linkedin). The fact that I’m working in a sector like VR development, trying to create software that is at the cutting edge doesn’t help for sure, because Copilot can’t find in its training data something that fits perfectly with what I’m doing. Probably if you are doing standard coding, maybe doing websites, or 2D platformers, the quality of the suggestions can be even higher.

Plus remember that this +15% is only on the pure code writing side, but my job requires me to do other tasks, like designing architectures, testing, and coordinating with the other developers… and in this, GitHub Copilot can’t help me at all.

In any case, I’m quite satisfied with my fellow Terminator copilot. Until Skynet will rebel against us, he will help me in doing my job faster. And at $19/month, is totally worth the time that is helping me to save.

(Header image by GitHub)