I just spent three days debugging a problem within my VR game. I don't quite understand the root cause of the problem, but I do know enough about the nature of the problem to create an effective work around.

Over the course of my career as a programmer and developer, I've gradually gotten better and better at debugging and troubleshooting. I think this is a really hard skill to get down and get good at because its more of a methodology and way of critical thinking to master than a particular debugger in an IDE. Without wasting a bunch of time, I'll get right to it.

The first step is to identify a problem. It's a problem if the behavior of the software application does not behave as it was designed or intended to behave.

The second step is to be able to reliably reproduce the problem. This isn't always easy. In theory, someone could create a random number generator which creates undesirable behavior in a 1 in 100 chance, and that would make the behavior very difficult to reproduce. If you can, it would be super helpful for you to be able to create a game "replay" file, and then save and replay the game and observe the behavior.

Let's say you get this far: You've identified a problem and you can reproduce it reliably. Now what?

This is when we start to practice the "scientific theory". This is where I've seen many, many people screw up and waste time chasing ghosts and trying to fix problems they don't have.

The first step in this process, is to write down your hypothesis. Seriously. Write it down. I'm not kidding. Open up notepad and write down your very best guess at what you think is causing the problem. Remember, this is a guess. It is NOT a diagnosis!!!

In science, when you have a hypothesis on how something works or why something happens, a scientist doesn't go around trying to prove that the hypothesis is correct -- instead, they try to falsify the hypothesis by finding contradictory evidence. The distinction here is ultra important! All you need to do is find one contradictory counter-example to invalidate the hypothesis. If a hypothesis is invalidated, it means that the hypothesis was wrong and we can now move on to formulating a new one. Having a hypothesis invalidated is the opposite of bad, it's very good because now you're one step closer to truth.

I take the same approach to debugging. I write down my hypothesis. Then I try to find ways to prove it is wrong. It is more often the case that I have about 10+ different hypotheses which could explain the cause of the problem. Either zero or one of them is correct. To find the correct hypothesis, I begin running tests and collecting data to invalidate each possible hypothesis by counter-example and contradiction. I want to prove via negation that a hypothesis is impossible. Usually, this involves a lot of isolation of variables and potential causes. During every test, I also write down what I tried and what the results were. This helps me be more rigorous and formalized with my methodology. I also forget things, especially after hours and days of testing, and writing things down helps me not waste precious time repeating experiments.

The process of narrowing down multiple hypotheses to a single hypothesis, is itself a skill which gets developed over time. Valuable knowledge isn't just about getting answers, it is also about asking the right questions. Let me illustrate with a scenario:

Imagine that one day, you meet an angel. It is a divine creature and meeting it is extremely rare. The angel tells you, "I will grant you the true answer to any question you want. What question would you like answered?". Some people might be tempted to ask about the meaning of life, or when they'll die, or next weeks winning lottery numbers, but the very best question to ask is, "What is the best question to ask you?"
For, if you know the right question to ask, finding the answer is relatively easy.

So, when you're trying to negate/break your hypothesis, focus carefully on asking the right questions. Don't be afraid to take five to ten minutes to think about it. It is better to move slowly in the correct direction than to rush quickly in the wrong direction.

Finally, when you have a hypothesis which has withstood the onslaught of testing, you have something which is demonstrably strong. The hypothesis may still be wrong and you just don't know it yet, but it's an operating hypothesis now -- you can reasonably assume it is correct until proven otherwise. I have had operating hypotheses proven wrong on multiple occasions, and that is always a humbling moment for pause and reflection. Once you have an operating hypothesis, you can begin diagnosing and fixing the problem.

I won't get into how to fix problems. That's beyond the scope of this and kind of irrelevant because its so subjective.

Once you believe you have fixed your problem, it is now time to TEST. Did the fix actually work? Is the problem behavior still persisting? If the problem is still there, then either your operating hypothesis is flawed or the fix is flawed.

If you think you fixed the problem, great! You have a new hypothesis to disprove! On multiple occasions, I have mistakenly believed that I have fixed a problem when in truth, my insufficient testing lead me to believe I had fixed it when I really had not fixed it.

Why is this rigorous scientific methodology so important to follow? Because it works, it saves time, and it finds truth.

I once worked with a few novice programmers and sys admin contractors in the US military. They were an embarrassment to the professions. Invariably, as with all software and IT systems, stuff would break or stop working. What did they do? They immediately rushed to the first hypothesis which came to mind, assumed it was true, made it their diagnosis, and began fixing the diagnosed problem. Their fixes would take days of effort and lots of coordination between groups in various parts of the organization. The problem is, they were often wrong. Very wrong, They'd apply a fix and it wouldn't fix anything and the problem would persist. Then they'd invent a new cause, rinse, lather, repeat, until eventually, they got lucky and stumbled into the right answer or everyone gave up and scraped the project. Lots of fingers would get pointed, lots of baseless conspiracy theories are invented, etc. You can imagine the nonsense they put everyone through.

So, back to my debugging experience today:

I used the process I outlined above. I wrote down my initial hypothesis and devised a test to prove it wrong. I successfully proved it wrong. I invented a new hypothesis. Proved that one wrong as well. I wrote down about 18 different hypotheses, before I eventually narrowed down the specifics of my problem.

The problem is that when I enter my game level with an oculus rift VR headset, the screen is entirely black. This only happens on "packaged builds" of my game, which means it is ready to ship to customers. This game breaking bug slipped through my informal QA because I had made some dangerous assumptions. Oddly, this bug is particular to just the Oculus Rift. If I repeated my reproduction steps with an HTC Vive, I would have no issues. Was it an engine bug or a project bug? I recreated a similar scenario with a new project using all of the same settings, and failed to reproduce the error. So, its a project specific error. Is my level corrupted somehow? I made a duplicate of the level and started deleting half of my assets. If it magically worked, then I knew that one of the assets I deleted was the culprit. Sort of. It turns out that if my level has sub levels and those sub levels are set to automatically load as the game loads, oculus can't handle it in the latest version of the engine, only in my project. I have no idea why, but putting a 2 second delay in the game followed by a manual loading fixed the problem.

So, to recap:
1. Formulate a hypothesis.
2. Try to invalidate / negate your hypothesis through data collection. Goto step 1 until step 2 fails.
3. You have an operating hypothesis, now apply a fix.
4. Test your fix. "My fix worked" is your new hypothesis, return to step 1.
5. Go slow and be right.

Okay, if you've read this far, here's the kicker: This process and methodology doesn't just apply to debugging software and IT systems. It applies to everything in life.

Problem: "People don't know my game exists."
Hypothesis: "If I create a bunch of facebook posts / ads, people will know about my game."
Test: Create posts
Data Collection: Look at your web traffic and analytics. Did you see a change? Is your hypothesis invalidated?

When you are armed with this process and methodology, you can find truth. When your actions are based upon truth, you will enjoy successful results based upon those actions grounded in reality.