Threading Nightmares and 75 FPS: What I Learned Building a VR Pipeline GUI

Working on my VR audio-visual scene reproduction project, I thought building a PyQt6 GUI would be the easy part. I mean, how hard could it be to wrap some machine learning modules in a nice interface, right?

Turns out, pretty hard when you're dealing with VR performance requirements and threading issues that make your GUI freeze faster than a computer in a freezer.

The Great GUI Freeze Mystery

Picture this: you click a button in your beautifully designed GUI, and then... nothing. The interface freezes, your progress bars stop updating, and you're left wondering if your computer has given up on life.

That was my reality for way too long.

The culprit? Thread safety issues in the shifter implementation. I was trying to run image processing operations on the main GUI thread like some kind of threading amateur. Every time the pipeline started processing a 360° image, the entire interface would lock up.

Here's what I was doing wrong:

# DON'T DO THIS - blocks the main thread
def process_image_badly(self):
    result = heavy_image_processing()  # This takes forever
    self.update_progress_bar(result)   # GUI is frozen by now

And here's how I fixed it:

# DO THIS - proper threading
def process_image_properly(self):
    self.worker_thread = ProcessingWorker(self.image_data)
    self.worker_thread.progress_updated.connect(self.update_progress_bar)
    self.worker_thread.processing_completed.connect(self.on_processing_done)
    self.worker_thread.start()

The Progress Bar Lies

Even after fixing the threading, I ran into another annoying issue. Progress bars are supposed to give users feedback about what's happening, but mine were basically lying.

The problem was with the infer360 process - it had some inconsistencies that made progress tracking really tricky. Sometimes it would zip through 80% of the work in seconds, then spend forever on the last 20%. Other times it would get stuck at random percentages.

I ended up having to implement a more sophisticated progress tracking system that actually monitored the underlying processes rather than just estimating based on time:

def track_real_progress(self):
    # Monitor actual file outputs instead of just time estimates
    expected_outputs = self.get_expected_output_files()
    
    while self.processing:
        completed = len([f for f in expected_outputs if os.path.exists(f)])
        progress = (completed / len(expected_outputs)) * 100
        self.progress_updated.emit(progress)
        time.sleep(0.5)

Much more accurate, and users actually knew what was happening.

VR Performance: Why 75 FPS Matters

Here's something I learned the hard way: VR isn't just about pretty graphics. It's about maintaining frame rates that don't make people sick.

I initially tried implementing WWise for spatial audio because it looked really sophisticated and had all these cool features for real-time acoustic modeling. The demos looked amazing!

But then I actually tried running it...

Voice starvation errors everywhere. Even in desktop mode, the system was struggling. I tried optimizing everything I could think of:

Reduced sample rate from 48000Hz to 44100Hz
Simplified parameter configurations
Reduced the number of active audio sources

Nothing worked. The performance was just too resource-intensive for VR requirements.

See, in VR, you need to maintain above 75 FPS to prevent motion sickness. It's not a suggestion - it's a hard requirement from Meta's VR performance guidelines. When your audio system is eating up resources and causing frame drops, people literally get nauseous.

I ended up switching to real-time Steam Audio, which was way more VR-friendly. The performance difference was immediately obvious:

Consistent frame rates above 75 FPS
No more voice starvation errors
Actually better spatial audio quality than the over-engineered WWise setup
Simpler integration with Unity

Sometimes the "fancier" solution isn't the better solution.

GUI Architecture Evolution

The GUI went through several major iterations as I learned more about what actually worked:

Version 1: Basic PyQt6 interface with everything running on main thread

Problem: GUI freezing during any processing
Lesson: Threading is mandatory, not optional

Version 2: Added worker threads but poor progress feedback

Problem: Users had no idea what was happening
Lesson: Progress bars need to reflect reality, not estimates

Version 3: Proper threading with real progress tracking

Problem: Still cluttered interface with too many options
Lesson: Feature creep makes UX worse

Final Version: Clean, modular interface with hidden debug tabs

Solution: Main interface for normal use, debug tabs hidden but accessible
Result: Both debug and production UI share the same code modules

The Debug Tab Strategy

One thing I'm actually proud of: instead of maintaining separate debug and production interfaces, I implemented a system where the debug tabs are always there but hidden in the main interface.

class MainInterface(QtWidgets.QMainWindow):
    def __init__(self, debug_mode=False):
        super().__init__()
        self.debug_mode = debug_mode
        
        # Always create debug tabs
        self.debug_tabs = self.create_debug_tabs()
        
        if not debug_mode:
            # Hide debug tabs but keep functionality
            for tab in self.debug_tabs:
                tab.setVisible(False)

This way, both interfaces use the same underlying code, which means:

No duplicate code maintenance
Debug features are always available when needed
Main interface stays clean for normal users
Developers can easily enable debug mode

Real User Feedback

I did user testing with 6 participants, and their feedback was... educational.

One person said: "The sound would sometimes be very low compared to my expectations when I was behind some objects."

Another suggested: "It would also be nice if there could be two speakers for example in the music hall."

This kind of feedback is gold because it tells you what actually matters to users vs. what you think matters as a developer. The spatial audio occlusion was working correctly from a technical perspective, but users found it confusing. Sometimes "realistic" isn't the same as "useful."

Docker: The Build Error Olympics

Oh, and Docker. Let's talk about Docker integration with the BoostingMonocularDepth module.

This thing gave me more build errors than I care to remember. The module has very specific dependency requirements, and getting it to play nice in a containerized environment was like solving a puzzle where half the pieces are missing.

I eventually got it working, but it required:

Specific CUDA versions
Particular Python package versions
Custom build configurations
Way more patience than I initially budgeted

The lesson here: always budget extra time for Docker integration with research code. Academic ML modules are not known for their deployment-friendliness.

What I'd Do Differently

Looking back, here's what I learned:

Start with threading from day one - Don't try to add it later
Test audio performance early - VR frame rate requirements eliminate many solutions
Real progress tracking beats estimated progress - Users notice when progress bars lie
User testing reveals assumptions - What seems obvious to you isn't obvious to users
Simple solutions often perform better - Don't over-engineer

Current System Status

The final VR application is now working well with:

Real-time Steam Audio for spatial audio
Proper threaded PyQt6 GUI with real progress tracking
Cross-platform VR support (Meta Quest, Windows Mixed Reality)
Published on Itch.io with full documentation
Consistent 90+ FPS performance

The GUI debugging tool saved me countless hours of troubleshooting, and the modular architecture means I can easily add new features without breaking existing functionality.

The Takeaway

Building VR applications teaches you that performance isn't just about making things fast - it's about making things consistently fast. A GUI that freezes for 2 seconds might be annoying in a desktop app, but in VR it can make someone physically sick.

Threading, progress feedback, and performance monitoring aren't just nice features - they're essential for any real-time interactive system. And sometimes the most sophisticated solution (WWise) isn't the right solution for your specific constraints.

Building VR applications or dealing with similar GUI/threading challenges? I'd love to hear about your experiences - drop me a line through the contact page!