The Frustration Fix : Script to avoid system hangs

    As a compiler engineer, I frequently find myself immersed in the task of building compilers. This process demands a significant amount of RAM, especially when dealing with debug builds. To expedite these builds, I rely on tools like Ninja to execute tasks in parallel. However, I've encountered a recurring issue: when Ninja attempts to utilize all available processors, its memory consumption skyrockets, often maxing out the system's RAM and leading to a frustrating system hang.

    Sure, I could limit Ninja's core usage with the "-j8 or -j1" flag, but this invariably extends the build time, which isn't an optimal solution. Instead, I observed that approximately 90% of the build tasks run smoothly in parallel, consuming less than 50 GB of RAM. However, it's the last 10% that proves problematic, as it demands more memory and frequently triggers system hangs.

    In the past, I've resorted to manual monitoring of RAM usage during builds, preemptively canceling jobs, and adjusting core usage to prevent system crashes. Yet, despite my vigilance, there are occasions where I miss the threshold, resulting in frustrating system hangs.

    Compounding this issue is the fact that I work on remote machines without direct access to power controls. When a system hang occurs, rebooting becomes a cumbersome process. It entails raising a ticket with the IT team and waiting for them to perform the necessary reboot, consuming valuable time and disrupting my workflow.

    To address these challenges, I've developed a scripting solution that automates the build process, eliminating the need for manual intervention without compromising on build times. The script allows me to set a RAM threshold and dynamically adjust the number of cores utilized based on RAM consumption.

Here's how it works: 

    The idea is simple, the script continuously monitors RAM usage during the build process. As the RAM consumption approaches the predefined threshold, the script dynamically adjusts the number of cores allocated to Ninja, ensuring that the system remains within safe memory limits.

My Script:

Log:    

    By implementing this script, I've regained control over my compiler builds, mitigating the risk of system hangs and streamlining the development process. No longer do I need to worry about manual monitoring or lengthy reboot procedures—my builds now proceed seamlessly, allowing me to focus on what truly matters.

Note: I want to emphasize that while the scripting solution I've developed works well for my specific needs, it may not be the only or ideal fix for everyone.

Have you encountered similar challenges in your work with compiler builds or system hangs? Perhaps you've developed your own innovative solutions or have valuable insights to contribute. I invite you to share your experiences and solutions in the comments below!