A printing company with offices around Southern California, approached us with a difficult problem. They had created an in-house print on demand engine whose purpose was to provide customers with a Microsoft Publisher replacement that ran entirely in a user's browser without plugins! This engine, while powerful, was starting to age - racking up a little over 8 years of technical debt. Furthermore, the increased popularity of the print on demand market for their customers was causing a high amount of stress on the engine and they needed help managing the resulting difficulties.
We started with researching what was currently functioning in the overloaded editor. During research, however, we learned of many difficulties the engine faced while attempting to function on the internet. To provide some context, the internet is a heavily optimized system designed with the express purpose of keeping content such as pages, images, and videos as small as possible in order to meet speed requirements. If users wait too long for content to load, they will simply leave and get that content elsewhere.
The big issue, however, was less of the internet and more of vendors - this included Microsoft. In order to create common ground for both the printers and the users, the Adobe PDF format is used for both previews to users and printing by staff. Because of the extremely high traffic, the print on demand engine was starting to generate memory leaks that had not previously been seen but were not exposed in both the PDF manipulation library being used, as well as in GDI+ part of Windows.
Rather than waiting for vendor responses the memory leak issue was addressed by altering the server design for the platform and isolating the offending code with a combination of the Visual Studio Profiler and Application Insights. Modifying the server design allowed us to safeguard the rest of the system which had been causing it to crash bringing down the engine as a whole and affecting profits.
We also addressed the traffic issues that were the result of the new demand for the system by developing a highly customized load balancer designed to address both the memory leak issue and distribute load. This, in turn, provided horizontal scaling. This allowed the company to dynamically scale resources where they were needed the most depending on current customer demand. When memory leaks are detected by the system it alerts the load balancer through Application Insights. Servers are then taken out of the rotation, cleaned, and re-added to the rotation without affecting customer traffic.
These changes have helped the engine stabilize even under high stress and ever-increasing traffic. This has allowed the companies customers to create more pieces at a quicker pace thus increasing profit.
