So I am working on an application at work that must be highly performant. The requirement is to handle 500 million messages in a 6 hour period using as little hardware as possible. I was able to get the application working at 12,000 messages per sec (260M in 6 hours) which is no where near my requirement. I started publishing rates from the application and I noticed times when the application was processing 0 messages in a second. I also noticed that the CPU would never top 50%-60%. The application is written in .Net and I am using the Task Parallel library which should use all cores available automatically. There was no need for me to determine how many threads to use because TPL does this automatically. Profiling the application, I determined that garbage collection was the main culprit. I used the profiler within Visual Studio Ultimate and I was able to see exactly what was happening when the CPU dropped. It indicated that Memory Management was the problem and I figured that pointed to garbage collection. I started researching on the internet and came across an interesting article Writing High Performance .Net Code. This article led me to two configuration items that helped with garbage collection.

• gcConcurrent enabled=”false” This is turned on by default and turning it off means that garbage collection stops the application threads for a shorter duration when absolutely necessary.
• gcServer enabled=“true” This turns on server garbage collection which is optimized for server based applications. It creates one garbage collection heap per processor and one garbage collection thread per garbage collection heap. Basically, it makes garbage collection more efficient for high throughput applications.

Adding these two items under runtime in the application config file significantly improved my application rate. It improved it by 100%. With just these changes, I am now processing at 21,000 messages per second and I am much closer to my requirement. The CPU now tops 80%-90% most times. My rate still drops to 0 but much less often. There isn’t much I can do about this because garbage collection has to occur. I am going to now go through the application and ensure that I am not creating objects unnecessarily and do more profiling to increase the throughput even more but this find was a life saver!