What Qentelli Did

During the initial investigation and discovery phases, there was a lot of uncertainty about the key bottleneck. After carefully studying the server logs, we suspected memory leak on the code. As It was a legacy application, the logging and alerting system was not intact. 
We teamed up with the in-house application team to understand all the critical business scenarios and started creating a common configuration for the business flows. Because every club that’s using the application had a different configuration and it was difficult to build the scripts to test each.

Strong and dynamic scripts were created:

As we learned that the application is highly configurable, we decided to make the performance test scripts more robust to ensure there are no scripts issues going further. This initiation of Script enhancement helped us add custom logic, enable basic parameters and dynamic value handling functions etc.

Tuning the system to improve Response time:

Performance Engineering team has simulated multiple load & stress tests and tuned many transactions which were designed to breach Service Level Agreement (SLA). The team focused on:

• Optimizing the DB query and Indexes
• Re-designing the web pages that are fetching lot of data at each request from end-user
• Focusing on JVM utilization, upgrading the Java scripts, and using optimized Garbage collection algorithms
• Improving load balancer algorithm
• Making sure Garbage collection remove the large objects form the heap memory
• Optimizing the Thread & Connection pool configuration

Fully automated test data pipeline setup:

Manging test data was one of the critical aspects as a lot of test data was not reusable. The application data was stored in Oracle and two other databases. So, before every test team has to request for the fresh data which was not ideal. Qentelli’s Performance Engineering team proposed to remove manual and data dependency completely and decided to automate this process without compromising the users’ access and data security. 
Our Performance team streamlined the entire data pipeline so that all the test data from multiple DBs would go through PostgreSQL (Performance test DB) and Load testing tool (JMeter) shall test the data from PostgreSQL DB without disturbing the main application DB. The benefit of having dedicated performance DB is that we can perform Read and Write operations more conveniently.

Investigating the memory leak:

Our Performance Engineering team executed multiple rounds of endurance tests, spending over 72 hours for each test. It helped the engineering team drill down and get to the root cause of memory leak. Application has multiple functionalities, it’s performing just fine when the user request required less data however, it was becoming a logjam when the user request needed a lot of data to display on the screen. The application was consuming a lot of memory not only from the server but from the end-user as well. Qentelli’s team suggested Pagination to reduce the load on JVM. The code was also identified as one of the reasons for the memory leak in the system apart from user enrolment business flow.

Tech Stack