Performance Engineering and its Evolution
The evolution from Performance Testing to Performance Engineering is a journey that culturally changes the organization’s focus from simply executing tests, analyzing, and delivering results to making business decisions and achieving business outcomes driven by performance. Moving from performance testing through the next levels as the organization matures is really about addressing some of the common challenges that prevent the organizations from successful performance.
- Load Testing : SLA based, wide range of tools, technologies, and protocols used, mainly focused on System and Hardware
- Performance Testing : Measures the client-side performance, always correlated with load testing, testing may be limited to the test environments
- Performance Analytics : Analyze operations data to realistically address performance profiles, create performance repository, utilize the results data to identify the root cause of performance issues
- Performance Engineering : Continuous business feedback and improvements required, built-in and automated performance, optimized applications for business and customer value, collaborative and interactive team focused on quality
Performance Engineering Accelerator
Performance Engineering Accelerator consists of 4 components/phases:
- Architect : Performance Engineering before starting development
- Build : DevOps aspects for continuous Integration for effective and fast delivery
- Deliver : Continuous deployment and operations
- Monitoring : Predictive, growth, features and continuous feedback
Phase #1 – Architect
This is the planning phase. The Performance Engineer (PE) will work with the business user, key stakeholders, project teams to identify and document the key business transactions, scenarios, user stories and corresponding SLAs. The next task is to identify the workload profile – number of users, average, peak, concurrent users, regional users, capacity planning, etc. One of the key aspects is to define the performance objectives and goals. Once you define the objectives, the required funding, resources, tools, infrastructure, environments have to be identified. The scenarios, transactions, workload profiles, etc have to reviewed and signed off with the stakeholders.
Phase #2 – Build
In this phase, the PE will define the performance test levels, develop the test data management plan, define the metrics to be captured, document the overall performance testing strategy, acceptance criteria, develop the test scripts, define the baseline, environment validation, monitoring tools, etc., and the DevOps aspects for continuous integration for effective and faster delivery should be considered.
Phase #3 – Deliver
Once the application is ready, the test scripts will be executed with a few users and/or a large number of users. This is can be done at the component level or user story level or the module level. Continue to execute tests for average, peak, double peak, and concurrent volumes. Performance monitoring and tuning play an important role. The PE should have the expertise on application performance management, performance monitoring, and tuning expertise. Implement continuous build and test execution if required.
Phase #4 – Monitoring
This involves monitoring the current growth, reviewing the capacity planning, improve the capacity planning for the near term and long term, improve operational efficiencies, and implement continuous feedback for both process and operations.
Testing Metrics
As part of the metrics collection and reporting, the PE will capture the Response Metrics – Average, Peak, Double Peak and Endurance response times, error rates, Volume Measurements – Concurrent Users, Requests per Second and Throughput. At the OS level, CPU, memory, memory leak, disk utilization, and network utilization will be captured. At the App/Web Server levels, memory utilization, garbage collection, number of sessions, hit ratio, and thread utilization will be captured. At the DB level, buffer pool, cache utilization, number of transactions, commits, % indexed queries, etc will be captured.
