Let’s think for a moment about the meaning of "Quality Assurance" in modern systems.
It means that a person or a team is responsible for creating a product that meets the customer’s expectations i.e. ensures that the product is of appropriate quality and can be used by the end-user. Referring to the manufacturing process in the industry fits perfectly into the process where the manufacturer of a given element does not have tools that are capable of verifying a product or its part, in other words, the manufacturer is not able to certify the quality of his work, assuring the required quality of the product and the third-party must ensure it. Relating this process to software production, the person creating the system is the development team that analyzes, designs, and implements the IT system. The next step in the process is quality assurance, i.e. testing the system and possibly introducing corrections to it, which provides feedback for the development team and leads to a situation where the system must be improved again and then tested again. This story is dedicated to reviewing available QA methodologies and suggesting much more effective possibilities. Let’s go!
Agenda
In the case of some IT systems, the testing process is still largely a process that takes place with the participation of the tester, i.e. it is a manual process, which causes the consumption of resources and a significant extension of the time of making the system or service available to the market.
In a slightly different situation, some projects have automated tests that were created by the QA department and their launch is automatic. Thanks to this, we can save a lot of people’s time and immediately verify the quality of the product within the QA team.
But what if the system we are testing has changed - our tests after launching indicate a problem and the question remains - should we adjust the tests or is the system actually malfunctioning?
Once again, we are in a situation where we need to involve human resources and undertake the analysis and then the necessary actions to explain why the tests were not successful.
In that case, are they really automated tests?
We are dealing here with a semi-automated process, so in fact, it is still a manual process.
This is because the QA team naturally cannot make changes to our IT system, just as the part inspection team at the factory cannot correct the part.
Well, bugs cannot be discovered so late, we need to figure out how to do this much earlier - it’s called Shift Left Testing methodology'
However, the software production process may be completely different, because the development team has all the tools to ensure the final and unquestionable quality of the product without the participation of the QA team. Are you surprised? Read on.
Unit tests
The software production process is one of the most complicated human development processes. The programmer and the team of programmers know every element of the system inside out, because, after prior analysis, he designed and brought it to life himself. This means that the programmer knows the component and knows how it should behave in any situation, standard or boundary - so he can successfully write tests that verify the component’s operation on a micro-scale - so he can write tests which we call unit tests. These tests are very fast and should cover most of the source code - it is assumed that it is good practice to cover 80% of the source code of the system. Unit tests are run automatically in the process of building the system, are fast, and are aimed at initial verification of functionalities created by the development team in isolation from interaction with other components.
The scrum methodology is characterized by the pursuit of the scrum team towards a common goal in an iterative approach. The team gains knowledge and experience with each successive sprint, which is the basis for further improvement of both the effectiveness of the scrum team and the resulting product. Scrum gives the opportunity to constantly focus on the business need, or to ensure the right speed of adaptation to the changing environment and market, which is crucial for the project’s success. "Gives the opportunity" was used consciously, because scrum alone does not solve anything by itself. It only gives the framework in which the teams operate, while the effectiveness depends only on the level of maturity. It is important to understand that the true value of Scrum is not measured by running scrum events or by producing artifacts, although they are very important. The real value of scrum is when the team takes full responsibility for the implementation of a given business initiative from start to the end, supporting it with all their experience and knowledge. The success of the project is the success of the team. This is the essence of Agile.
Module tests
Unit testing is the first stage of automated testing that a developer creates when designing and implementing an IT system. Although according to good practices, unit tests should verify the system extensively, we cannot base our approach to the preparation of automated tests of modern IT systems only on them. Another equally important element is module tests, which have a completely different responsibility. They are launched after unit tests when we know that the initial stability of each of our components is already assured. To prepare module tests, we need knowledge that allows us to use the system as a black box, ie. In the case of backend systems, we should know the API of our services, know how to seed the system with data, and know the mechanisms and processes taking place in our system. So we need development and analytical knowledge - testers do not have this knowledge - they can get it, but it is a cost that should be added to the cost of software development. So who has the necessary knowledge to test the module? The answer again is our development team who designed and brought this module to life. So the question arises - transfer knowledge to the QA team and introduce this team to our production pipeline, or can it test the module within the development team?
There is only one answer for a modern production model - the module should not leave the table without prior testing and the development team will do it most efficiently in case of any defects it will be able to quickly locate, repair, or adjust existing test scenarios if needed. Here it’s necessary to answer an important question - for what purpose do we write automated tests? Here, contrary to popular opinion, tests are not created to verify the currently developing functionality, but mainly to avoid introducing bugs in existing and operating processes when creating new functionalities combined with the previous ones.
Module tests give us confidence that it has passed all running tests in isolation from other components, it is operational and ready to be integrated with other elements of the microservices ecosystem or frontend components.
If you use Docker environment please consider TestContainers framework to provide third party components like databases, cache etc. to your microservice. This is a tool recommended by Technology Radar.
This is the End … To End
When the second phase of tests is behind us and we are sure that our component works stable in isolation from other services of our architecture, and has an efficient and working API, it is high time to move on to the last phase of automated tests, which we cannot ignore or skip. This phase is often assigned to external teams, E2E or QA, and verifies a working system often consisting of hundreds of microservices and frontend or micro frontend component layers. Here, the key element is the knowledge of the system as well as the technology that is appropriate and adequate for use in the E2E tests. Depending on the application we deal with, it may be a different technology, but if we are talking in terms of Web applications, it is worth considering the JavaScript Cypress framework, which is currently the best tool for testing applications recommended by Technology Radar. To create E2E tests, it is worth considering and taking into account all aspects of an automated test process, as well as bad and good testing practices. We know from experience that some teams carry out E2E tests in isolation from the development process, i.e. implement advanced E2E scenarios in isolation from the system and run test runs during the so-called "night builds" due to the long duration of runs.
This is the first and key problem with overloaded E2E tests, which are inefficient, slow, and not stable. Therefore, we recommend that you adopt the following principles, which are consistent with best practices presented by Martin Fowler:
-
E2E tests should be completely automated
-
E2E tests should be placed as part of the Continuous Integration flow Therefore, they should be fast - a maximum of 20 minutes.
-
E2E tests should be repeatable and stable They should run on predictable data that is imported before each test case.
If our E2E tests are prepared following the above principles, we are sure that our system has a chance to be well tested and we can sleep well during the next releases of our software development. The last question we should ask ourselves is - who should be responsible for the implementation of E2E tests? QA team or maybe our development team again?
Summary
Experience shows that often QA teams try to carry out E2E tests, choosing their technological stack, and trying to automate the solution, but ultimately this solution doesn’t work as we would like it. The process is not closely related to the development process and is rather an extension than a strict element. We are dealing here with a situation similar to module testing, where the process should be closely related to the development process. So we have to make a decision - who will be responsible for the implementation of E2E tests? To answer this question, we should know the structure of our organization, operating model, and mindset. In the case of organizations where the Agile methodology is not popular, the E2E team will most likely be responsible for E2E testing and the approach to E2E testing will be very radical and will be based on testing all systems in the production line. If, on the other hand, our organization is based on Agile, and we develop a modern system based on the microservice model, distributed between several dozen/hundreds of components, then we certainly cannot afford to perform E2E tests at such a late production stage in isolation from our pipeline. Always following the left test shifting methodology, we should try to test our solution to the maximum and at the earliest possible stage of its production. More information about automation you will find on our DevOps post.
Azure DevOps for tests automation