Let’s think a moment about the meaning of the "Quality Assurance" in modern systems.
It means that a person or a team is responsible for creating a product that meets the customer’s expectations i.e. ensures that the product is of appropriate quality and can be used by the end user. Referring to the manufacturing process in industry, it fits perfectly into the process where the manufacturer of a given element does not have tools that are capable of verifying a product or its part, in other words, the manufacturer is not able to certify the quality of his work, assuring the required quality of the product and the third-party must ensure it. Relating this process to software production, the person creating the system is the development team that analyzes, designs and implements the IT system. The next step in the process is quality assurance, i.e. testing the system and possibly introducing corrections to it, which causes feedback with the development team and leads to a situation where the system must be improved again and then tested again. This story is dedicated to review available QA methodologies and suggest much more effective possibilities. Let’s go!
In the case of some IT systems, the testing process is still largely a process that takes place with the participation of the tester, i.e. it is a manual process, which causes the consumption of resources and a significant extension of the time of making the system or service available to the market.
In a slightly different situation, there are projects that have automated tests that were created by the QA department and their launch is automatic. Thanks to this, we are able to save a lot of people’s time and immediately verify the quality of the product within the QA team.
But what if the system we are testing has changed - our tests after launching indicate a problem and the question remains - should we adjust the tests or is the system actually malfunctioning?
Once again, we are in a situation where we need to involve human resources and undertake the analysis and then the necessary actions to explain why the tests were not successful.
In that case, are they really automated tests?
We are dealing here with a semi-automated process, so in fact it is still a manual process.
This is because the QA team naturally cannot make changes to our IT system, just as the part inspection team at the factory cannot correct the part.
Well, bugs cannot be discovered so late, we need to figure out how to do this much earlier - it’s called
Shift Left Testing methodology'
However, the software production process may be completely different, because the development team has all the tools to ensure the final and unquestionable quality of the product without the participation of the QA team. Are you surprised? Read on.
The software production process is one of the most complicated human development processes. The programmer and the team of programmers knows every element of the system inside out, because, after prior analysis, he designed and brought it to life himself. This means that the programmer knows the component and knows how it should behave in any situation, standard or boundary - so he can successfully write tests that verify the component’s operation on a micro scale - so he can write tests which we call unit tests. These tests are very fast and should cover most of the source code - it is assumed that it is good practice to cover 80% of the source code of the system. Unit tests are run automatically in the process of building the system, are fast and are aimed at initial verification of functionalities created by the development team in isolation from interaction with other components.
The scrum methodology is characterized by the pursuit of the scrum team towards a common goal in an iterative approach. The team gains knowledge and experience with each successive sprint, which is the basis for further improvement of both the effectiveness of the scrum team and the resulting product. Scrum gives the opportunity to focus and constantly focus on the business need, or to ensure the right speed of adaptation to the changing environment and market, which is crucial for the project’s success. "Gives the opportunity" was used consciously, because scrum alone does not solve anything by itself. It only gives the framework in which the teams operate, while the effectiveness depends only on the level of maturity. It is important to understand that the true value of Scrum is not measured by running scrum events or by producing artifacts, although they are very important. The real value of scrum is when the team takes full responsibility for the implementation of a given business initiative from start to the end, supporting it with all their experience and knowledge. The success of the project is the success of the team. This is the essence of Agile.
Unit testing is the first stage of automated testing that a developer creates when designing and implementing an IT system. Despite the fact that, according to good practices, unit tests should verify the system extensively, we cannot base our approach to the preparation of automated tests of modern IT systems only on them. Another equally important element are module tests, which have a completely different responsibility. They are launched after unit tests, when we know that the initial stability of each of our components is already assured. To prepare module tests, we need knowledge that allows us to use the system as a black box, ie. In the case of backend systems, we should know the API of our services, know how to seed the system with data and know the mechanisms and processes taking place in our system. So we need development and analytical knowledge - testers do not have this knowledge - they can get it, but it is a cost that should be added to the cost of software development. So who has the necessary knowledge to test the module? The answer again is our development team who designed and brought this module to life. So the question arises - transfer knowledge to the QA team and introduce this team to our production pipeline, or can it test the module within the development team?
There is only one answer for a modern production model - the module should not leave the table without prior testing and the development team will do it most efficiently and in case of any defects it will be able to quickly locate, repair or adjust existing test scenarios if needed. Here it’s necessary to answer an important question - for what purpose do we write automated tests? Here, contrary to popular opinion, tests are not created to verify the currently developing functionality, but mainly to avoid introducing bugs in existing and operating processes when creating new functionalities combined with the previous ones.
Module tests give us confidence that it has passed all running tests in isolation from other components, it is operational and ready to be integrated with other elements of the microservices ecosystem or frontend components.
This is the End … To End
This is the first and key problem with overloaded E2E tests, which are inefficient, slow and not stable. Therefore, we recommend that you adopt the following principles, which are consistent with best practices presented by Martin Fowler:
E2E tests should be completely automated
E2E tests should be placed as part of the Continuous Integration flow Therefore, they should be fast - a maximum of 20 minutes.
E2E tests should be repeatable and stable They should run on predictable data that is imported before each test case.
If our E2E tests are prepared in accordance with the above principles, we are sure that our system has a chance to be well tested and we can sleep well during next releases of our software development. The last question we should ask ourselves is - who should be responsible for the implementation of E2E tests? QA team or maybe our development team again?
Experience shows that often QA teams try to carry out E2E tests, choosing their own technological stack, trying to automate the solution, but ultimately this solution doesn’t work as we would like it. The process is not closely related to the development process and is rather an extension than a strict element. We are dealing here with a situation similar to module testing, where the process should be closely related to the development process. So we have to make a decision - who will be responsible for the implementation of E2E tests? To answer this question, we should know the structure of our organization, operating model and mindset. In the case of organizations where the Agile methodology is not popular, the E2E team will most likely be responsible for E2E testing and the approach to E2E testing will be very radical and will be based on testing all systems in the production line. If, on the other hand, our organization is based on Agile, and we develop a modern system based on the microservice model, distributed between several dozen / hundreds of components, then we certainly cannot afford to perform E2E tests at such a late production stage in isolation from our pipeline. Always in accordance with the left test shifting methodology, we should try to test our solution to the maximum and at the earliest possible stage of its production. More information about automation you will find on our DevOps post.